General Use
This section discusses basic configuration options of MRNet with TotalView. If you are working on a Cray computer, look at the section specific to that system as well.
Disabling MRNet Before Startup
By default, TotalView uses the MRNet infrastructure on the platforms where it is supported (see the TotalView Platforms Guide for specifics). On platforms where MRNet is not supported, TotalView uses its standard vector-of-servers infrastructure.
If for some reason you do not want to use the MRNet infrastructure to debug an MPI job, you must first disable MRNet in TotalView before launching the MPI job. MRNet can be disabled by:
Starting TotalView with the -nomrnet option:prompt> totalview -nomrnet
With TotalView running, use the command line interface (CLI) to set the TV::mrnet_enabled state variable:prompt> dset TV::mrnet_enabled false
MRNet Server Launch String
Option: -mrnet_server_launch_string string
State variable: TV::mrnet_server_launch_string string
Default string: %B/tvdsvr%K -working_directory %D -set_pw %P -verbosity %V %F
The server launch string defines configuration options when launching a debugging server. TotalView has a default string it uses when launching a server using the vector-of-servers architecture, and an option and state variable that allow you to modify the default string. The MRNet usage of TotalView also has a default launch string and corresponding option and state variables.
The MRNet launch string differs from the standard launch string in two ways: it does not contain a remote shell command expansion (e.g., rsh or ssh), and it has no -callback option.
TotalView always appends the following string to the expanded MRNet launch string:
-mrnet_launch node_id
where node_id is an integer that specifies the server's TotalView node ID within the job. If node_id is 0, the server assigns itself a node ID equal to its MRNet rank plus 1.
Controlling the Shape of an MRNet Tree
The shape of the MRNet tree calculated by TotalView can be controlled through a collection of options and state variables. Given the list of hosts, which is typically extracted from the MPIR proctable, TotalView calculates an MRNet topology string to create various shapes of trees.
These are the basic controls:
Tree fan-out: specifies the maximum number of children a node can have. If the number of leaves in the tree is not a power of the fan-out, some of the tree nodes will have fewer children.Tree depth: specifies the maximum depth of the tree (that is, the number of levels below the root). If the number of leaves is not greater than the square of the tree depth value, a shallower tree is built.Extra root node: Whether to allocate an extra communications node below the root.Create a “super bushy” tree: Create one debugger server process per MPI process rather than the default of creating one debugger server process per node, to overcome a CUDA limitation.MRNet Tree Fan-Out
Option: –mrnet_fanout integer
State variable: TV::mrnet_fanout integer
Default value: 32
If you change the default value, the new value must be greater than or equal to 2 and less than or equal to 32768.
MRNet Tree Depth
Option: –mrnet_levels integer
State variable: TV::mrnet_levels integer
Default value: 2
The MRNet tree depth can be specified in terms of the number of levels below the root. If you change the default value, the new value must be greater than or equal to -2 and less than or equal to 32.
If the tree depth is 0, the MRNet tree fan-out value is used, and TotalView attempts to honor the fan-out value near the bottom of the tree (the leaves).If the tree depth is set to a value that is greater than 0 (which includes the default value of 2), the fan-out value is ignored and a balanced tree is built with at most the specified number of levels.If the tree depth is -1, the fan-out value is used, and TotalView attempts to honor the fan-out value near the top (the root) of the tree, rather than near the bottom of the tree (the leaves).If the tree depth is -2, TotalView builds a tree similar to the one created when the tree depth is -1, except that the tree is unbalanced from side-to-side.As an example, consider a tree with a root node and eight leaf nodes. If the fan-out value is 4 and the tree depth value is 0, a tree that is “bushy” near the leaves is built because TotalView honors fan-out at the leaf end of the tree.
root:1 => n1:2 n5:2 ;
n1:2 => n1:1 n2:1 n3:1 n4:1 ;
n5:2 => n5:1 n6:1 n7:1 n8:1 ;
However, for the same tree when the tree depth setting is -1, a tree that is “bushy” near the root is built because TotalView honors fan-out at the root end of the tree.
root:1 => n1:2 n3:2 n5:2 n7:2 ;
n1:2 => n1:1 n2:1 ;
n3:2 => n3:1 n4:1 ;
n5:2 => n5:1 n6:1 ;
n7:2 => n7:1 n8:1 ;
Allocate an Extra Root Node
Option: –mrnet_extra_root boolean
State variable: TV::mrnet_extra_root boolean
Default value: false
For example, for a tree with a root and eight leaf nodes, using a fan-out value of 4, a tree depth value of 0, and requesting an extra root node, the following topology string will be calculated:
root:3 => root:1 ;
root:1 => n1:2 n5:2 ;
n1:2 => n1:1 n2:1 n3:1 n4:1 ;
n5:2 => n5:1 n6:1 n7:1 n8:1 ;
Create a “Super Bushy” Tree
Option: -mrnet_super_bushy
State variable: TV::mrnet_super_bushy
Default value: false
Set this option to true if you are debugging an MPI job in which more than one CUDA process is running on a node. This option addresses the CUDA debug API limitation that allows a debugger process (such as the tvdsvr) to debug at most one target process using a GPU.
Path to MRNet Components
Option: -mrnet_commnode_path path-to-mrnet_commnode
State variable: TV::mrnet_commnode_path path-to-mrnet_commnode
Default value: tv-installation-root/platform/bin/mrnet_commnode
In a TotalView distribution, this is a path to a shell script that sets environment variables and execs the proper executable for the platform.
Path to the MRNet shared library directory
Option: -mrnet_filterlib_dir path-to-mrnet-shlib-directory
State variable: TV::mrnet_filterlib_dir path-to-mrnet-shlib-directory
Default value: tv-installation-root/platform/shlib/mrnet/obj
The TotalView server tree filters library libservertree_filters.so.1 and the MRNet libxplat.so and libmrnet.so libraries are stored in this directory.
Performance Notes
Perforce has conducted performance tests on some specific systems, and based on this testing, we’re providing a few tips. These tips should be considered as guidelines. The only way to know how performance is affected by different tree configurations on your system is by trying out alternatives with your own jobs.
In general, higher fan-outs seem to perform better than deeper trees. Specifically, trees deeper than two levels consistently performed worse than a two-level tree.In our testing, a one-level tree failed due to resource shortages at around 512 nodes, so this is not a viable option at higher scales. MRNet and ssh/rsh
Controlling MRNet’s Use of rsh vs ssh
When MRNet is used as the infrastructure in a Linux cluster front-end node, MRNet's built-in support is used to instantiate the tree of debugger servers and communications processes. Tree instantiation is based on a remote shell startup mechanism. By default, MRNet uses ssh as the remote shell program, but some environments require that rsh be used instead. TotalView controls the remote shell used by MRNet using the TV::xplat_rsh state variable or the -xplat_rsh TotalView command option to set this state variable. If this variable isn't explicitly set and the XPLAT_RSH environment variable is not set or is empty, TotalView uses the value of TV::launch_command when instantiating an MRNet tree.
On Cray XT, XE, and XK systems, MRNet uses the ALPS Tool Helper library to instantiate the tree, which does not require the use of a separate remote shell program.
Tips on Using ssh/rsh with MRNet
The use of rsh / ssh differs in every system environment, therefore you should consult your system's documentation to know whether rsh or ssh should be used for your system. The rsh and ssh man pages are also a useful resource. Regardless, we offer the following tips as a guideline for how to configure rsh and ssh:
Configure rsh or ssh to allow accessing the remote nodes without a password. rsh typically uses a file named $HOME/.rhosts (see man 5 rhost on a Linux system). ssh typically uses a pair of private/public keys stored in files under your $HOME/.ssh directory (see man 1 ssh on a Linux system).Disable X11 forwarding in ssh in your $HOME/.ssh/config file (see man 5 ssh_config on a Linux system).Set StrictHostKeyChecking to no in ssh in your $HOME/.ssh/config file (see man 5 ssh_config on a Linux system). If the ssh host keys change for a remote host, you may need to delete the lines for the host from the $HOME/.ssh/known_hosts file, or remove the file.
