Using ReplayEngine with Infiniband MPIs

TotalView : TotalView User Guide : PART III Parallel Debugging : Setting Up Parallel Sessions : MPI Program Setup : Using ReplayEngine with Infiniband MPIs

In general, using ReplayEngine with MPI versions that communicate over Infiniband is no different than using it with other MPIs, but its use requires certain environment settings, as described here. If you are launching the MPI job from within TotalView, these are set for you; if instead, you start the MPI program from outside TotalView, you must explicitly set your environment.

Required Environment Settings

When you start the MPI program from within TotalView with ReplayEngine enabled, TotalView inserts environment variable settings into the MPI processes to disable certain RDMA optimizations. (These are optimizations that hinder ReplayEngine’s ability to identify the memory regions being actively used for RDMA, and their use can therefore result in unreasonably slow execution in record mode.) These variables are set for you, requiring no extra tasks compared to using a non-Infiniband MPI.

The inserted settings are:

VIADEV_USE_DREG_CACHE=0 (addresses MVAPICH1 versions)

MV2_DREG_CACHE_LIMIT=1 (addresses MVAPICH2 versions)

MV2_RNDV_PROTOCOL=R3 (addresses Intel MPI versions, also affects MVAPICH2)

OMPI_MCA_mpool_rdma_rcache_size_limit=1 (addresses Open MPI versions)

When the MPI program is started outside TotalView (for example, when using a command like mpirun -tv, or when you attach TotalView to an MPI program that is already running), you must set the relevant environment variable for your MPI version, as described above. Also, two additional environment variables are required to make the MPI program's use of RDMA memory visible to ReplayEngine, as follows:

IBV_FORK_SAFE: Set to any value, for example IBV_FORK_SAFE=1

LD_PRELOAD: Set to include a preload library, which can be found under the TotalView installation directory at toolworks/totalview.<version>/linux-x86/lib/undodb_infiniband_preload_x32.so or toolworks/totalview.<version>/linux-x86-64/lib/undodb_infiniband_preload_x64.so.

For example, here’s how to set the environment for the MVAPICH1 implementation of MPI:

mpirun_rsh -np 8 -hostfile myhosts \

VIADEV_USE_DREG_CACHE=0 IBV_FORK_SAFE=1 \

LD_PRELOAD=/<path>/undodb_infiniband_preload_x64.so myprogram

For more information, consult your MPI version documentation for specifics on setting environment variables.

Cray XT/XE/XK/XC MPIs

On Cray XT/XE/XK/XC (x86_64 only) systems, although Infiniband is not used, the MPIs do use RDMA techniques. As a result, using Replay on these systems requires some particular environmental settings. Briefly, the required settings are MPICH_SMP_SINGLE_COPY_OFF = 1, and LD_PRELOAD set to the location of the Infiniband preload library described above. Refer to Cray XT/XE/XK/XC Applications for details.

Possible Errors

ReplayEngine checks environment settings before it attaches to the MPI program, but in some cases, may not detect incompatible settings, reporting the following errors:

If ReplayEngine finds that either the IBV_FORK_SAFE setting is absent, or that the preload library has not been loaded, it declines to attach and issues an error message citing unmet prerequisites. You can still attach TotalView to the program without ReplayEngine - for example, in the GUI by using the New Program dialog.

If ReplayEngine cannot determine that the environment variable setting to disable an MPI optimization has been set, it continues to attach, but issues a warning message that it could not verify prerequisites. Depending on your program's use of memory for RDMA, you may find that it runs unreasonably slowly in record mode, or encounters errors that would not occur if ReplayEngine were not attached.


Using ReplayEngine in general	Using Replay Engine