Using ReplayEngine with Infiniband MPIs
In general, using ReplayEngine with MPI versions that communicate over Infiniband is no different than using it with other MPIs, but its use requires certain environment settings, as described here. If you are launching the MPI job from within TotalView, these are set for you; if instead, you start the MPI program from outside TotalView, you must explicitly set your environment.
Required Environment Settings
When you start the MPI program from within TotalView with ReplayEngine enabled, TotalView inserts environment variable settings into the MPI processes to disable certain RDMA optimizations. (These are optimizations that hinder ReplayEngine’s ability to identify the memory regions being actively used for RDMA, and their use can therefore result in unreasonably slow execution in record mode.) These variables are set for you, requiring no extra tasks compared to using a non-Infiniband MPI.
The inserted settings are:
VIADEV_USE_DREG_CACHE=0 (addresses MVAPICH1 versions)
MV2_DREG_CACHE_LIMIT=1 (addresses MVAPICH2 versions)
MV2_RNDV_PROTOCOL=R3 (addresses Intel MPI versions, also affects MVAPICH2)
OMPI_MCA_mpool_rdma_rcache_size_limit=1 (addresses Open MPI versions)
When the MPI program is started outside TotalView (for example, when using a command like mpirun -tv, or when you attach TotalView to an MPI program that is already running), you must set the relevant environment variable for your MPI version, as described above. Also, two additional environment variables are required to make the MPI program's use of RDMA memory visible to ReplayEngine, as follows:
IBV_FORK_SAFE: Set to any value, for example
IBV_FORK_SAFE=1 LD_PRELOAD: Set to include a preload library, which can be found under the TotalView installation directory at
toolworks/totalview.<version>/linux-x86/lib/undodb_infiniband_preload_x32.so or
toolworks/totalview.<version>/linux-x86-64/lib/undodb_infiniband_preload_x64.so.
For example, here’s how to set the environment for the MVAPICH1 implementation of MPI:
mpirun_rsh -np 8 -hostfile myhosts \
VIADEV_USE_DREG_CACHE=0 IBV_FORK_SAFE=1 \
LD_PRELOAD=/<path>/undodb_infiniband_preload_x64.so myprogram
For more information, consult your MPI version documentation for specifics on setting environment variables.
Possible Errors
ReplayEngine checks environment settings before it attaches to the MPI program, but in some cases, may not detect incompatible settings, reporting the following errors:
If ReplayEngine finds that either the
IBV_FORK_SAFE setting is absent, or that the preload library has not been loaded, it declines to attach and issues an error message citing unmet prerequisites. You can still attach TotalView to the program without ReplayEngine - for example, in the GUI by using the New Program dialog.
If ReplayEngine cannot determine that the environment variable setting to disable an MPI optimization has been set, it continues to attach, but issues a warning message that it could not verify prerequisites. Depending on your program's use of memory for RDMA, you may find that it runs unreasonably slowly in record mode, or encounters errors that would not occur if ReplayEngine were not attached.