Special Requirements for Using ReplayEngine
On Cray XC, XK, and XC systems, the MPIs use RDMA techniques, similar to Infiniband MPIs. When using ReplayEngine on MPI programs, certain environment variable settings must be in effect for the MPI rank processes. These settings ensure that memory mapping operations are visible to ReplayEngine. The required settings are:
• MPICH_SMP_SINGLE_COPY_OFF = 1
• LD_PRELOAD: Set to include a preload library, which can be found under the TotalView installation directory at toolworks/totalview.<version>/linux-x86/lib/undodb_infiniband_preload_x32.so or toolworks/totalview.<version>/linux-x86-64/lib/undodb_infiniband_preload_x64.so.
These settings may be applied with the aprun -e option. For example, to have TotalView launch an MPI program with ReplayEngine enabled, use a command similar to this:
totalview -replay -args aprun -n 8 \
-e MPICH_SMP_SINGLE_COPY_OFF=1 \
-e LD_PRELOAD=/<path>/undodb_infiniband_preload_x64.so \
myprogram