Special Requirements for Using ReplayEngine
On Crayx86_64 systems, the MPIs use RDMA techniques, similar to Infiniband MPIs. When using ReplayEngine on MPI programs, certain environment variable settings must be in effect for the MPI rank processes. These settings ensure that memory mapping operations are visible to ReplayEngine. The required environment variable settings are:
MPICH_SMP_SINGLE_COPY_OFF=1LD_PRELOAD: Set to include a preload library, which can be found under the TotalView installation directory at toolworks/totalview.<version>/linux-x86-64/lib/undodb_infiniband_preload_x64.so.When using APLS, these settings may be applied with the aprun -e option. For example, to have TotalView launch an MPI program with ReplayEngine enabled, use a command similar to this:
totalview -replay -args aprun -n 8 \
-e MPICH_SMP_SINGLE_COPY_OFF=1 \
-e LD_PRELOAD=/<path>/undodb_infiniband_preload_x64.so \
myprogram
When using SLURM, these settings may be applied with the srun --export option. For example:
totalview -replay -args srun -n 8 \
--export=ALL,MPICH_SMP_SINGLE_COPY_OFF=1,LD_PRELOAD=/<path>/undodb_infiniband_preload_x64.so \
myprogram
