SGI MPI uses the xpmem module to map memory from one MPI process to another during job startup. Memory mapping is enabled by default. The size of this mapped memory can be quite large, and can have a negative effect on TotalView’s ReplayEngine performance. Therefore, mapped memory is limited by default for the xpmem module if Replay is enabled. The environment variable, MPI_MEMMAP_OFF, is set to 1 in the TotalView file parallel_support.tvd by adding the variable to the replay_env: specification as follows: replay_env: MPI_MEMMAP_OFF=1

If full memory mapping is required, set the startup environment variable in the Arguments field of the Program Session dialog. Add the following to the environment variables: MPI_MEMMAP_OFF=0.

Be aware that the default mapped memory size may prove to be too large for ReplayEngine to deal with, and it could be quite slow. You can limit the size of the mapped heap area by using the MPI_MAPPED_HEAP_SIZE environment variable documented in the SGI documentation. After turning off MEMMAP_OFF as described above, you can set the size (in bytes) in the TotalView startup parameters.

For example:

SGI has a patch for an MPT/XPMEM issue. Without this patch, XPMEM can crash the system if ReplayEngine is turned on. To get the XPMEM fix for the munmap problem, either upgrade to ProPack 6 SP 4 or install SGI patch 10570 on top of ProPack 6 SP 3.