Displaying the Message Queue
The Tools > Message Queue Window displays your MPI program’s message queue state textually. This can be useful when you need to find out why a deadlock occurred.
MPI versions that support message queue display are described in the Platforms Guide in the product distribution at <installdir>/totalview.<version>/doc/pdf, or available on the TotalView documentation website at https://help.totalview.io/.
About the Message Queue Display
After an MPI process returns from the call to MPI_Init(), you can display the internal state of the MPI library by selecting the Tools > Message Queue command, Figure 232.
 |
This window displays the state of the process’ MPI communicators. If user-visible communicators are implemented as two internal communicator structures, TotalView displays both. One is used for point-to-point operations and the other is used for collective operations.
NOTE: You cannot edit any of the fields in the Message Queue Window.
The contents of the Message Queue Window are valid only when a process is stopped.
Using Message Operations
For each communicator, TotalView displays a list of pending receive operations, pending unexpected messages, and pending send operations. Each operation has an index value displayed in brackets ([n]).
Message Queue field descriptions available for display | "Message Queue Window" in the in-product help |
Message operations | "Message Operations" in the in-product Help |
Diving on MPI Processes
To display more detail, you can dive into fields in the Message Queue Window. When you dive into a process field, TotalView does one of the following:
Raises its Process Window if it exists.Sets the focus to an existing Process Window on the requested process.Creates a new Process Window for the process if a Process Window doesn’t exist.Diving on MPI Buffers
When you dive into the buffer fields, TotalView opens a Variable Window. It also guesses the correct format for the data based on the buffer length and data alignment. You can edit the Type field within the Variable Window, if necessary.
NOTE: TotalView doesn’t use the MPI data type to set the buffer type.
About Pending Receive Operations
TotalView displays each pending receive operation in the Pending receives list. Figure 233 shows an example of an MPICH pending receive operation.
 |
NOTE: TotalView displays all receive operations maintained by the IBM MPI library. Set the environment variable MP_EUIDEVELOP to DEBUG to make blocking operations visible; otherwise, the library maintains only nonblocking operations. For more details on this variable, see the IBM Parallel Environment Operations and Use manual.
About Unexpected Messages
The Unexpected messages portion of the Message Queue Window shows information for retrieved and enqueued messages that are not yet matched with a receive operation.
Some MPI libraries, such as MPICH, only retrieve messages that have already been received as a side effect of calls to functions such as MPI_Recv() or MPI_Iprobe(). (In other words, while some versions of MPI may know about the message, the message may not yet be in a queue.) This means that TotalView can’t list a message until after the destination process makes a call that retrieves it.
About Pending Send Operations
TotalView displays each pending send operation in the Pending sends list.
MPICH does not normally keep information about pending send operations. If you want to see them, start your program under TotalView control and use the mpirun -ksq or -KeepSendQueue command.
Depending on the device for which MPICH was configured, blocking send operations may or may not be visible. However, if TotalView doesn’t display them, you can see that these operations occurred because the call is in the stack backtrace.
If you attach to an MPI program that isn’t maintaining send queue information, TotalView displays the following message:
Pending sends : no information available
