Launch parallel Python programs

This topic describes how to launch parallel Python programs that use mpi4py with a supported MPI implementation. Use an MPI launcher to start these programs under TotalView. See Python debugging supported platforms and requirements for supported implementations.

Validate your MPI Python runtime before debugging

Before launching your program in TotalView, check that it runs directly with MPI. This check helps distinguish runtime problems from problems with the debugger launch configuration.

From the command line, run the following command. Replace <N> with the number of processes:

mpirun --mca op ^avx -np <N> <path-to-python> <path-to-script.py>

If the command succeeds but the TotalView launch fails, the problem may be in the debugger launch configuration.

Launch a parallel Python program

  1. From the command line, run one of the following commands:

    • Single-script parallel Python program: Replace <N> with the number of processes to launch and <script.py> with the path to your Python script. The command uses python as the interpreter.

      totalview -args mpirun -np <N> %tv_python_wrapper% python <script.py>

    • Multiple-program, multiple-data (MPMD) mpi4py session: Place %tv_python_wrapper% immediately before each python command. For example, to launch two different Python scripts, run:

      totalview -args mpirun -n <N> %tv_python_wrapper% python producer.py : -n <M> %tv_python_wrapper% python consumer.py

      Replace <N> and <M> with the number of processes to launch for producer.py and consumer.py, respectively. Replace the script names with the paths to your Python scripts. The colon (:) separates the two program specifications.

    During launch, take the appropriate action for each condition that occurs:

    • If you encounter "mpirun" in the Source view, select Go (Go command icon) to continue the launch.

    • If a dialog asks whether to stop the parallel job, select No.

    • If execution stops and displays the Python wrapper program (tv_python_wrapper.c) in the Source view, select Go (Go command icon).

      TotalView Source view showing the Python wrapper program

    TotalView loads your Python scripts and displays the MPI launcher and process ranks in the Processes & Threads view.

    TotalView showing the Processes and Threads view with a parallel Python program loaded

  2. (Optional) Create a grouping of the Processes & Threads view that is tailored to multiprocessing Python programs, see Group Python processes and threads.

  3. (Optional) Set any breakpoints you want in your Python code. See Set breakpoints in Python code.

  4. Select Go (Go command icon) to run your Python program.

View and switch process context

Use the Processes & Threads view to examine a specific process or thread in your parallel Python program. You can also navigate between processes and threads. The Source, Call Stack, and Local Variables views update to show the selected process or thread's context.

To examine a specific process or thread, use one of the following methods:

  • In the Processes & Threads view, select the process or thread you want to examine.

  • Use the Processes & Threads toolbar to navigate to another process or thread. Options include the following: 

    Processes & Threads toolbar with controls to move between processes and threads or jump to a specific process or thread

    • P+ / P-: Move to the next or previous process. You can also use the menu options Process > Next Process and Process > Previous Process.

    • T+ / T-: Move to the next or previous thread in the current process. You can also use the menu options Thread > Next Thread and Thread > Previous Thread.

    • Px: Opens the Jump To dialog, which you can use to jump to a specific process or thread. In the Jump To dialog, specify a debugger ID or, for an MPI process, a rank, then select OK. You can also use the menu option Process > Jump to Process/Thread (Px).

To learn more, see Change the process or thread of focus.

Skip over the Python wrapper during MPI debugging

To avoid stopping at the Python wrapper each time you launch an MPI program, configure the parallel attach behavior:

  1. Select File > Preferences to open the Preferences dialog.

  2. On the PARALLEL tab, under After Attach Behavior, select Continue the processes.

Preferences dialog with the Continue the processes option highlighted

With this setting enabled, TotalView continues MPI processes after attaching instead of stopping at the Python wrapper. This setting applies to MPI programs only. For single-process Python programs, select Go (Go command icon) if execution stops in the wrapper.

Resolve launch errors

If you encounter the following error when launching a parallel Python program, use the recommended action to resolve the issue.

Error: mpirun was unable to find the specified executable file

This error in the Input/Output view indicates a TotalView version that does not support native Python debugging. To resolve this error, use a TotalView version that supports native Python debugging.

Input/Output view showing the mpirun executable not found error

Related Topics

The Processes & Threads view