AMD ROCm Wave Thread Debugging

As mentioned in The TotalView AMD ROCm Debugging Model, TotalView supports two modes of viewing GPU execution within a process, agent-thread mode and wave-thread mode, with the ability to toggle a process between the two modes. TotalView uses agent-thread mode for processes by default, as this is known to perform better in the debugger because it is handling far fewer debugger threads per process. However, wave-thread mode enables more granular execution control at the GPU wavefront level by leveraging TotalView’s asynchronous thread control features.

Asynchronous thread control refers to a set of TotalView features that allow the debugger to individually control the execution of “threads” in a process. This granular thread execution control enables you to run and halt individual threads, single-step a group of threads in lockstep, hold and release the execution of individual threads, and create stop-thread and thread barrier breakpoints. When a ROCm process is in wave-thread mode, many of the asynchronous thread control features apply to the GPU wave threads.

Benefits of using asynchronous wave thread debugging include the following:

  • Wave-Level Execution Control

    Increase execution-control precision by managing individual wavefronts. By controlling threads at the wave level, you can specify the execution of individual wavefronts for a finer level of control. You can set and clear stop-thread breakpoints, hold and release threads, and perform single-steps for specific waves, giving you more precise execution control than in agent-thread mode.

  • Enhanced GPU Breakpoint Functionality

    Set breakpoints at the wave level to stop only the triggering wave threads while allowing other threads in the process to continue running. Stop-thread and thread-specific conditional breakpoints allow all waves to reach the same location.

  • Improved Data Visibility

    Examining variables and data across wave threads within a process allows you to view variable values across the waves in a process.

TotalView provides asynchronous wave thread control through both the command line interface and the TotalView UI. For more information, see Using Wave Thread Debugging.

In this section: