Viewing GPU Threads
Once the GPU kernel starts executing, it will hit breakpoints planted in the GPU code, as shown in Figure 155.
Figure 155, ROCm thread stopped at a breakpoint, focused on GPU work-item (0,0,0)[1,0,0]
The logical coordinates of the GPU focus work-group and work-items are displayed in the GPU logical toolbar at the top of Figure 155 above. (See GPU Toolbars.) The WorkGroup control shows the 3-D focus position of the work-group in the kernel dispatch. The WorkItem control shows the 3-D focus position of the work-item in the work-group.
The GPU focus work-group and/or work-item can be changed using the GPU focus selector in the logical toolbar. When you change the GPU focus work-group or work-item, the logical coordinates displayed also change, and the Call Stack and Source view are updated to reflect the state of the new GPU focus work-group or work-item.
The execution location of the GPU focus work-item is identified by the yellow PC highlighted line in the Source view. The work-items are grouped into "wavefronts" (or "waves" for short) that execute in parallel, so multiple work-items may have the same PC value. The work-items may be part of the same wave (consisting of 32 or 64 lanes that execute concurrently), or part of different waves.
The Local Variables view shows the parameters and local variables for the function in the selected stack frame. The variables for the selected GPU kernel code, function, or inlined function expansion are shown.
The Call Stack shows the stack backtrace and inlined functions:
Each stack frame in the stack backtrace represents either the PC location of GPU kernel code, or the expansion of an inlined function.
GPU Agent Thread IDs and Coordinate Spaces
Again, TotalView gives host threads a positive debugger thread ID and GPU agent threads a negative thread ID. In this example, the initial host thread in process "1" is labeled "1.1" and the GPU agent thread is labeled "1.-1".
Figure 156, ROCm GPU Thread IDs
Use the "GPU focus selector" on the GPU toolbar (See GPU Toolbars) to change the logical coordinates of the GPU focus work-group and/or work-item.
Figure 157, GPU Focus Selector on the GPU logical toolbar
GPU Toolbars
Two GPU toolbars display the two coordinate spaces. One is the logical coordinate space that, in AMD GPU terms, incorporates work-groups and work-items: (Bx,By,Bz)[Tx,Ty,Tz].
The other is the "physical" coordinate space that is, in hardware terms, the wavefront ID (wave ID) assigned by the AMD GPU debug API and the lane number index within the wave. Note that the wave ID is an arbitrary, unique value, assigned by the debug API, and has no relationship to any physical aspect of the GPU agent.
Any given work-item has both a position in the physical coordinate space, and a position in the 6-D logical coordinate space. These indices are shown in the two GPU toolbars. Changing the GPU focus in one coordinate space changes the other.
Figure 158, GPU logical and physical toolbars
To view a host CPU thread, select a thread with a positive thread ID in the Process and Threads view. To view a GPU agent thread, select an agent thread with a negative thread ID, then use the GPU work-group and/or work-item selector on the logical toolbar to focus on a specific GPU work-item.
There is one GPU focus per GPU agent thread, and changing the GPU focus affects all windows displaying information for that agent and all command line interface commands targeting that agent. In other words, changing the GPU focus work-group or work-item can change the GPU data that is displayed and affect other commands, such as single-stepping.
Note that in all cases, when you select a work-item, TotalView automatically updates the Source view, Call Stack, Data View, and Action Points view to match the selected work-item.
