NextGen TotalView for HPC User Guide : PART III Using the CUDA Debugger : Chapter 14 CUDA Debugging Tutorial : GPU Error Reporting
GPU Error Reporting
By default, TotalView reports GPU exception errors as "signals." Continuing the application after these errors can lead to application termination or unpredictable results.
Table 4 lists reported errors, according to these platforms and settings:
• Exception codes Lane Illegal Address and Lane Misaligned Address are detected using all supported SDK versions when CUDA memcheck is enabled, on supported Tesla and Fermi hardware.
• All other CUDA errors are detected only for GPUs with sm_20 or higher (for example Fermi) running SDK 3.1 or higher. It is not necessary to enable CUDA memcheck to detect these errors.
 
Table 9: CUDA Exception Codes
Exception code
Error Precision
Error Scope
Description
CUDA_EXCEPTION_0:
“Device Unknown Exception”
Not precise
Global error on the GPU
An application-caused global GPU error that does not match any of the listed error codes below.
CUDA_EXCEPTION_1:
“Lane Illegal Address”
Precise (Requires memcheck on)
Per lane/thread error
A thread has accessed an illegal (out of bounds) global address.
CUDA_EXCEPTION_2:
“Lane User Stack Overflow”
Precise
Per lane/thread error
A thread has exceeded its stack memory limit.
CUDA_EXCEPTION_3:
“Device Hardware Stack Overflow”
Not precise
Global error on the GPU
The application has triggered a global hardware stack overflow, usually caused by large amounts of divergence in the presence of function calls.
CUDA_EXCEPTION_4:
“Warp Illegal Instruction”
Not precise
Warp error
A thread within a warp has executed an illegal instruction.
CUDA_EXCEPTION_5:
“Warp Out-of-range Address”
Not precise
Warp error
A thread within a warp has accessed an address that is outside the valid range of local or shared memory regions.
CUDA_EXCEPTION_6:
“Warp Misaligned Address”
Not precise
Warp error
A thread within a warp has accessed an incorrectly aligned address in the local or shared memory segments.
CUDA_EXCEPTION_7:
“Warp Invalid Address Space”
Not precise
Warp error
A thread within a warp has executed an instruction that attempts to access a memory space not permitted for that instruction.
CUDA_EXCEPTION_8:
“Warp Invalid PC”
Not precise
Warp error
A thread within a warp has advanced its PC beyond the 40-bit address space.
CUDA_EXCEPTION_9:
“Warp Hardware Stack Overflow”
Not precise
Warp error
A thread within a warp has triggered a hardware stack overflow.
CUDA_EXCEPTION_10:
“Device Illegal Address”
Not precise
Global error
A thread has accessed an illegal (out of bounds) global address. For increased precision, enable memcheck.
CUDA_EXCEPTION_11:
“Lane Misaligned Address”
Precise (Requires memcheck on)
Per lane/thread error
A thread has accessed an incorrectly aligned global address.
CUDA_EXCEPTION_12 : "Warp Assert"
Precise
Per warp
Any thread in the warp has hit a device side assertion.
CUDA_EXCEPTION_13: "Lane Syscall Error"
Precise (Requires memcheck on)
Per lane/thread error
A thread has corrupted the heap by invoking free with an invalid address (for example, trying to free the same memory region twice)
CUDA_EXCEPTION_14 : "Warp Illegal Address"
Not precise
Per warp
A thread has accessed an illegal (out of bounds) global/local/shared address. For increased precision, enable the CUDA memcheck option. See "Enabling CUDA Memory Checker Feature".
CUDA_EXCEPTION_15 : "Invalid Managed Memory Access"
Precise
Per host thread
A host thread has attempted to access managed memory currently used by the GPU.