New in TotalView 2020.1
TotalView 2020.1 Supported Platforms
Versions
Support Notes
Platform Support
TotalView User Guide
PART I An Introduction to TotalView
Getting Started
Introducing TotalView
An Initial Look at the Interface
Customizing the Interface
Preferences
Resizing
Drawers
Undocking and Docking
A Tour of the Interface
Central Area
Toolbars
Processes and Threads View
Call Stack View and Local Variables View
Data View
Lookup View
Action Points, CLI, and Logger Views
Help
Starting TotalView and Creating a Debugging Session
Debugging Commands
Creating and Managing Sessions
Setting up Debugging Sessions
Loading Programs from the Session Editor
Starting a Debugging Session
Debug a Program
Debug a Parallel Program
Attach to Process
Debug a Core or Replay Recording File
Load a Recent Session
Editing a Previous Session
Loading Programs Using the CLI
Options and Program Arguments
Options: Reverse Debugging
Program Environment: Environment Variables
Standard Input and Output
Modifying Arguments in an Open Session
Managing Sessions
Starting a Session from your Shell
Starting TotalView on a Script
Basic Debugging
Program Load and Navigation
Load the Program to Debug
Initial Display
Program Navigation
Stepping and Executing
Simple Stepping
Setting and Running to a Breakpoint (Action Point)
Set and Control Breakpoints
Run Your Program and Observe the Call Stack
Examining Data
Viewing Variables in the Local Variables View
Viewing Variables in the Data View
Watching Data Values Update
Moving On
Program Navigation
Navigating from within the Source Pane
Highlighting a String and the Find Function
The Lookup File or Function View
The Documents View
PART II Debugging Tools and Tasks
Working with Processes and Threads
The Processes and Threads View
How the Processes and Threads View Works
Debugging Command Width
Understanding Thread Width
Understanding Process Width
Understanding Group Width
Controlling Processes and Threads in a Running Session
Controlling Processes by Detaching from the Debugger
Process > Detach
Group > Detach
Program Execution and Share Groups
Synchronizing Processes and Threads Using Run To
Breakpoint Width Impact on Execution
Setting the Program Counter
Displaying a Thread Name
Thread Names in the UI
Thread Properties
Thread Options on dstatus
Process and Thread Attributes
Setting and Managing Action Points (Breakpoints)
About Action Points
Breakpoints
Setting Source-Level Breakpoints
Sliding Breakpoints
Breakpoints at a Specific Location
Pending Breakpoints
Pending Breakpoints on a Function
Pending Breakpoints on a Line Number
Conflicting Breakpoints
Breakpoints at Execution
Modifying a Breakpoint
Setting Breakpoints When Using the fork()/execve() Functions
Debugging Processes That Call the fork() Function
Debugging Processes that Call the execve() Function
Example: Multi-process Breakpoint
Evalpoints
Setting an Evalpoint
Creating a Pending Evalpoint
Modifying an Evalpoint
Creating Conditional Breakpoints
Patching Programs
Branching Around Code
Adding a Function Call
Correcting Code
Using Programming Language Constructs
Watchpoints
Creating Watchpoints
Displaying, Deleting, or Disabling Watchpoints
Modifying Watchpoints
Watching Memory
Triggering Watchpoints
Using Multiple Watchpoints
Performance Impact of Copying Previous Data Values
Using Watchpoint Expressions
Using Watchpoints on Different Architectures
Barrier Points
About Barrier Breakpoint States
Setting a Barrier Breakpoint
Creating a Satisfaction Set
Hitting a Barrier Point
Releasing Processes from Barrier Points
Changing Settings and Disabling a Barrier Point
Using Barrier Points
Barrier Point Illustration
Controlling an Action Point’s Width
About an Action Point’s Width: Group, Process or Thread
Setting the Action Point’s Width
Action Point Width and Process/Thread State
Managing and Diving on Action Points
Sorting
Diving
Deleting, Disabling, and Suppressing
Saving and Loading Action Points
More on Action Points Using the CLI
Saving Action Points to a File Using the CLI
Suppressing and Unsuppressing Action Points
Examining and Editing Data
Viewing Data in TotalView
About Expressions
Using C++
The Call Stack and Local Variables Views
The Call Stack View
The Local Variables View
Viewing Call Stack Data
Viewing Data in Fortran
Viewing Modules and Their Data
Common Blocks
Fortran 90 User-Defined Types
Fortran 90 Deferred Shape Array Types
Fortran 90 Pointer Types
Fortran Parameters
The Data View
Adding Variables to the Data View
Add to the Data View from the Local Variables View
Move an Expression or Variable from the Source View to the Data View
Creating a New Expression by Diving on a Variable in the Data View
Entering a New Expression into the Data View
Editing an Expression
Dereferencing a Pointer
Changing the Value of Data
Casting to Another Type
Displaying Arrays
Controlling STL Data Transformation
Customizing the Data View
Duplicating the Data View
The Data View Drawer
Viewing Array Statistics
Slicing Arrays
Array Statistics Detail
Using the CLI to Examine Data
Changing the Display of Data
Displaying Variables
Debugging Python
Overview
Python Debugging Requirements
Python Version
Limitations and Extensions:
Starting a Python Debugging Session
Debugging Python and C/C++ with TotalView
Transforming the Stack
Viewing and Comparing Python and C/C++ Variables
Leveraging Other Debugging Technologies for Python Debugging
Supported Python Extension Technologies for Stack Transformations
Using the Command Line Interface (CLI)
Access to the CLI
Introduction to the CLI
About the CLI and Tcl
Integration of the CLI and the UI
Invoking CLI Commands
Starting the CLI in a Terminal Window
Startup Example
Starting Your Program
About CLI Output
‘more’Processing
Using Command Arguments
Using Namespaces
About the CLI Prompt
Using Built-in and Group Aliases
How Parallelism Affects Behavior
Types of IDs
Controlling Program Execution
Advancing Program Execution
Using Action Points
Examples of Using the CLI
Setting the CLI EXECUTABLE_PATH Variable
Initializing an Array Slice
Printing an Array Slice
Writing an Array Variable to a File
Automatically Setting Breakpoints
Controlling fork, vfork, and execve Handling
The exec_handling and fork_handling Command Options and State Variables
Exec Handling
Fork Handling
Example
Reverse Connections
About Reverse Connections
Reverse Connection Environment Variables
TV_REVERSE_CONNECT_DIR
TV_CONNECT_OPTIONS
Starting a Reverse Connect Session
Listening for Reverse Connections
Reverse Connect Examples
CLI Example
MPI Batch Script Example
Troubleshooting Reverse Connections
Stale Files in the Reverse Connect Directory
Directory Permissions
User ID Issues
Reverse Connect Directory Environment Variable
Preferences
About Preferences
Action Points
Display Settings
Tool Bar
Search Path
Parallel Configuration
PART III Accessing TotalView Remotely
TotalView Remote Display
Remote Display Supported Platforms
Remote Display Components
Installing the Client
Installing on Linux
Installing on Microsoft Windows
Installing on Apple Mac OS X Intel
Client Session Basics
Working on the Remote Host
Advanced Options
Naming Intermediate Hosts
Submitting a Job to a Batch Queuing System
Setting Up Your Systems and Security
Session Profile Management
Batch Scripts
tv_PBS.csh Script
tv_LoadLeveler.csh Script
PART IV Using the CUDA Debugger
About the TotalView CUDA Debugger
Overview
Installing the CUDA SDK Tool Chain
Directive-Based Accelerator Programming Languages
CUDA Debugging Model and Unified Display
The TotalView CUDA Debugging Model
Pending and Sliding Breakpoints
Unified Source View and Breakpoint Display
CUDA Debugging Tutorial
Compiling for Debugging
Compiling for Fermi
Compiling for Fermi and Tesla
Compiling for Kepler
Compiling for Pascal
Compiling for Volta
Starting a TotalView CUDA Session
Controlling Execution
Viewing GPU Threads
CUDA Thread IDs and Coordinate Spaces
Single-Stepping GPU Code
Halting a Running Application
Displaying CUDA Program Elements
GPU Assembler Display
GPU Variable and Data Display
Managed Memory Variables
About Managed Memory
How TotalView Displays Managed Variables
CUDA Built-In Runtime Variables
Type Casting
PTX Registers
Enabling CUDA Memory Checker Feature
GPU Core Dump Support
GPU Error Reporting
CUDA Problems and Limitations
Hangs or Initialization Failures
CUDA and ReplayEngine
Sample CUDA Program
PART V Appendices
Appendix A More on Expressions
Calling Functions: Problems and Issues
Using Built-in Variables and Statements
Using TotalView Variables
Using Built-In Statements
Using Programming Language Elements
Using C and C++
Using Fortran
Fortran Statements
Fortran Intrinsics
Appendix B Compiling for Debugging
Compiling with Debugging Symbols
Maintaining Debug Information Separate from an Executable
Controlling Separate Debug Files
Searching for the Debug Files
Appendix C Platform-Specific Topics
Swap Space
Shared Libraries
Changing Linkage Table Entries and LD_BIND_NOW
Linking with the dbfork Library
Linux or Mac OS X
Appendix D Resources
Classic TotalView Documentation
Conventions
Contacting Us
TotalView Reference Guide
PART I Using the CLI
CLI Command Summary
CLI Commands
Commands by Category
General CLI Commands
CLI Initialization and Termination Commands
Program Information Commands
Execution Control Commands
Action Points
Platform-Specific CLI Commands
Other Commands
All Commands
alias
capture
dactions
dassign
dattach
dbarrier
dbreak
dcache
dcalltree
dcheckpoint
dcont
dcuda
ddelete
ddetach
ddisable
ddlopen
ddown
denable
dexamine
dflush
dfocus
dga
dgo
dgroups
dhalt
dheap
dhistory
dhold
dkill
dlappend
dlist
dload
dmstat
dnext
dnexti
dout
dprint
dptsets
drerun
drestart
drun
dsession
dset
dskip
dstacktransform
dstatus
dstep
dstepi
dunhold
dunset
duntil
dup
dwait
dwatch
dwhat
dwhere
dworker
exit
help
quit
spurs
stty
unalias
CLI Namespace Commands
Commands by Category
Accessor Functions
Helper Functions
All Commands
actionpoint
dec2hex
dll
errorCodes
expr
focus_groups
focus_processes
focus_threads
group
hex2dec
process
read_symbols
respond
scope
source_process_startup
symbol
thread
type
type_transformation
Batch Debugging Using tvscript
About tvscript
tvscript Command Syntax
tvscript Options
tvscript External Script Files
Logging Functions API
Process Functions API
Thread Functions API
Action Point API
Event API
Example tvscript Script File
TotalView Variables
Top-Level (::) Namespace
TV:: Namespace
TV::MEMDEBUG:: Namespace
TV::GUI:: Namespace
PART II Transformations
Creating Type Transformations
About the Type Transformation Facility
Why Type Transformations
Creating Structure and Class Transformations
Transforming Structures
build_struct_transform Function
Type Transformation Expressions
Using Type Transformations
C++View
Writing a Data Display Function
TV_ttf_type_ascii_string
TV_ttf_type_int
Templates
Precedence - Searching for TV_ttf_display_type
TV_ttf_add_row
TV_ttf_ec_ok
TV_ttf_ec_not_ active
TV_ttf_ec_invalid_characters
TV_ttf_ec_buffer_exhausted
Return values from TV_ttf_display_type
TV_ttf_format_ok
TV_ttf_format_ok_elide
TV_ttf_format_ failed
TV_ttf_format_ raw
TV_ttf_format_ never
Elision
Other Constraints
Safety
Memory Management
Multithreading
Tips and Tricks
Core Files
Using C++View with ReplayEngine
C
Compiling and linking tv_data_display.c
C++View Example Files
Limitations
Licensing
PART III Running TotalView
TotalView Command Syntax
Command-Line Syntax
Command-Line Options
PART IV Platforms and Operating Systems
Platforms and Compilers
Compiling with Debugging Symbols
Apple Running Mac OS X
IBM AIX on RS/6000 Systems
IBM Blue Gene
IBM Power Linux
Linux Running on an x86 Platform
Linux Running on an x86-64 Platform
Linux Running on an ARM64 Platform
Sun Solaris
Maintaining Debug Information Separate from an Executable
Controlling Separate Debug Files
Searching for the Debug Files
Linking with the dbfork Library
dbfork on IBM AIX on RS/6000 Systems
Linking C++ Programs with dbfork
Linux or Mac OS X
SunOS 5 SPARC
Compiling and Linking Split DWARF
Using GNU DebugFission Split DWARF on Linux
Using Split DWARF on Solaris
Operating Systems
Supported Operating Systems
Troubleshooting Mac OS X Installations
Mounting the /proc File System
Mounting /proc with SunOS 5
Swap Space
Swap Space on IBM AIX
Swap Space on Linux
Swap Space on SunOS 5
Shared Libraries
Changing Linkage Table Entries and LD_BIND_NOW
Debugging Your Program’s Dynamically Loaded Libraries
dlopen Options for Scalability
Filtering dlopen Events
Handling dlopen Events in Parallel
Known Limitations
Remapping Keys
Expression System
Expression System on IBM AIX-Power and Blue Gene/Q
Architectures
AMD and Intel x86-64
x86-64 General Registers
x86-64 Floating-Point Registers
x86-64 FPCR Register
Using the x86-64 FPCR Register
x86-64 FPSR Register
x86-64 MXCSR Register
Power Architectures
Power General Registers
Blue Gene Power Registers
Blue Gene/Q QPX Floating-Point Registers
Power MSR Register
Power Floating-Point Registers
Power FPSCR Register
Using the Power FPSCR Register
ARM64
ARM64 General Registers
ARM64 Floating-Point Registers
ARM64 FPCR Register
ARM64 FPSR Register
Intel x86
Intel x86 General Registers
Intel x86 Floating-Point Registers
Intel x86 FPCR Register
Using the Intel x86 FPCR Register
Intel x86 FPSR Register
Intel x86 MXCSR Register
Sun SPARC
SPARC General Registers
SPARC PSR Register
SPARC Floating-Point Registers
SPARC FPSR Register
Using the SPARC FPSR Register
TotalView Installation Guide
Installing TotalView
Installing TotalView
Cray XT Installation
Installing
About the Install
Mac OS X Installations
Configuring Your Environment
Installing a Demo License
FlexNet Publisher (FNP) Demo License Installation
FlexNet Embedded (FNE) Demo License Installation
Installing a FlexNet Publisher (FNP) License
Updating an Existing License
FlexNet Update
No FlexNet Update
Installing a New License
Installing a Flex-Net Embedded (FNE) License
Prerequisites
Installing a FlexNet Embedded (FNE) Node-Locked License
License Location
License Installation
Renewing a Node-Locked License
Installing a FlexNet Embedded (FNE) License Server
1. Install the FNE Server
2. Generate a Host ID for Each License Server
3. Configure Hosts to Use the FNE Server
Installing a Failover Server
Renewing a Served License
License Reservation
Managing License Reservations
Uninstalling an FNE License Server
Installing or Changing a Regular FlexNet Publisher (FNP) License
Is My License Valid for this Release?
Planning for a Regular License
Choosing a License Server
Finding a License Server Host ID
Creating a FlexNet User Account
Installing your Permanent License
Accessing a License Server
Starting the License Manager Manually
Stopping the License Manager
Testing the License Manager
Starting the License Manager When the System Boots
Installing FlexNet Standalone
Starting the Install Program
More FlexNet Information
Installing FlexNet Publisher License Keys
Step 1: Locating the FlexNet License Server Directory
Step 2: Deleting Older License Keys
Step 3: Adding New License Keys
Adding a New license.dat File
Revising Your license.dat File
Step 4: Starting and Stopping the License Server
Step 5: Configuring TotalView to Use the License Server
Reporting Problems with License Installation
Uninstalling TotalView
The Three-Step Removal Process
ReplayEngine User Guide
About ReplayEngine
How ReplayEngine Works
Play It Backwards
The Process of Recording and Playback
System Resource Issues
Replaying Your Program
Threads and Processes
Attaching to Running Programs
Saving and Loading the Execution History
Using ReplayEngine
Enabling and Disabling ReplayEngine
Enabling ReplayEngine at Program Load
Enabling and Disabling ReplayEngine for a Loaded Program
Enabling Replay
Disabling Replay
Examining Program State and History
Replay Bookmarks
Creating bookmarks
Activating bookmarks
Setting Preferences for ReplayEngine
CLI Support
Known Issues and Limitations
Performance Issues
TotalView 2020.1
Performance Issues
GettingStartedPartIntro
GettingStartedFullInterface
GettingStartedCustomizingInterface
Preferences
GettingStartedPreferences
GettingStartedResizing
GettingStartedDockingUndocking
GettingStartedSourceViews
GettingStartedToolbars
GettingStartedLogger
GettingStartedStartPage
SessionsOverview
SessionsSettingUp
SessionsLoadingfromEditor
SessionsStarting
SessionsDebugaProgram
SessionsDebugParallel
SessionsAttachtoProcess
SessionsAttachtoProcessFields
SessionsAttachtoProcessSearch
SessionsAttachtoProcessDebugOptions
SessionsCoreorReplayFile
SessionsLoadRecent
SessionsEditPrevious
SessionsLoadingProgramwithCLI
SessionsOptionsReverseDebugging
SessionsProgramEnvironment
SessionsStandardInandOut
SessionsManagingSessions
SessionsManageSessionsWindow
SessionsManagerDisplaySessionData
SessionsManagerIconsDescription
BasicDebugging
LookupView
OrganizingProcessesandThreads
TheProcessesandThreadsView
ProcessesandThreadsTreeView
ProcessesandThreadsInteractions
DebuggingCommandWidth
ProcessesandThreadsAttributes
AttributeListButtons
ActionPoints
ActionPointProperties
ActionPointsSetting
ActionPointsWidth
ActionPointsManagingandDiving
ActionPointsSavingToaFileUsingTheCLI
CallStackandData
LocalVariablesView
CallStackandDataViewing
DataViewGeneral
DataViewEditingExpressions
DataViewValue
DataViewCasting
ArrayStatistics
CommandViewandCLI
ReverseConnect
CudaToolbars
TypeTransformations
AboutReplayEngine
ReplayingYourProgram
UsingReplayEngine
ReplayBookmarks
ReplayEngineKnownIssues