• New in TotalView 2021.2
• TotalView 2021.2 Supported Platforms
  • Versions
  • Support Notes
  • Platform Support
• TotalView User Guide
  • PART I An Introduction to TotalView
    • Getting Started
      • Introducing TotalView
      • An Initial Look at the Interface
        • Customizing the Interface
          • Preferences
          • Resizing
          • Drawers
          • Undocking and Docking
        • A Tour of the Interface
          • Central Area
          • Toolbars
          • Processes and Threads View
          • Call Stack View and Local Variables View
          • Data View
          • Lookup View
          • Action Points, CLI, and Logger Views
          • Input/Output View
          • Help
      • Starting TotalView and Creating a Debugging Session
        • Debugging Commands
    • Creating and Managing Sessions
      • Setting up Debugging Sessions
        • Loading Programs from the Session Editor
          • Starting a Debugging Session
          • Debug a Program
          • Debug a Parallel Program
          • Attach to Process
          • Debug a Core or Replay Recording File
          • Load a Recent Session
          • Editing a Previous Session
        • Loading Programs Using the CLI
      • Options and Program Arguments
        • Options: Reverse Debugging
        • Program Environment: Environment Variables
        • Standard Input and Output
        • Modifying Arguments in an Open Session
      • Managing Sessions
      • Starting a Session from your Shell
        • Starting TotalView on a Script
    • Basic Debugging
      • Program Load and Navigation
        • Load the Program to Debug
          • Initial Display
        • Program Navigation
      • Stepping and Executing
        • Simple Stepping
      • Setting and Running to a Breakpoint (Action Point)
        • Set and Control Breakpoints
        • Run Your Program and Observe the Call Stack
      • Examining Data
        • Viewing Variables in the Local Variables View
        • Viewing Variables in the Data View
          • Watching Data Values Update
      • Moving On
    • Program Navigation
      • Navigating from within the Source Pane
      • Highlighting a String and the Find Function
      • The Lookup File or Function View
      • The Documents View
  • PART II Debugging Tools and Tasks
    • Setting and Managing Action Points (Breakpoints)
      • About Action Points
      • Breakpoints
        • Setting Source-Level Breakpoints
          • Sliding Breakpoints
        • Breakpoints at a Specific Location
        • Pending Breakpoints
          • Pending Breakpoints on a Function
          • Pending Breakpoints on a Line Number
          • Conflicting Breakpoints
        • Breakpoints at Execution
        • Modifying a Breakpoint
        • Setting Breakpoints When Using the fork()/execve() Functions
          • Debugging Processes That Call the fork() Function
          • Debugging Processes that Call the execve() Function
          • Example: Multi-process Breakpoint
      • Evalpoints
        • Setting an Evalpoint
        • Creating a Pending Evalpoint
        • Modifying an Evalpoint
        • Creating Conditional Breakpoints
        • Patching Programs
          • Branching Around Code
          • Adding a Function Call
          • Correcting Code
          • Using Programming Language Constructs
      • Watchpoints
        • Creating Watchpoints
          • Displaying, Deleting, or Disabling Watchpoints
        • Modifying Watchpoints
        • Watching Memory
        • Triggering Watchpoints
          • Using Multiple Watchpoints
          • Performance Impact of Copying Previous Data Values
        • Using Watchpoint Expressions
        • Using Watchpoints on Different Architectures
      • Barrier Points
        • About Barrier Breakpoint States
        • Setting a Barrier Breakpoint
        • Creating a Satisfaction Set
        • Hitting a Barrier Point
        • Releasing Processes from Barrier Points
        • Changing Settings and Disabling a Barrier Point
        • Using Barrier Points
          • Barrier Point Illustration
      • Controlling an Action Point’s Width
        • About an Action Point’s Width: Group, Process or Thread
        • Setting the Action Point’s Width
        • Action Point Width and Process/Thread State
      • Managing and Diving on Action Points
        • Sorting
        • Diving
        • Deleting, Disabling, and Suppressing
        • Saving and Loading Action Points
      • More on Action Points Using the CLI
        • Saving Action Points to a File Using the CLI
        • Suppressing and Unsuppressing Action Points
    • Examining and Editing Data
      • Viewing Data in TotalView
      • About Expressions
        • Using C++
      • The Call Stack and Local Variables Views
        • The Call Stack View
        • The Local Variables View
        • Viewing Call Stack Data
        • Viewing Data in Fortran
          • Viewing Modules and Their Data
          • Common Blocks
          • Fortran 90 User-Defined Types
          • Fortran 90 Deferred Shape Array Types
          • Fortran 90 Pointer Types
          • Fortran Parameters
      • The Data View
        • Adding Variables to the Data View
          • Add to the Data View from the Local Variables View
          • Move an Expression or Variable from the Source View to the Data View
          • Creating a New Expression by Diving on a Variable in the Data View
          • Entering a New Expression into the Data View
        • Editing an Expression
          • Dereferencing a Pointer
          • Changing the Value of Data
          • Casting to Another Type
        • Displaying Arrays
        • Viewing Individual Elements in an Array of Structures
          • Displaying a Fortran Structure
          • Displaying a C++ Structure
          • Using Dive In All to Deference a Pointer in an Array of Pointers
        • Controlling STL Data Transformation
        • Customizing the Data View
          • Focusing on Specific Data
          • The Data View Drawer
      • Viewing Array Statistics
        • Slicing Arrays
        • Array Statistics Detail
      • Using the CLI to Examine Data
        • Changing the Display of Data
        • Displaying Variables
    •  Working with Processes and Threads
      • The Processes and Threads View
      • How the Processes and Threads View Works
        • Debugging Command Width
          • Understanding Thread Width
          • Understanding Process Width
          • Understanding Group Width
      • Controlling Processes and Threads in a Running Session
        • Controlling Processes by Detaching from the Debugger
          • Process > Detach
          • Group > Detach
        • Program Execution and Share Groups
        • Synchronizing Processes and Threads Using Run To
        • Breakpoint Width Impact on Execution
        • Setting the Program Counter
      • Displaying a Thread Name
        • Thread Names in the UI
        • Thread Properties
        • Thread Options on dstatus
      • Process and Thread Attributes
    • Debugging Python
      • Overview
      • Python Debugging Requirements
        • Python Version
        • Limitations and Extensions:
      • Starting a Python Debugging Session
      • Debugging Python and C/C++ with TotalView
        • Transforming the Stack
      • Viewing and Comparing Python and C/C++ Variables
      • Leveraging Other Debugging Technologies for Python Debugging
      • Supported Python Extension Technologies for Stack Transformations
    • Using the Command Line Interface (CLI)
      • Access to the CLI
      • Introduction to the CLI
      • About the CLI and Tcl
        • Integration of the CLI and the UI
        • Invoking CLI Commands
      • Starting the CLI in a Terminal Window
        • Startup Example
        • Starting Your Program
      • About CLI Output
        • ‘more’Processing
      • Using Command Arguments
      • Using Namespaces
      • About the CLI Prompt
      • Using Built-in and Group Aliases
      • How Parallelism Affects Behavior
        • Types of IDs
      • Controlling Program Execution
        • Advancing Program Execution
        • Using Action Points
      • Examples of Using the CLI
        • Setting the CLI EXECUTABLE_PATH Variable
        • Initializing an Array Slice
        • Printing an Array Slice
        • Writing an Array Variable to a File
        • Automatically Setting Breakpoints
    • Reverse Connections
      • About Reverse Connections
        • Reverse Connection Environment Variables
          • TV_REVERSE_CONNECT_DIR
          • TV_CONNECT_OPTIONS
      • Starting a Reverse Connect Session
        • Listening for Reverse Connections
      • Reverse Connect Examples
        • CLI Example
        • MPI Batch Script Example
      • Troubleshooting Reverse Connections
        • Stale Files in the Reverse Connect Directory
        • Directory Permissions
        • User ID Issues
        • Reverse Connect Directory Environment Variable
    • Preferences
      • About Preferences
      • Action Points
      • Display Settings
      • Tool Bar
      • Search Path
      • Parallel Configuration
      • Remote Connection Settings
  • PART III Parallel Debugging
    • About Parallel Debugging in TotalView
      • Program Execution Models
      • View Process and Thread State
      • Control Program Execution
        • Using Groups
        • Synchronizing Execution with Barrier Points
    • Debugging OpenMP Applications
      • OpenMP and the OMPD API
        • OMPD Requirements
      • OpenMP Setup and Configuration
        • Enabling OpenMP Debugging
        • Enabling Stack Filtering
      • Running Your Program
        • The Call Stack
        • The OpenMP View
      • Hybrid Programming: Combining OpenMP with MPI
    • Controlling fork, vfork, and execve Handling
      • The exec_handling and fork_handling Command Options and State Variables
        • Exec Handling
        • Fork Handling
        • Example
  • PART IV Accessing TotalView Remotely
    • TotalView Remote Connections
      • About Remote Connections
      • Configuring a Remote Connection
      • Debugging on a Remote Connection
    • TotalView Remote Display
      • Remote Display Supported Platforms
      • Remote Display Components
      • Installing the Client
        • Installing on Linux
        • Installing on Microsoft Windows
        • Installing on macOS
      • Client Session Basics
        • Working on the Remote Host
      • Advanced Options
      • Naming Intermediate Hosts
      • Submitting a Job to a Batch Queuing System
      • Setting Up Your Systems and Security
      • Session Profile Management
      • Batch Scripts
        • tv_PBS.csh Script
        • tv_LoadLeveler.csh Script
  • PART V Using the CUDA Debugger
    • About the TotalView CUDA Debugger
      • Overview
      • Installing the CUDA SDK Tool Chain
      • Directive-Based Accelerator Programming Languages
    • CUDA Debugging Model and Unified Display
      • The TotalView CUDA Debugging Model
      • Pending and Sliding Breakpoints
      • Unified Source View and Breakpoint Display
    • CUDA Debugging Tutorial
      • Compiling for Debugging
        • Compiling for Fermi
        • Compiling for Fermi and Tesla
        • Compiling for Kepler
        • Compiling for Pascal
        • Compiling for Volta
      • Starting a TotalView CUDA Session
      • Controlling Execution
        • Viewing GPU Threads
          • CUDA Thread IDs and Coordinate Spaces
        • Single-Stepping GPU Code
        • Halting a Running Application
      • Displaying CUDA Program Elements
        • GPU Assembler Display
        • GPU Variable and Data Display
        • Managed Memory Variables
          • About Managed Memory
          • How TotalView Displays Managed Variables
        • CUDA Built-In Runtime Variables
        • Type Casting
        • PTX Registers
      • Enabling CUDA Memory Checker Feature
      • GPU Core Dump Support
      • GPU Error Reporting
    • CUDA Problems and Limitations
      • Hangs or Initialization Failures
      • CUDA and ReplayEngine
    • Sample CUDA Program
  • PART VI Memory Debugging
    • About TotalView Memory Debugging
      • Debugging Memory in TotalView
      • About Program Memory
      • How TotalView Intercepts Memory Data
      • Your Program’s Data
        • The Data Section
        • The Stack
        • The Heap
          • Finding Heap Allocation Problems
          • Finding Heap Deallocation Problems
          • realloc() Problems
          • Memory Leaks
    • Running a Memory Debugging Session
      • Starting Memory Debugging in TotalView
      • Memory Leak Detection
        • Using the Leak Report
          • Updating the Leak Report
          • MPI Programs and Leak Reports
      • Memory Heap Reports
        • Using the Heap Report
          • Updating the Heap Report
      • Dangling Pointer Problems
        • Dangling Pointers in the Local Variables and Data Views
    • Memory Scripting
      • display_specifiers Command-Line Option
      • event_action Command-Line Option
      • Other Command Line Options
      • memscript Example
    • Preparing Programs for Memory Debugging
      • Compiling Programs for Memory Debugging
      • Linking Your Application with the Agent
      • Using env to Insert the Agent
      • Installing tvheap_mr.a on AIX
        • LIBPATH and Linking
      • Using TotalView in Selected Environments
        • MPICH
        • IBM PE
        • Mac OS
          • Background
          • Calls to system() on Mac OS
          • Setting the Environment Variable TV_MACOS_SYSTEM
        • Linux
          • dlopen and RTLD_DEEPBIND
  • PART VII Appendices
    • Appendix A More on Expressions
      • Calling Functions: Problems and Issues
      • Using Built-in Variables and Statements
        • Using TotalView Variables
        • Using Built-In Statements
      • Using Programming Language Elements
        • Using C and C++
        • Using Fortran
          • Fortran Statements
          • Fortran Intrinsics
    • Appendix B Compiling for Debugging
      • Compiling with Debugging Symbols
      • Maintaining Debug Information Separate from an Executable
        • Controlling Separate Debug Files
        • Searching for the Debug Files
    • Appendix C Platform-Specific Topics
      • Swap Space
      • Shared Libraries
        • Changing Linkage Table Entries and LD_BIND_NOW
        • Linking with the dbfork Library
          • Linux or Mac OS X
    • Appendix D Resources
      • Classic TotalView Documentation
      • Conventions
      • Contacting Us
• TotalView Reference Guide
  • PART I Using the CLI
    • CLI Command Summary
    • CLI Commands
      • Commands by Category
        • General CLI Commands
        • CLI Initialization and Termination Commands
        • Program Information Commands
        • Execution Control Commands
        • Action Points
        • Platform-Specific CLI Commands
        • Other Commands
      • All Commands
        • alias
        • capture
        • dactions
        • dassign
        • dattach
        • dbarrier
        • dbreak
        • dcache
        • dcalltree
        • dcheckpoint
        • dcont
        • dcuda
        • ddelete
        • ddetach
        • ddisable
        • ddlopen
        • ddown
        • denable
        • dexamine
        • dflush
        • dfocus
        • dga
        • dgo
        • dgroups
        • dhalt
        • dheap
        • dhistory
        • dhold
        • dkill
        • dlappend
        • dlist
        • dload
        • dmstat
        • dnext
        • dnexti
        • domp
        • dout
        • dprint
        • dptsets
        • drerun
        • drestart
        • drun
        • dsession
        • dset
        • dskip
        • dstacktransform
        • dstatus
        • dstep
        • dstepi
        • dunhold
        • dunset
        • duntil
        • dup
        • dwait
        • dwatch
        • dwhat
        • dwhere
        • dworker
        • exit
        • help
        • quit
        • spurs
        • stty
        • unalias
    • CLI Namespace Commands
      • Commands by Category
        • Accessor Functions
        • Helper Functions
      • All Commands
        • actionpoint
        • dec2hex
        • dll
        • errorCodes
        • expr
        • focus_groups
        • focus_processes
        • focus_threads
        • group
        • hex2dec
        • process
        • read_symbols
        • respond
        • scope
        • source_process_startup
        • symbol
        • thread
        • type
        • type_transformation
    • Batch Debugging Using tvscript
      • About tvscript
      • tvscript Command Syntax
      • tvscript Options
      • tvscript External Script Files
        • Logging Functions API
        • Process Functions API
        • Thread Functions API
        • Action Point API
        • Event API
        • Example tvscript Script File
    • TotalView Variables
      • Top-Level (::) Namespace
      • TV:: Namespace
      • TV::MEMDEBUG:: Namespace
      • TV::GUI:: Namespace
  • PART II Transformations
    • Creating Type Transformations
      • About the Type Transformation Facility
      • Why Type Transformations
      • Creating Structure and Class Transformations
        • Transforming Structures
        • build_struct_­transform ­Function
        • Type Transformation Expressions
        • Using Type Transformations
      • C++View
        • Writing a Data Display Function
          • TV_ttf_type_ascii_string
          • TV_ttf_type_int
        • Templates
        • Precedence - Searching for TV_ttf_display_type
        • TV_ttf_add_row
          • TV_ttf_ec_ok
          • TV_ttf_ec_not_ active
          • TV_ttf_ec_invalid_characters
          • TV_ttf_ec_buffer_exhausted
        • Return values from TV_ttf_display_type
          • TV_ttf_format_ok
          • TV_ttf_format_ok_elide
          • TV_ttf_format_ failed
          • TV_ttf_format_ raw
          • TV_ttf_format_ never
        • Elision
        • Other Constraints
        • Safety
        • Memory Management
        • Multithreading
        • Tips and Tricks
        • Core Files
        • Using C++View with ReplayEngine
        • C
        • Compiling and linking tv_data_display.c
        • C++View Example Files
        • Limitations
        • Licensing
  • PART III Running TotalView
    • TotalView Command Syntax
      • Command-Line Syntax
      • Command-Line Options
  • PART IV Platforms and Operating Systems
    • Platforms and Compilers
      • Compiling with Debugging Symbols
        • Apple Running macOS
        • IBM AIX on RS/6000 Systems
        • IBM Power Linux
        • Linux Running on an x86 Platform
        • Linux Running on an x86-64 Platform
        • Linux Running on an ARM64 Platform
        • Sun Solaris
      • Maintaining Debug Information Separate from an Executable
        • Controlling Separate Debug Files
        • Searching for the Debug Files
      • Linking with the dbfork Library
        • dbfork on IBM AIX on RS/6000 Systems
          • Linking C++ Programs with dbfork
        • Linux or macOS
        • SunOS 5 SPARC
      • Compiling and Linking Split DWARF
        • Using GNU DebugFission Split DWARF on Linux
        • Using Split DWARF on Solaris
    • Operating Systems
      • Supported Operating Systems
      • Troubleshooting macOS Installations
      • Mounting the /proc File System
        • Mounting /proc with SunOS 5
      • Swap Space
        • Swap Space on IBM AIX
        • Swap Space on Linux
        • Swap Space on SunOS 5
      • Shared Libraries
        • Changing Linkage Table Entries and LD_BIND_NOW
      • Debugging Your Program’s Dynamically Loaded Libraries
        • dlopen Options for Scalability
          • Filtering dlopen Events
          • Handling dlopen Events in Parallel
        • Known Limitations
      • Remapping Keys
    • Architectures
      • AMD and Intel x86-64
        • x86-64 General Registers
        • x86-64 Floating-Point Registers
        • x86-64 FPCR Register
        • x86-64 FPSR Register
        • x86-64 MXCSR Register
      • Power Architectures
        • Power General Registers
        • Power MSR Register
        • Power Floating-Point Registers
        • Power FPSCR Register
      • ARM64
        • ARM64 General Registers
        • ARM64 Floating-Point Registers
        • ARM64 FPCR Register
        • ARM64 FPSR Register
      • Intel x86
        • Intel x86 General Registers
        • Intel x86 Floating-Point Registers
        • Intel x86 FPCR Register
        • Intel x86 FPSR Register
        • Intel x86 MXCSR Register
      • Sun SPARC
        • SPARC General Registers
        • SPARC PSR Register
        • SPARC Floating-Point Registers
        • SPARC FPSR Register
          • Using the SPARC FPSR Register
• TotalView Installation Guide
  • Introduction
    • Installation Overview
      • TotalView Distribution
      • TotalView Installation and Directory Hierarchy
      • TotalView Licensing
  • Evaluation Installations
    • Installing TotalView
    • FlexNet Publisher (FNP) Demo License Installation
    • FlexNet Embedded (FNE) Demo License Installation
  • Transitioning to a Licensed Install
  • TotalView for HPC
    • Overview
    • Install TotalView for HPC
    • Install a TotalView for HPC License Server
      • Choosing a License Server
      • Installing a FlexNet Publisher (FNP) License Server
        • Prerequisites
        • Installing an FNP License Server
        • Starting and Stopping the License Server
      • Installing a FlexNet Embedded (FNE) License Server
        • Prerequisites
        • Installing a Regular FNE Server
        • Managing License Access with Model Definitions
        • Installing a Failover Server
        • Uninstalling an FNE License Server
  • Developer / Developer for HPC
    • Install TotalView
    • Find the Host ID
    • Verify Your License Type and Product
    • Install a New Developer / Developer for HPC License
      • FNP: Install a New Developer / Developer for HPC License
      • FNE: Install a New Developer / Developer for HPC License
  • TotalView Student
  • Updating or Renewing a License
    • FNP License Renewal
      • TotalView Developer / Developer for HPC FNP License Renewal
      • TotalView for HPC FNP License Renewal
        • Deleting Older License Keys
    • FNE License Renewal
      • TotalView Developer / Developer for HPC FNE License Renewal
      • TotalView for HPC FNE License Renewal
  • Update License Software
    • Updating FlexNet for FNP
    • Updating FlexNet for FNE
  • Appendix
    • OS-Specific Installation Info
      • Cray Installations
        • Installing
        • About the Install
      • macOS Installations
• ReplayEngine User Guide
  • About ReplayEngine
    • How ReplayEngine Works
      • Play It Backwards
      • The Process of Recording and Playback
    • System Resource Issues
    • Replaying Your Program
    • Threads and Processes
    • Attaching to Running Programs
    • Saving and Loading the Execution History
  • Using ReplayEngine
    • Enabling and Disabling ReplayEngine
      • Enabling ReplayEngine at Program Load
      • Enabling and Disabling ReplayEngine for a Loaded Program
        • Enabling Replay
        • Disabling Replay
    • Examining Program State and History
    • Replay Bookmarks
      • Creating bookmarks
      • Activating bookmarks
    • Setting Preferences for ReplayEngine
    • CLI Support
  • Known Issues and Limitations
    • Performance Issues

TotalView 2021.2

Performance Issues

GettingStartedPartIntro

GettingStartedFullInterface

GettingStartedCustomizingInterface

Preferences

GettingStartedPreferences

GettingStartedResizing

GettingStartedDockingUndocking

GettingStartedSourceViews

GettingStartedToolbars

GettingStartedLogger

InputOutputView

GettingStartedStartPage

SessionsOverview

SessionsSettingUp

SessionsLoadingfromEditor

SessionsStarting

SessionsDebugaProgram

SessionsDebugParallel

SessionsAttachtoProcess

SessionsAttachtoProcessFields

SessionsAttachtoProcessSearch

SessionsAttachtoProcessDebugOptions

SessionsCoreorReplayFile

SessionsLoadRecent

SessionsEditPrevious

SessionsLoadingProgramwithCLI

SessionsOptionsReverseDebugging

SessionsProgramEnvironment

SessionsStandardInandOut

SessionsManagingSessions

SessionsManageSessionsWindow

SessionsManagerDisplaySessionData

SessionsManagerIconsDescription

BasicDebugging

LookupView

ActionPoints

ActionPointProperties

ActionPointsSetting

ActionPointsWidth

ActionPointsManagingandDiving

ActionPointsSavingToaFileUsingTheCLI

CallStackandData

LocalVariablesView

CallStackandDataViewing

DataViewGeneral

DataViewEditingExpressions

DataViewValue

DataViewCasting

ArrayStatistics

OrganizingProcessesandThreads

TheProcessesandThreadsView

ProcessesandThreadsTreeView

ProcessesandThreadsInteractions

DebuggingCommandWidth

ProcessesandThreadsAttributes

AttributeListButtons

CommandViewandCLI

ReverseConnect

RemoteConnections

CudaToolbars

MemoryDebugLinkReport

MemoryDebugHeapReport

Installingtvheap_mraonAIX

LIBPATHandLinking

OtherTopics

TypeTransformations

AboutReplayEngine

ReplayingYourProgram

UsingReplayEngine

ReplayBookmarks

ReplayEngineKnownIssues