Web Search

TotalView User Guide

About This Guide

TotalView Family Differences

Using the CLI

Audience

Conventions

TotalView Documentation

Contacting Us

Introduction

A Tour of TotalView

Basic Debugging with TotalView

TotalView Action Points vs. Print Statements

Examining Data

Simplifying Array Display

Seeing Groups of Variables

Setting Watchpoints

Debugging Multi-process and Multi-threaded Programs

Program Using Almost Any Execution Model

Supporting Multi-process and Multi-threaded Programs

Using Groups and Barriers

Memory Debugging

Introducing the CLI

What’s Next

About Threads, Processes, and Groups

A Couple of Processes

Threads

Complicated Programming Models

Types of Threads

Organizing Chaos

How TotalView Creates Groups

Simplifying What You’re Debugging

Setting Up

Accessing TotalView Remotely

Remote Display Supported Platforms

Remote Display Components

Installing the Client

Installing on Linux

Installing on Microsoft Windows

Installing on Apple Mac OS X Intel

Client Session Basics

Working on the Remote Host

Advanced Options

Naming Intermediate Hosts

Submitting a Job to a Batch Queuing System

Setting Up Your Systems and Security

Session Profile Management

Batch Scripts

tv_PBS.csh Script

tv_LoadLeveler.csh Script

Setting Up a Debugging Session

Compiling Programs

Using File Extensions

Starting TotalView

Initializing TotalView

Exiting from TotalView

Loading Programs

Loading Programs Using the GUI

Loading Programs Using the CLI

Attaching to Processes

Detaching from Processes

Examining Core Files

Setting Command-line Arguments and Environment Variables

Altering Standard I/O

Viewing Process and Thread States

Seeing Attached Process States

Seeing Unattached Process States

Handling Signals

Setting Search Paths

Setting Startup Parameters

Setting Preferences

Setting Preferences, Options, and X Resources

Setting Up Remote Debugging Sessions

Platform Issues when Remote Debugging

Automatically Launching a Process on a Remote Server

Troubleshooting Server Autolaunch

Changing the Remote Shell Command

Changing Arguments

Autolaunching Sequence

Starting the TotalView Server Manually

TotalView Server Launch Options and Commands

Setting Bulk Launch Window Options

Customizing Server Launch Commands

Setting the Single-Process Server Launch Command

Setting the Bulk Server Launch Command

Debugging Over a Serial Line

Starting the TotalView Debugger Server

Setting Up MPI Debugging Sessions

Debugging MPI Programs

Starting MPI Programs Using File > New Program

Debugging MPICH Applications

Starting TotalView on an MPICH Job

Attaching to an MPICH Job

Using MPICH P4 procgroup Files

Debugging MPICH2 Applications

Downloading and Configuring MPICH2

Starting TotalView Debugging on an MPICH2 Hydra Job

Starting TotalView Debugging on an MPICH2 MPD Job

Starting the MPI MPD Job with MPD Process Manager

Starting an MPICH2 MPD Job

Starting MPI Issues

MPI Rank Display

Displaying the Message Queue Graph Window

Displaying the Message Queue

About the Message Queue Display

Using Message Operations

Diving on MPI Processes

Diving on MPI Buffers

About Pending Receive Operations

About Unexpected Messages

About Pending Send Operations

Debugging Cray MPI Applications

Debugging IBM MPI Parallel Environment (PE) Applications

Preparing to Debug a PE Application

Using Switch-Based Communications

Performing a Remote Login

Setting Timeouts

Starting TotalView on a PE Program

Setting Breakpoints

Starting Parallel Tasks

Attaching to a PE Job

Attaching from a Node Running poe

Attaching from a Node Not Running poe

Debugging IBM Blue Gene Applications

Debugging LAM/MPI Applications

Debugging QSW RMS Applications

Starting TotalView on an RMS Job

Attaching to an RMS Job

Debugging SiCortex MPI Applications

Debugging SGI MPI Applications

Starting TotalView on an SGI MPI Job

Attaching to an SGI MPI Job

Using ReplayEngine with SGI MPI

Debugging Sun MPI Applications

Attaching to a Sun MPI Job

Using ReplayEngine with Infiniband MPIs

Debugging Parallel Applications Tips

Attaching to Processes Tips

Parallel Debugging Tips

MPICH Debugging Tips

IBM PE Debugging Tips

Setting Up Parallel Debugging Sessions

Debugging OpenMP Applications

Debugging OpenMP Programs

About TotalView OpenMP Features

About OpenMP Platform Differences

Viewing OpenMP Private and Shared Variables

Viewing OpenMP THREADPRIVATE Common Blocks

Viewing the OpenMP Stack Parent Token Line

Using SLURM

Debugging IBM Cell Broadband Engine Programs

The PPU

The SPU

Cell Programing

PPU and SPU Executable Organization

PPU and SPU Executable Naming

Thread IDs

Breakpoints

Registers, Unions, and Casting

Debugging Cray XT Applications

Cray XT Catamount

Configuring Cray XT for TotalView

Using TotalView with your Cray XT System

Cray XT CNL

Debugging SiCortex Applications

Installation Notes

Using TotalView on SiCortex

MPI Debugging

Debugging Global Arrays Applications

Debugging PVM (Parallel Virtual Machine) Applications

Supporting Multiple Sessions

Setting Up ORNL PVM Debugging

Starting an ORNL PVM Session

Automatically Acquiring PVM Processes

Attaching to PVM Tasks

About Reserved Message Tags

Cleaning Up Processes

Debugging Shared Memory (SHMEM) Code

Debugging UPC Programs

Invoking TotalView

Viewing Shared Objects

Displaying Pointer to Shared Variables

Using the GUI

Using TotalView Windows

Using Mouse Buttons

Using the Root Window

Viewing Remote Processes

Viewing Data Linearly or Hierarchically

Controlling the Display of Processes and Threads

Using the Process Window

Viewing the Assembler Version of Your Code

Diving into Objects

Resizing and Positioning Windows and Dialog Boxes

Editing Text

Saving the Contents of Windows

Visualizing Programs and Data

Displaying Call Graphs

Parallel Backtrace View

Visualizing Array Data

Command Summary

How the Visualizer Works

Viewing Data Types in the Visualizer

Viewing Data

Visualizing Data Manually

Using the Visualizer

Using Dataset Window Commands

Using View Window Commands

Using the Graph Window

Displaying Graph Views

Using the Surface Window

Displaying Surface Views

Manipulating Surface Data

Visualizing Data Programmatically

Launching the Visualizer from the Command Line

Configuring TotalView to Launch the Visualizer

Setting the Visualizer Launch Command

Adapting a Third Party Visualizer

Using the CLI

Using the Command Line Interface (CLI)

About the Tcl and the CLI

About The CLI and TotalView

Using the CLI Interface

Starting the CLI

Startup Example

Starting Your Program

About CLI Output

‘more’ Processing

Using Command Arguments

Using Namespaces

About the CLI Prompt

Using Built-in and Group Aliases

How Parallelism Affects Behavior

Types of IDs

Controlling Program Execution

Advancing Program Execution

Using Action Points

Seeing the CLI at Work

Setting the CLI EXECUTABLE_PATH Variable

Initializing an Array Slice

Printing an Array Slice

Writing an Array Variable to a File

Automatically Setting Breakpoints

Debugging

Debugging Programs

Searching and Looking For Program Elements

Searching for Text

Looking for Functions and Variables

Finding the Source Code for Functions

Resolving Ambiguous Names

Finding the Source Code for Files

Resetting the Stack Frame

Editing Source Text

Manipulating Processes and Threads

Using the Toolbar to Select a Target

Stopping Processes and Threads

Using the Processes/Ranks Tab

Using the Threads Tab

Updating Process Information

Holding and Releasing Processes and Threads

Using Barrier Points

Barrier Point Illustration

Examining Groups

Placing Processes in Groups

Starting Processes and Threads

Creating a Process Without Starting It

Creating a Process by Single-Stepping

Stepping and Setting Breakpoints

Using Stepping Commands

Stepping into Function Calls

Stepping Over Function Calls

Executing to a Selected Line

Executing Out of a Function

Continuing with a Specific Signal

Killing (Deleting) Programs

Restarting Programs

Checkpointing

Fine-Tuning Shared Library Use

Preloading Shared Libraries

Controlling Which Symbols TotalView Reads

Specifying Which Libraries are Read

Reading Excluded Information

Setting the Program Counter

Interpreting the Status and Control Registers

Using Groups, Processes, and Threads

Defining the GOI, POI, and TOI

Setting a Breakpoint

Stepping (Part I)

Understanding Group Widths

Understanding Process Width

Understanding Thread Width

Using Run To and duntil Commands

Using P/T Set Controls

Setting Process and Thread Focus

Understanding Process/Thread Sets

Specifying Arenas

Specifying Processes and Threads

Defining the Thread of Interest (TOI)

About Process and Thread Widths

Specifier Examples

Setting Group Focus

Specifying Groups in P/T Sets

About Arena Specifier Combinations

‘All’ Does Not Always Mean ‘All’

Setting Groups

Using the g Specifier: An Extended Example

Merging Focuses

Naming Incomplete Arenas

Naming Lists with Inconsistent Widths

Stepping (Part II): Examples

Using P/T Set Operators

Creating Custom Groups

Examining and Changing Data

Changing How Data is Displayed

Displaying STL Variables

Changing Size and Precision

Displaying Variables

Displaying Program Variables

Controlling the Information Being Displayed

Seeing Value Changes

Seeing Structure Information

Displaying Variables in the Current Block

Viewing Variables in Different Scopes as Program Executes

Scoping Issues

Freezing Variable Window Data

Locking the Address

Browsing for Variables

Displaying Local Variables and Registers

Dereferencing Variables Automatically

Examining Memory

Displaying Areas of Memory

Displaying Machine Instructions

Rebinding the Variable Window

Closing Variable Windows

Diving in Variable Windows

Displaying an Array of Structure’s Elements

Changing What the Variable Window Displays

Viewing a List of Variables

Entering Variables and Expressions

Seeing Variable Value Changes in the Expression List Window

Entering Expressions into the Expression Column

Using the Expression List with Multi-process/Multi-threaded Programs

Reevaluating, Reopening, Rebinding, and Restarting

Seeing More Information

Sorting, Reordering, and Editing

Changing the Values of Variables

Changing a Variable’s Data Type

Displaying C and C++ Data Types

Viewing Pointers to Arrays

Viewing Arrays

Viewing typedef Types

Viewing Structures

Viewing Unions

Casting Using the Built-In Types

Viewing Character Arrays ($string Data Type)

Viewing Wide Character Arrays ($wchar Data Types)

Viewing Areas of Memory ($void Data Type)

Viewing Instructions ($code Data Type)

Viewing Opaque Data

Type-Casting Examples

Displaying Declared Arrays

Displaying Allocated Arrays

Displaying the argv Array

Changing the Address of Variables

Displaying C++ Types

Viewing Classes

C++View

Displaying Fortran Types

Displaying Fortran Common Blocks

Displaying Fortran Module Data

Debugging Fortran 90 Modules

Viewing Fortran 90 User-Defined Types

Viewing Fortran 90 Deferred Shape Array Types

Viewing Fortran 90 Pointer Types

Displaying Fortran Parameters

Displaying Thread Objects

Scoping and Symbol Names

Qualifying Symbol Names

Examining Arrays

Examining and Analyzing Arrays

Displaying Array Slices

Using Slices and Strides

Using Slices in the Lookup Variable Command

Array Slices and Array Sections

Data Format Selection Box

Filtering Array Data Overview

Filtering Array Data

Filtering by Comparison

Filtering for IEEE Values

Filtering a Range of Values

Creating Array Filter Expressions

Using Filter Comparisons

Sorting Array Data

Obtaining Array Statistics

Displaying a Variable in all Processes or Threads

Diving on a “Show Across” Pointer

Editing a “Show Across” Variable

Visualizing Array Data

Visualizing a “Show Across” Variable Window

Setting Action Points

About Action Points

Setting Breakpoints and Barriers

Setting Source-Level Breakpoints

Choosing Source Lines

Setting Breakpoints at Locations

Ambiguous Functions and Pending Breakpoints

Displaying and Controlling Action Points

Disabling Action Points

Deleting Action Points

Enabling Action Points

Suppressing Action Points

Setting Breakpoints on Classes and Virtual and Overloaded Functions

Setting Machine-Level Breakpoints

Setting Breakpoints for Multiple Processes

Setting Breakpoints When Using the fork()/execve() Functions

Debugging Processes That Call the fork() Function

Debugging Processes that Call the execve() Function

Example: Multi-process Breakpoint

Setting Barrier Points

About Barrier Breakpoint States

Setting a Barrier Breakpoint

Creating a Satisfaction Set

Hitting a Barrier Point

Releasing Processes from Barrier Points

Deleting a Barrier Point

Changing Settings and Disabling a Barrier Point

Defining Eval Points and Conditional Breakpoints

Setting Eval Points

Creating Conditional Breakpoint Examples

Patching Programs

Branching Around Code

Adding a Function Call

Correcting Code

About Interpreted and Compiled Expressions

About Interpreted Expressions

About Compiled Expressions

Allocating Patch Space for Compiled Expressions

Allocating Dynamic Patch Space

Allocating Static Patch Space

Using Watchpoints

Using Watchpoints on Different Architectures

Creating Watchpoints

Displaying Watchpoints

Watching Memory

Triggering Watchpoints

Using Multiple Watchpoints

Copying Previous Data Values

Using Conditional Watchpoints

Saving Action Points to a File

Evaluating Expressions

Why is There an Expression System?

Calling Functions: Problems and Issues

Expressions in Eval Points and the Evaluate Window

Using C++

Using Programming Language Elements

Using Fortran

Fortran Statements

Fortran Intrinsics

Using the Evaluate Window

Writing Assembler Code

Using Built-in Variables and Statements

Using TotalView Variables

Using Built-In Statements

Using the CUDA Debugger

TotalView CUDA Debugging Model

Installing the CUDA SDK Tool Chain

TotalView CUDA Debugging Tutorial

Compiling for Debugging

Compiling for Fermi

Compiling for Fermi and Tesla

Backward Compatibility with CUDA Device Drivers

Starting TotalView

Loading the CUDA Kernel

Running to a Breakpoint in the GPU code

Single-Stepping GPU Code

Halting a Running Application

GPU Assembler Display

GPU Variable and Data Display

CUDA Built-In Runtime Variables

Type Casting

PTX Registers

Enabling CUDA MemoryChecker Feature

GPU Error Reporting

Displaying Device Information

Problems and Limitations

System Limitations

SDK 3.0, 3.1, 3.2, 4.0, and 4.1 Limitations

SDK 3.0 and 3.1 Limitations

Hangs or Initialization Failures

CUDA and ReplayEngine

Sample CUDA Program

Directive-Based Accelerator Programimg Languages

Glossary

Debugging Memory Problems with MemoryScape

Locating Memory Problems

Checking for Problems

Finding Heap Allocation Problems

Finding Heap Deallocation Problems

realloc() Problems

Finding Memory Leaks

Starting MemoryScape

Using MemoryScape Options

Preloading MemoryScape

Understanding How Your Program is Using Memory

Finding free() and realloc() Problems

Event and Error Notification

Types of Problems

Freeing Stack Memory

Freeing bss Data

Freeing Data Section Memory

Freeing Memory That Is Already Freed

Tracking realloc() Problems

Freeing the Wrong Address

Finding Memory Leaks

Fixing Dangling Pointer Problems

Dangling Pointers

Batch Scripting and Using the CLI

Using the –dheap Command

dheap Example

dheap

Notification When free Problems Occur

Showing Backtrace Information: dheap –backtrace:

Guarding Memory Blocks: dheap -guards

Memory Reuse: dheap –hoard

Writing Heap Information: dheap –export

Filtering Heap Information: dheap –filter

Checking for Dangling Pointers: dheap –is_dangling:

Detecting Leaks: dheap –leaks

Block Painting: dheap –paint

Red Zones Bounds Checking: dheap –red_zones

Deallocation Notification: dheap –tag_alloc

Additional Memory Block Information

Filtering

Using Guard Blocks

Using Red Zones

Using Guard Blocks and Red Zones

Block Painting

Hoarding

Example 1: Finding a Multithreading Problem

Example 2: Finding Dangling Pointer References

Debugging with TotalView

Memory Tasks

Getting Started

Starting MemoryScape

Adding Programs and Files to MemoryScape

Attaching to Programs and Adding Core Files

Attaching to a Process Owned by Another User

Stopping Before Finishing Execution

Exporting Memory Data

MemoryScape Information

Where to Go Next

Adding Parallel Programs

Where to Go Next

Setting MemoryScape Options

Basic Options

Advanced Options

Halt execution at process exit

Halt execution on memory event or error

Guard allocated memory

Use Red Zones to find memory access violations

Restricting Red Zones

Customizing Red Zones

Paint memory

Hoard deallocated memory

Where to Go Next

Controlling Program Execution

Controlling Program Execution from the Home | Summary Screen

Controlling Program Execution from the Manage Processes Screen

Controlling Program Execution from a Context Menu

Where to Go Next

Seeing Memory Usage

Information Types

Process and Library Reports

Deallocation and Reuse Notifications

Where to Go Next

Graphically Viewing the Heap

Obtaining Detailed Heap Information

Heap Status Source Report

Heap Status Source Backtrace Report

Where to Go Next

Seeing Leaks

Filtering Reports

Adding, Deleting, Enabling and Disabling Filters

Adding and Editing Filters

Where to Go Next

Viewing Corrupted Memory

Examining Corrupted Memory Blocks

Viewing Memory Contents

Saving and Restoring Memory State Information

Procedures for Exporting and Adding Memory Data

Using Saved State Information

Where to Go Next

Comparing Memory

Overview

Obtaining a Comparison

Memory Comparison Report

Where to Go Next

Saving Memory Information as HTML

Saving Report Information

Hoarding Deallocated Memory

Painting Memory

Accessing MemoryScape Remotely

Creating Programs for Memory Debugging

Compiling Programs

Linking with the dbfork Library

dbfork on IBM AIX on RS/6000 Systems

Linking C++ Programs with dbfork

dbfork and Linux or Mac OS X

dbfork and SunOS 5 SPARC

Starting MemoryScape

Attaching to Programs

Setting Up MPI Debugging Sessions

Debugging MPI Programs

Debugging MPICH Applications

Starting MemoryScape on an MPICH Job

Attaching to an MPICH Job

Using MPICH P4 procgroup Files

Starting MPI Issues

Debugging IBM MPI Parallel Environment (PE) Applications

Using Switch-Based Communications

Performing a Remote Login

Starting MemoryScape on a PE Program

Attaching to a PE Job

Debugging LAM/MPI Applications

Debugging QSW RMS Applications

Starting MemoryScape on an RMS Job

Attaching to an RMS Job

Debugging Sun MPI Applications

Attaching to a Sun MPI Job

Linking Your Application with the Agent

Using env to Insert the Agent

Installing tvheap_mr.a on AIX

LIBPATH and Linking

Using MemoryScape in Selected Environments

MPICH

IBM PE

RMS MPI

MemoryScape Scripting

display_specifiers Command-Line Option

event_action Command-Line Option

Other Command Line Options

memscript Example

MemoryScape Command-Line Options

Invoking MemoryScape

Syntax

Options

Reverse Debugging with ReplayEngine

Understanding ReplayEngine

How ReplayEngine Works

System Resources ReplayEngine Uses

Replaying Your Program

Threads and Processes

Attaching to Running Programs

Using ReplayEngine

Enabling and Disabling ReplayEngine

Enabling ReplayEngine at Program Load

Enabling and Disabling ReplayEngine for a Loaded Program

Enabling Replay

Disabling Replay

ReplayEngine and CUDA

Examining Program State and History

Setting Preferences

CLI Support

Known Limitations and Issues

Performance Issues