dlopen and RTLD_DEEPBIND
In most circumstances, memory debugging works seamlessly with programs that call dlopen to dynamically load shared objects (DSOs).
However, the Linux implementation of dlopen accepts RTLD_DEEPBIND in the flags/mode argument. RTLD_DEEPBIND affects how undefined references in a DSO are bound. By default, when RTLD_DEEPBIND is not set, the dynamic linker first looks up any symbols needed by a newly-loaded DSO in the global scope.
RTLD_DEEPBIND modifies this behavior. When set, the dynamic linker places the lookup scope of the DSO ahead of the global scope. This means that the dynamic linker seeks to bind any undefined references in the DSO to definitions in the DSO, or any of the DSOs on which it depends. Only after these have been searched and a symbol not found is the global scope examined.
RTLD_DEEPBIND can affect memory debugging because, in some circumstances, references to all or part of the heap manager interface in a DSO can become bound to definitions in the standard library directly, rather than to those in the HIA. As a result, the HIA may not see all the traffic between the program and heap manager. If this occurs, the information the HIA is able to collect for TotalView will be incomplete, reducing its usefulness. In some circumstances, memory debugging may even fail.
How the HIA Handles RTLD_DEEPBIND
The HIA deals with the challenges posed by RTLD_DEEPBIND by intercepting calls to dlopen. If the program specifies RTLD_DEEPBIND, the HIA inserts itself as one of the to-be-loaded DSO's dependents. It does this by creating a new ELF wrapper file that lists the HIA and the DSO the program wants to dlopen as needed files. Instead of opening the DSO the program named, the HIA dlopens the new wrapper DSO it constructed. Since the DSO given by the program code is listed as a needed file, it too is opened.
As far as the program is concerned, the dlopen behaves as it would in the absence of the HIA. After the call to dlopen, the HIA cleans up and deletes the wrapper DSO that it created.
Modifying How the HIA Handles RTLD_DEEPBIND
The basic behavior described in
How the HIA Handles RTLD_DEEPBIND can be modified by setting the
TVHEAP_DEEPBIND environment variable. The following comma-separated settings are supported options:
pass_through=boolean
If true, the HIA does no special processing to handle RTLD_DEEPBIND. It does not create the ELF wrapper, and instead passes the operation through to the standard dlopen.
Example: "TVHEAP_DEEPBIND=pass_through=true"
keep_wrapper=boolean
If true, the HIA does not delete the ELF wrapper it creates after it has been used. The default is false, in which case the HIA deletes the ELF wrapper after it dlopens it.
Example: "TVHEAP_DEEPBIND=keep_wrapper=true"
tmpdir=<directory_name>
If defined, the HIA creates the ELF wrapper it generates in the directory specified by the tmpdir setting. The default is to use the setting of the environment variable TMPDIR. If TMPDIR is not defined, the ELF wrapper is created in /tmp.
