RHEL 6 Deployment - 21.4. Generating Backtraces

Red Hat Enterprise Linux 9 Essentials Book now available.

Purchase a copy of Red Hat Enterprise Linux 9 (RHEL 9) Essentials

Red Hat Enterprise Linux 9 Essentials Print and eBook (PDF) editions contain 34 chapters and 298 pages

21.4. Generating Backtraces

In order to analyze a reported crash, developers need as much detail about the crash as possible. A stack backtrace is an important source of information when a crash in a binary program (caught by the CCpp analyzer plugin) occurs.

ABRT is configured to generate a backtrace whenever a crash is reported through the ABRT GUI application or the ABRT command line interface.

ABRT completes the following steps to generate a backtrace:

It examines the core dump (which consists of the recorded contents of the memory of an application at a specific time), which is saved in the crash dump directory. From this file, ABRT extracts the information about the crashed binary program and information about every loaded dynamic library.
It queries Yum to determine which debuginfo packages correspond to all the files extracted from the crash dump. This is the first potentially slow operation. Yum may need to refresh the filelists of various repositories in order to find the correct package names. This process may take a few minutes.
It downloads the needed debuginfo packages, and extracts and saves the debuginfo files. In order to speed up future backtrace generation, debuginfo files are cached in the /var/cache/abrt-di directory.
It generates a backtrace using GDB (the GNU Debugger) and saves it into the crash dump directory.

You can change the following backtrace generation parameters in the /etc/abrt/plugins/CCpp.conf file:

Backtrace = <yes/no> — Enables/Disables backtrace generation.
BacktraceRemotes = <yes/no> — For more information about this parameter, refer to Section 21.7, “Configuring Centralized Crash Collection”.
InstallDebugInfo = <yes/no> — Enables/Disables the installation of debuginfo packages (useful if your network is not available or it is firewalled).
ReadonlyLocalDebugInfoDirs = /path1:/path2:... — Specifies the paths of local repositories (available, for example, through a network mount) that contain pre-downloaded debuginfo packages.
DebugInfoCacheMB = 4000 — Specifies the maximum size of the cached debuginfo packages in the /var/cache/abrt-di directory.

21.4.1. Troubleshooting Backtrace Generation

In some cases, a long delay in the ABRT GUI application occurs after choosing a crash and pressing the Report button. In this case, open the Details in the Generating backtrace window and examine the messages.

The following is a typical output seen in the Generating backtrace window:

Starting the debuginfo installation
Getting list of build IDs
12 missing debuginfos, getting package list from cache
12 missing debuginfos, getting package list from repositories
Downloading 7 packages
Download 1/7: acl-debuginfo-2.2.49-6.fc13.x86_64
Unpacking: acl-debuginfo-2.2.49-6.fc13.x86_64.rpm
Caching debuginfo:
usr/lib/debug/.build-id/3d/e20df1db609bd9313b1dc440796004f95911fd.debug
Download 2/7: firefox-debuginfo-3.6.7-1.fc13.x86_64
Unpacking: firefox-debuginfo-3.6.7-1.fc13.x86_64.rpm
Caching debuginfo:
usr/lib/debug/.build-id/3d/b29c9308cb276431ce8854a2d88cf83518afc6.debug
Caching debuginfo:
usr/lib/debug/.build-id/a3/86884285365c8288e4e761ec034fafaa1daee1.debug
⋮
Download 7/7: zlib-debuginfo-1.2.3-23.fc12.x86_64
Unpacking: zlib-debuginfo-1.2.3-23.fc12.x86_64.rpm
Caching debuginfo:
usr/lib/debug/.build-id/f7/933750da80f555321576e72b375caf7a3cc075.debug
All needed debuginfos are present
Generating backtrace

This process is performed by the /usr/bin/abrt-debuginfo-install shell script. This script uses a temporary directory (e.g. /var/run/abrt/tmp-29177-1283344373) for its operations. Normally, this directory is removed when abrt-debuginfo-install exits.

If the debuginfo installation hangs, or is unable to download anything, you may debug the problem by editing the abrt-debuginfo-install script. Change the following parameters:

debug=false
keep_tmp=false

at the top of the script to:

debug=true
keep_tmp=true

The first parameter instructs abrt-debuginfo-install to be verbose, the second parameter instructs abrt-debuginfo-install to not delete the /var/run/abrt/tmp-NNN-NNN directory. You can examine the log files in this directory, they may contain useful error messages.

abrt-debuginfo-install uses yum and yumdownloader to handle the debuginfo packages. In order to quickly check that your yum configuration does not cause any problems which prevent abrt-debuginfo-install from working properly, change to the /tmp directory and run the following commands, as root:

tmp]# yum --enablerepo=*debuginfo* --quiet provides /usr/bin/true
tmp]# yumdownloader --enablerepo=*debuginfo* --quiet coreutils

Both of these commands should complete successfully, with no error messages. The second command should download the coreutils-*.rpm file. If any error messages appear, check your yum configuration files in the /etc/yum.repos.d/* directory and the /etc/yum/* directory. If any of these commands hang, check that you do not have another instance of yum running, and that your network connection is working properly.