Periodically, the OProfile daemon, oprofiled, collects the samples and writes them to the /var/lib/oprofile/samples/ directory. Before reading the data, make sure all data has been
written to this directory by executing the following command as root:
opcontrol --dump
Each sample file name is based on the name of the executable. For example, the samples for the default event on a Pentium III processor for /bin/bash becomes:
\{root\}/bin/bash/\{dep\}/\{root\}/bin/bash/CPU_CLK_UNHALTED.100000
The following tools are available to profile the sample data once it has been collected:
Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.
Warning
The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, backup the executable used to create the samples as well as the sample files. Please note that the sample file and the binary have to agree. Making a backup isn't going to work if they do not match. oparchive can be used to address this problem.
Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version.
The oprofile analysis tools use the executable file that created the samples during analysis. If the executable changes the analysis tools will be unable to analyze the associated samples. Refer to Section 38.4, “Saving Data” for details on how to backup the sample file.
The opreport tool provides an overview of all the executables being profiled.
The following is part of a sample output:
Profiling through timer interrupt
TIMER:0|
samples| %|
------------------
25926 97.5212 no-vmlinux
359 1.3504 pi
65 0.2445 Xorg
62 0.2332 libvte.so.4.4.0
56 0.2106 libc-2.3.4.so
34 0.1279 libglib-2.0.so.0.400.7
19 0.0715 libXft.so.2.1.2
17 0.0639 bash
8 0.0301 ld-2.3.4.so
8 0.0301 libgdk-x11-2.0.so.0.400.13
6 0.0226 libgobject-2.0.so.0.400.7
5 0.0188 oprofiled
4 0.0150 libpthread-2.3.4.so
4 0.0150 libgtk-x11-2.0.so.0.400.13
3 0.0113 libXrender.so.1.2.2
3 0.0113 du
1 0.0038 libcrypto.so.0.9.7a
1 0.0038 libpam.so.0.77
1 0.0038 libtermcap.so.2.0.8
1 0.0038 libX11.so.6.2
1 0.0038 libgthread-2.0.so.0.400.7
1 0.0038 libwnck-1.so.4.9.0
Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is the name of the executable.
Refer to the opreport man page for a list of available command line options, such as the -r option used to sort the output from the executable with the smallest number of samples to the one with the
largest number of samples.
38.5.2. Using opreport on a Single Executable
To retrieve more detailed profiled information about a specific executable, use opreport:
opreport <mode> <executable>
<executable> must be the full path to the executable to be analyzed. <mode> must be one of the following:
-
-l
-
List sample data by symbols. For example, the following is part of the output from running the command opreport -l /lib/tls/libc-<version>.so:
samples % symbol name
12 21.4286 __gconv_transform_utf8_internal
5 8.9286 _int_malloc
4 7.1429 malloc
3 5.3571 __i686.get_pc_thunk.bx
3 5.3571 _dl_mcount_wrapper_check
3 5.3571 mbrtowc
3 5.3571 memcpy
2 3.5714 _int_realloc
2 3.5714 _nl_intern_locale_data
2 3.5714 free
2 3.5714 strcmp
1 1.7857 __ctype_get_mb_cur_max
1 1.7857 __unregister_atfork
1 1.7857 __write_nocancel
1 1.7857 _dl_addr
1 1.7857 _int_free
1 1.7857 _itoa_word
1 1.7857 calc_eclosure_iter
1 1.7857 fopen@@GLIBC_2.1
1 1.7857 getpid
1 1.7857 memmove
1 1.7857 msort_with_tmp
1 1.7857 strcpy
1 1.7857 strlen
1 1.7857 vfprintf
1 1.7857 write
The first column is the number of samples for the symbol, the second column is the percentage of samples for this symbol relative to the overall samples for the executable, and the third column is the symbol name.
To sort the output from the largest number of samples to the smallest (reverse order), use -r in conjunction with the -l option.
-
-i <symbol-name>
-
List sample data specific to a symbol name. For example, the following output is from the command opreport -l -i __gconv_transform_utf8_internal /lib/tls/libc-<version>.so:
samples % symbol name
12 100.000 __gconv_transform_utf8_internal
The first line is a summary for the symbol/executable combination.
The first column is the number of samples for the memory symbol. The second column is the percentage of samples for the memory address relative to the total number of samples for the symbol. The third column is the symbol name.
-
-d
-
List sample data by symbols with more detail than -l. For example, the following output is from the command opreport -l -d __gconv_transform_utf8_internal /lib/tls/libc-<version>.so:
vma samples % symbol name
00a98640 12 100.000 __gconv_transform_utf8_internal
00a98640 1 8.3333
00a9868c 2 16.6667
00a9869a 1 8.3333
00a986c1 1 8.3333
00a98720 1 8.3333
00a98749 1 8.3333
00a98753 1 8.3333
00a98789 1 8.3333
00a98864 1 8.3333
00a98869 1 8.3333
00a98b08 1 8.3333
The data is the same as the -l option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the
symbol is displayed.
-
-x
<symbol-name>
-
Exclude the comma-separated list of symbols from the output.
-
session:<name>
-
Specify the full path to the session or a directory relative to the /var/lib/oprofile/samples/ directory.
38.5.3. Getting more detailed output on the modules
OProfile collects data on a system-wide basis for kernel- and
user-space code running on the machine. However, once a module is
loaded into the kernel, the information about the origin of the kernel
module is lost. The module could have come from the initrd file on
boot up, the directory with the various kernel modules, or a locally
created kernel module. As a result when OProfile records sample for a
module, it just lists the samples for the modules for an executable in
the root directory, but this is unlikely to be the place with the actual
code for the module. You will need to take some steps to make sure
that analysis tools get the executable.
For example on an AMD64 machine the sampling is set up to record "Data
cache accesses" and "Data cache misses" and assuming you would like to see the
data for the ext3 module:
$ opreport /ext3
CPU: AMD64 processors, speed 797.948 MHz (estimated)
Counted DATA_CACHE_ACCESSES events (Data cache accesses) with a unit mask of 0x00 (No unit mask) count 500000
Counted DATA_CACHE_MISSES events (Data cache misses) with a unit mask of 0x00 (No unit mask) count 500000
DATA_CACHE_ACC...|DATA_CACHE_MIS...|
samples| %| samples| %|
------------------------------------
148721 100.000 1493 100.000 ext3
To get a more detailed view of the actions of the module, you will
need to either have the module unstripped (e.g. installed from a
custom build) or have the debuginfo RPM installed for the kernel.
Find out which kernel is running, "uname -a", get the appropriate
debuginfo rpm, and install on the machine.
Then make a symbolic link so oprofile finds the code for the module in the
correct place:
# ln -s /lib/modules/`uname -r`/kernel/fs/ext3/ext3.ko /ext3
Then the detailed information can be obtained with:
# opreport image:/ext3 -l|more
warning: could not check that the binary file /ext3 has not been modified since the profile was taken. Results may be inaccurate.
CPU: AMD64 processors, speed 797.948 MHz (estimated)
Counted DATA_CACHE_ACCESSES events (Data cache accesses) with a unit mask of 0x00 (No unit mask) count 500000
Counted DATA_CACHE_MISSES events (Data cache misses) with a unit mask of 0x00 (No unit mask) count 500000
samples % samples % symbol name
16728 11.2479 7 0.4689 ext3_group_sparse
16454 11.0637 4 0.2679 ext3_count_free_blocks
14583 9.8056 51 3.4159 ext3_fill_super
8281 5.5681 129 8.6403 ext3_ioctl
7810 5.2514 62 4.1527 ext3_write_info
7286 4.8991 67 4.4876 ext3_ordered_writepage
6509 4.3767 130 8.7073 ext3_new_inode
6378 4.2886 156 10.4488 ext3_new_block
5932 3.9887 87 5.8272 ext3_xattr_block_list
...
The opannotate tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting files generated should have the samples for the lines at the left. It also puts
in a comment at the beginning of each function listing the total samples for the function.
For this utility to work, the executable must be compiled with GCC's -g option. By default, Red Hat Enterprise Linux packages are not compiled with this option.
The general syntax for opannotate is as follows:
opannotate --search-dirs <src-dir> --source <executable>
The directory containing the source code and the executable to be analyzed must be specified. Refer to the opannotate man page for a list of additional command line options.
