RHEL 6 SystemTap Guide - 2.3. Running SystemTap Scripts

Red Hat Enterprise Linux 9 Essentials Book now available.

Purchase a copy of Red Hat Enterprise Linux 9 (RHEL 9) Essentials

Red Hat Enterprise Linux 9 Essentials Print and eBook (PDF) editions contain 34 chapters and 298 pages

2.3. Running SystemTap Scripts

SystemTap scripts are run through the command stap. stap can run SystemTap scripts from standard input or from file.

Running stap and staprun requires elevated privileges to the system. However, not all users can be granted root access just to run SystemTap. In some cases, for instance, a non-privileged user may need to to run SystemTap instrumentation on their machine.

To allow ordinary users to run SystemTap without root access, add them to one of these user groups:

stapdev: Members of this group can use stap to run SystemTap scripts, or staprun to run SystemTap instrumentation modules.

Running stap involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This requires elevated privileges to the system, which are granted to stapdev members. Unfortunately, such privileges also grant effective root access to stapdev members. As such, only grant stapdev group membership to users who can be trusted with root access.
stapusr: Members of this group can only use staprun to run SystemTap instrumentation modules. In addition, they can only run those modules from /lib/modules/kernel_version/systemtap/. Note that this directory must be owned only by the root user, and must only be writable by the root user.

Below is a list of commonly used stap options:

-v: Makes the output of the SystemTap session more verbose. This option (for example, stap -vvv script.stp) can be repeated to provide more details on the script's execution. It is particularly useful if errors are encountered when running the script. This option is particularly useful if you encounter any errors in running the script.

For more information about common SystemTap script errors, refer to Chapter 5, Understanding SystemTap Errors.
-o filename: Sends the standard output to file (filename).
-S size,count: Limit files to size megabytes and limit the number of files kept around to count. The file names will have a sequence number suffix. This option implements logrotate operations for SystemTap.

When used with -o, the -S will limit the size of log files.
-x process ID: Sets the SystemTap handler function target() to the specified process ID. For more information about target(), refer to SystemTap Functions.
-c command: Sets the SystemTap handler function target() to the specified command. The full path to the specified command must be used; for example, instead of specifying cp, use /bin/cp (as in stap script -c /bin/cp). For more information about target(), refer to SystemTap Functions.
-e 'script': Use script string rather than a file as input for systemtap translator.
-F: Use SystemTap's Flight recorder mode and make the script a background process. For more information about flight recorder mode, refer to Section 2.3.1, “SystemTap Flight Recorder Mode”.

stap can also be instructed to run scripts from standard input using the switch -. To illustrate:

Example 2.1. Running Scripts From Standard Input

echo "probe timer.s(1) {exit()}" | stap -

Example 2.1, “Running Scripts From Standard Input” instructs stap to run the script passed by echo to standard input. Any stap options to be used should be inserted before the - switch; for instance, to make the example in Example 2.1, “Running Scripts From Standard Input” more verbose, the command would be:

echo "probe timer.s(1) {exit()}" | stap -v -

For more information about stap, refer to man stap.

To run SystemTap instrumentation (i.e. the kernel module built from SystemTap scripts during a cross-instrumentation), use staprun instead. For more information about staprun and cross-instrumentation, refer to Section 2.2, “Generating Instrumentation for Other Computers”.

Note

The stap options -v and -o also work for staprun. For more information about staprun, refer to man staprun.

2.3.1. SystemTap Flight Recorder Mode

SystemTap's flight recorder mode allows a SystemTap script to be ran for long periods and just focus on recent output. The flight recorder mode (the -F option) limits the amount of output generated. There are two variations of the flight recorder mode: in-memory and file mode. In both cases the SystemTap script runs as a background process.

2.3.1.1. In-memory Flight Recorder

When flight recorder mode (the -F option) is used without a file name, SystemTap uses a buffer in kernel memory to store the output of the script. Next, SystemTap instrumentation module loads and the probes start running, then instrumentation will detatch and be put in the background. When the interesting event occurs, the instrumentation can be reattached and the recent output in the memory buffer and any continuing output can be seen. The following command starts a script using the flight recorder in-memory mode:

stap -F /usr/share/doc/systemtap-version/examples/io/iotime.stp

Once the script starts, a message that provides the command to reconnect to the running script will appear:

Disconnecting from systemtap module.
To reconnect, type "staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556"

When the interesting event occurs, reattach to the currently running script and output the recent data in the memory buffer, then get the continuing output with the following command:

staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556

By default, the kernel buffer is 1MB in size, but it can be increased with the -s option specifying the size in megabytes (rounded up to the next power over 2) for the buffer. For example -s2 on the SystemTap command line would specify 2MB for the buffer.

2.3.1.2. File Flight Recorder

The flight recorder mode can also store data to files. The number and size of the files kept is controlled by the -S option followed by two numerical arguments separated by a comma. The first argument is the maximum size in megabytes for the each output file. The second argument is the number of recent files to keep. The file name is specified by the -o option followed by the name. SystemTap adds a number suffix to the file name to indicate the order of the files. The following will start SystemTap in file flight recorder mode with the output going to files named /tmp/pfaults.log.[0-9]+ with each file 1MB or smaller and keeping latest two files:

stap -F -o /tmp/pfaults.log -S 1,2  pfaults.stp

The number printed by the command is the process ID. Sending a SIGTERM to the process will shutdown the SystemTap script and stop the data collection. For example if the previous command listed the 7590 as the process ID, the following command whould shutdown the systemtap script:

kill -s SIGTERM 7590

Only the most recent two file generated by the script are kept and the older files are been removed. Thus, ls -sh /tmp/pfaults.log.* shows the only two files:

1020K /tmp/pfaults.log.5    44K /tmp/pfaults.log.6

One can look at the highest number file for the latest data, in this case /tmp/pfaults.log.6.