Monitoring bandwidth and CPU utilization under Red Hat Enterprise Linux entails using
the tools discussed in Chapter 2 Resource Monitoring; therefore, if you
have not yet read that chapter, you should do so before
continuing.
As stated in Section 2.4.2 Monitoring Bandwidth, it is difficult to
directly monitor bandwidth utilization. However, by examining
device-level statistics, it is possible to roughly gauge whether
insufficient bandwidth is an issue on your system.
By using vmstat, it is possible to determine if
overall device activity is excessive by examining the
bi and
bo fields; in addition, taking note
of the si and
so fields give you a bit more insight
into how much disk activity is due to swap-related I/O:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 0 248088 158636 480804 0 0 2 6 120 120 10 3 87
|
In this example, the bi field
shows two blocks/second written to block devices (primarily disk
drives), while the bo field shows six
blocks/second read from block devices. We can determine that none of
this activity was due to swapping, as the
si and
so fields both show a swap-related
I/O rate of zero kilobytes/second.
By using iostat, it is possible to gain a bit
more insight into disk-related activity:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003
avg-cpu: %user %nice %sys %idle
5.34 4.60 2.83 87.24
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 1.10 6.21 25.08 961342 3881610
dev8-1 0.00 0.00 0.00 16 0
|
This output shows us that the device with major number 8 (which is
/dev/sda, the first SCSI disk) averaged slightly
more than one I/O operation per second (the
tsp field). Most of the I/O activity
for this device were writes (the
Blk_wrtn field), with slightly more
than 25 blocks written each second (the
Blk_wrtn/s field).
If more detail is required, use iostat's
-x option:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003
avg-cpu: %user %nice %sys %idle
5.37 4.54 2.81 87.27
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz
/dev/sda 13.57 2.86 0.36 0.77 32.20 29.05 16.10 14.53 54.52
/dev/sda1 0.17 0.00 0.00 0.00 0.34 0.00 0.17 0.00 133.40
/dev/sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.56
/dev/sda3 0.31 2.11 0.29 0.62 4.74 21.80 2.37 10.90 29.42
/dev/sda4 0.09 0.75 0.04 0.15 1.06 7.24 0.53 3.62 43.01
|
Over and above the longer lines containing more fields, the first
thing to keep in mind is that this iostat output is
now displaying statistics on a per-partition level. By using
df to associate mount points with device names, it
is possible to use this report to determine if, for example, the
partition containing /home/ is experiencing an
excessive workload.
Actually, each line output from iostat -x is
longer and contains more information than this; here is the remainder
of each line (with the device column added for easier reading):
Device: avgqu-sz await svctm %util
/dev/sda 0.24 20.86 3.80 0.43
/dev/sda1 0.00 141.18 122.73 0.03
/dev/sda2 0.00 6.00 6.00 0.00
/dev/sda3 0.12 12.84 2.68 0.24
/dev/sda4 0.11 57.47 8.94 0.17
|
In this example, it is interesting to note that
/dev/sda2 is the system swap partition; it is
obvious from the many fields reading
0.00 for this partition that swapping
is not a problem on this system.
Another interesting point to note is
/dev/sda1. The statistics for this partition are
unusual; the overall activity seems low, but why are the average I/O
request size (the avgrq-sz field),
average wait time (the await field),
and the average service time (the
svctm field) so much larger than the
other partitions? The answer is that this partition contains the
/boot/ directory, which is where the kernel and
initial ramdisk are stored. When the system boots, the read I/Os
(notice that only the rsec/s and
rkB/s fields are non-zero; no writing
is done here on a regular basis) used during the boot process are for
large numbers of blocks, resulting in the relatively long wait and
service times iostat displays.
It is possible to use sar for a longer-term
overview of I/O statistics; for example, sar -b
displays a general I/O report:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003
12:00:00 AM tps rtps wtps bread/s bwrtn/s
12:10:00 AM 0.51 0.01 0.50 0.25 14.32
12:20:01 AM 0.48 0.00 0.48 0.00 13.32
…
06:00:02 PM 1.24 0.00 1.24 0.01 36.23
Average: 1.11 0.31 0.80 68.14 34.79
|
Here, like iostat's initial display, the
statistics are grouped for all block devices.
Another I/O-related report is produced using sar
-d:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (raptor.example.com) 07/21/2003
12:00:00 AM DEV tps sect/s
12:10:00 AM dev8-0 0.51 14.57
12:10:00 AM dev8-1 0.00 0.00
12:20:01 AM dev8-0 0.48 13.32
12:20:01 AM dev8-1 0.00 0.00
…
06:00:02 PM dev8-0 1.24 36.25
06:00:02 PM dev8-1 0.00 0.00
Average: dev8-0 1.11 102.93
Average: dev8-1 0.00 0.00
|
This report provides per-device information, but with little
detail.
While there are no explicit statistics showing bandwidth
utilization for a given bus or datapath, we can at least determine
what the devices are doing and use their activity to indirectly
determine the bus loading.
Unlike bandwidth, monitoring CPU utilization is much more
straightforward. From a single percentage of CPU utilization in
GNOME System Monitor, to the more in-depth
statistics reported by sar, it is possible to
accurately determine how much CPU power is being consumed and by
what.
Moving beyond GNOME System Monitor,
top is the first resource monitoring tool discussed
in Chapter 2 Resource Monitoring to provide a more in-depth
representation of CPU utilization. Here is a top
report from a dual-processor workstation:
9:44pm up 2 days, 2 min, 1 user, load average: 0.14, 0.12, 0.09
90 processes: 82 sleeping, 1 running, 7 zombie, 0 stopped
CPU0 states: 0.4% user, 1.1% system, 0.0% nice, 97.4% idle
CPU1 states: 0.5% user, 1.3% system, 0.0% nice, 97.1% idle
Mem: 1288720K av, 1056260K used, 232460K free, 0K shrd, 145644K buff
Swap: 522104K av, 0K used, 522104K free 469764K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
30997 ed 16 0 1100 1100 840 R 1.7 0.0 0:00 top
1120 root 5 -10 249M 174M 71508 S < 0.9 13.8 254:59 X
1260 ed 15 0 54408 53M 6864 S 0.7 4.2 12:09 gnome-terminal
888 root 15 0 2428 2428 1796 S 0.1 0.1 0:06 sendmail
1264 ed 15 0 16336 15M 9480 S 0.1 1.2 1:58 rhn-applet-gui
1 root 15 0 476 476 424 S 0.0 0.0 0:05 init
2 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU0
3 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU1
4 root 15 0 0 0 0 SW 0.0 0.0 0:01 keventd
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0
6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU1
7 root 15 0 0 0 0 SW 0.0 0.0 0:05 kswapd
8 root 15 0 0 0 0 SW 0.0 0.0 0:00 bdflush
9 root 15 0 0 0 0 SW 0.0 0.0 0:01 kupdated
10 root 25 0 0 0 0 SW 0.0 0.0 0:00 mdrecoveryd
|
The first CPU-related information is present on the very first
line: the load average. The load average is a number corresponding to
the average number of runnable processes on the system. The load
average is often listed as three sets of numbers (as
top does), which represent the load average for the
past 1, 5, and 15 minutes, indicating that the system in this example
was not very busy.
The next line, although not strictly related to CPU utilization,
has an indirect relationship, in that it shows the number of runnable
processes (here, only one -- remember this number, as it means
something special in this example). The number of runnable processes
is a good indicator of how CPU-bound a system might be.
Next are two lines displaying the current utilization for each of
the two CPUs in the system. The utilization statistics show whether
the CPU cycles were expended for user-level or system-level
processing; also included is a statistic showing how much CPU time was
expended by processes with altered scheduling priorities. Finally,
there is an idle time statistic.
Moving down into the process-related section of the display, we
find that the process using the most CPU power is
top itself; in other words, the one runnable
process on this otherwise-idle system was top
taking a "picture" of itself.
 | Tip |
|---|
| | It is important to remember that the very act of running a
system monitor affects the resource utilization statistics you
receive. All software-based monitors do this to some extent.
|
To gain more detailed knowledge regarding CPU utilization, we must
change tools. If we examine output from vmstat, we
obtain a slightly different understanding of our example
system:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 0 233276 146636 469808 0 0 7 7 14 27 10 3 87
0 0 0 0 233276 146636 469808 0 0 0 0 523 138 3 0 96
0 0 0 0 233276 146636 469808 0 0 0 0 557 385 2 1 97
0 0 0 0 233276 146636 469808 0 0 0 0 544 343 2 0 97
0 0 0 0 233276 146636 469808 0 0 0 0 517 89 2 0 98
0 0 0 0 233276 146636 469808 0 0 0 32 518 102 2 0 98
0 0 0 0 233276 146636 469808 0 0 0 0 516 91 2 1 98
0 0 0 0 233276 146636 469808 0 0 0 0 516 72 2 0 98
0 0 0 0 233276 146636 469808 0 0 0 0 516 88 2 0 97
0 0 0 0 233276 146636 469808 0 0 0 0 516 81 2 0 97
|
Here we have used the command vmstat 1 10 to
sample the system every second for ten times. At first, the
CPU-related statistics (the us,
sy, and
id fields) seem similar to what
top displayed, and maybe even appear a bit less
detailed. However, unlike top, we can also gain a
bit of insight into how the CPU is being used.
If we examine at the system
fields, we notice that the CPU is handling about 500 interrupts per
second on average and is switching between processes anywhere from 80
to nearly 400 times a second. If you think this seems like a lot of
activity, think again, because the user-level processing (the
us field) is only averaging 2%, while
system-level processing (the sy
field) is usually under 1%. Again, this is an idle system.
Reviewing the tools Sysstat offers, we find that
iostat and mpstat provide little
additional information over what we have already experienced with
top and vmstat. However,
sar produces a number of reports that can come in
handy when monitoring CPU utilization.
The first report is obtained by the command sar
-q, which displays the run queue length, total number of
processes, and the load averages for the past one and five minutes.
Here is a sample:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003
12:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5
12:10:00 AM 3 122 0.07 0.28
12:20:01 AM 5 123 0.00 0.03
…
09:50:00 AM 5 124 0.67 0.65
Average: 4 123 0.26 0.26
|
In this example, the system is always busy (given that more than
one process is runnable at any given time), but is not overly loaded
(due to the fact that this particular system has more than one
processor).
The next CPU-related sar report is produced by
the command sar -u:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003
12:00:01 AM CPU %user %nice %system %idle
12:10:00 AM all 3.69 20.10 1.06 75.15
12:20:01 AM all 1.73 0.22 0.80 97.25
…
10:00:00 AM all 35.17 0.83 1.06 62.93
Average: all 7.47 4.85 3.87 83.81
|
The statistics contained in this report are no different from
those produced by many of the other tools. The biggest benefit here
is that sar makes the data available on an ongoing
basis and is therefore more useful for obtaining long-term averages,
or for the production of CPU utilization graphs.
On multiprocessor systems, the sar -U command
can produce statistics for an individual processor or for all
processors. Here is an example of output from sar -U
ALL:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003
12:00:01 AM CPU %user %nice %system %idle
12:10:00 AM 0 3.46 21.47 1.09 73.98
12:10:00 AM 1 3.91 18.73 1.03 76.33
12:20:01 AM 0 1.63 0.25 0.78 97.34
12:20:01 AM 1 1.82 0.20 0.81 97.17
…
10:00:00 AM 0 39.12 0.75 1.04 59.09
10:00:00 AM 1 31.22 0.92 1.09 66.77
Average: 0 7.61 4.91 3.86 83.61
Average: 1 7.33 4.78 3.88 84.02
|
The sar -w command reports on the number of
context switches per second, making it possible to gain additional
insight in where CPU cycles are being spent:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003
12:00:01 AM cswch/s
12:10:00 AM 537.97
12:20:01 AM 339.43
…
10:10:00 AM 319.42
Average: 1158.25
|
It is also possible to produce two different
sar reports on interrupt activity. The first,
(produced using the sar -I SUM command) displays a
single "interrupts per second" statistic:
Linux 2.4.21-1.1931.2.349.2.2.entsmp (falcon.example.com) 07/21/2003
12:00:01 AM INTR intr/s
12:10:00 AM sum 539.15
12:20:01 AM sum 539.49
…
10:40:01 AM sum 539.10
Average: sum 541.00
|
By using the command sar -I PROC, it is
possible to break down interrupt activity by processor (on
multiprocessor systems) and by interrupt level
(from 0 to 15):
Linux 2.4.21-1.1931.2.349.2.2.entsmp (pigdog.example.com) 07/21/2003
12:00:00 AM CPU i000/s i001/s i002/s i008/s i009/s i011/s i012/s
12:10:01 AM 0 512.01 0.00 0.00 0.00 3.44 0.00 0.00
12:10:01 AM CPU i000/s i001/s i002/s i008/s i009/s i011/s i012/s
12:20:01 AM 0 512.00 0.00 0.00 0.00 3.73 0.00 0.00
…
10:30:01 AM CPU i000/s i001/s i002/s i003/s i008/s i009/s i010/s
10:40:02 AM 0 512.00 1.67 0.00 0.00 0.00 15.08 0.00
Average: 0 512.00 0.42 0.00 N/A 0.00 6.03 N/A
|
This report (which has been truncated horizontally to fit on the
page) includes one column for each interrupt level (for example, the
i002/s field illustrating the rate for interrupt
level 2). If this were a multiprocessor system, there would be one
line per sample period for each CPU.
Another important point to note about this report is that
sar adds or removes specific interrupt fields if no
data is collected for that field. The example report above provides
an example of this, the end of the report includes interrupt levels (3
and 10) that were not present at the start of the sampling
period.
 | Note |
|---|
| | There are two other interrupt-related sar
reports — sar -I ALL and sar -I
XALL. However, the default configuration for the
sadc data collection utility does not collect the
information necessary for these reports. This can be changed by
editing the file /etc/cron.d/sysstat, and
changing this line: */10 * * * * root /usr/lib/sa/sa1 1 1
|
to this: */10 * * * * root /usr/lib/sa/sa1 -I 1 1
|
Keep in mind this change does cause additional information to be
collected by sadc, and results in larger data
file sizes. Therefore, make sure your system configuration can
support the additional space consumption. |
