Due to the inherent complexity of being a demand-paged virtual
memory operating system, monitoring memory-related resources under Red Hat Enterprise Linux
can be confusing. Therefore, it is best to start with the more
straightforward tools, and work from there.
Using free, it is possible to get a concise (if
somewhat simplistic) overview of memory and swap utilization. Here is
an example:
total used free shared buffers cached
Mem: 1288720 361448 927272 0 27844 187632
-/+ buffers/cache: 145972 1142748
Swap: 522104 0 522104
|
We note that this system has 1.2GB of RAM, of which only about 350MB
is actually in use. As expected for a system with this much free RAM,
none of the 500MB swap partition is in use.
Contrast that example with this one:
total used free shared buffers cached
Mem: 255088 246604 8484 0 6492 111320
-/+ buffers/cache: 128792 126296
Swap: 530136 111308 418828
|
This system has about 256MB of RAM, the majority of which is in use,
leaving only about 8MB free. Over 100MB of the 512MB swap partition is
in use. Although this system is certainly more limited in terms of
memory than the first system, to determine if this memory limitation is
causing performance problems we must dig a bit deeper.
Although more cryptic than free,
vmstat has the benefit of displaying more than memory
utilization statistics. Here is the output from vmstat 1
10:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 111304 9728 7036 107204 0 0 6 10 120 24 10 2 89
2 0 0 111304 9728 7036 107204 0 0 0 0 526 1653 96 4 0
1 0 0 111304 9616 7036 107204 0 0 0 0 552 2219 94 5 1
1 0 0 111304 9616 7036 107204 0 0 0 0 624 699 98 2 0
2 0 0 111304 9616 7052 107204 0 0 0 48 603 1466 95 5 0
3 0 0 111304 9620 7052 107204 0 0 0 0 768 932 90 4 6
3 0 0 111304 9440 7076 107360 92 0 244 0 820 1230 85 9 6
2 0 0 111304 9276 7076 107368 0 0 0 0 832 1060 87 6 7
3 0 0 111304 9624 7092 107372 0 0 16 0 813 1655 93 5 2
2 0 2 111304 9624 7108 107372 0 0 0 972 1189 1165 68 9 23
|
During this 10-second sample, the amount of free memory (the
free field) varies somewhat, and there
is a bit of swap-related I/O (the si
and so fields), but overall this system
is running well. It is doubtful, however, how much additional workload
it could handle, given the current memory utilization.
When researching memory-related issues, it is often necessary to
determine how the Red Hat Enterprise Linux virtual memory subsystem is making use of
system memory. By using sar, it is possible to
examine this aspect of system performance in much more detail.
By reviewing the sar -r report, we can examine
memory and swap utilization more closely:
Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/22/2003
12:00:01 AM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached
12:10:00 AM 240468 1048252 81.34 0 133724 485772
12:20:00 AM 240508 1048212 81.34 0 134172 485600
…
08:40:00 PM 934132 354588 27.51 0 26080 185364
Average: 324346 964374 74.83 0 96072 467559
|
The kbmemfree and
kbmemused fields show the typical free
and used memory statistics, with the percentage of memory used displayed
in the %memused field. The
kbbuffers and
kbcached fields show how many kilobytes
of memory are allocated to buffers and the system-wide data
cache.
The kbmemshrd field is always zero
for systems (such as Red Hat Enterprise Linux) using the 2.4 Linux kernel.
The lines for this report have been truncated to fit on the page.
Here is the remainder of each line, with the timestamp added to the left
to make reading easier:
12:00:01 AM kbswpfree kbswpused %swpused
12:10:00 AM 522104 0 0.00
12:20:00 AM 522104 0 0.00
…
08:40:00 PM 522104 0 0.00
Average: 522104 0 0.00
|
For swap utilization, the kbswpfree
and kbswpused fields show the amount of
free and used swap space, in kilobytes, with the
%swpused field showing the swap space
used as a percentage.
To learn more about the swapping activity taking place, use the
sar -W report. Here is an example:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003
12:00:01 AM pswpin/s pswpout/s
12:10:01 AM 0.15 2.56
12:20:00 AM 0.00 0.00
…
03:30:01 PM 0.42 2.56
Average: 0.11 0.37
|
Here we notice that, on average, there were three times fewer pages
being brought in from swap (pswpin/s)
as there were going out to swap
(pswpout/s).
To better understand how pages are being used, refer to the
sar -B report:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003
12:00:01 AM pgpgin/s pgpgout/s activepg inadtypg inaclnpg inatarpg
12:10:00 AM 0.03 8.61 195393 20654 30352 49279
12:20:00 AM 0.01 7.51 195385 20655 30336 49275
…
08:40:00 PM 0.00 7.79 71236 1371 6760 15873
Average: 201.54 201.54 169367 18999 35146 44702
|
Here we can determine how many blocks per second are paged in from
disk (pgpgin/s) and paged out to disk
(pgpgout/s). These statistics serve as
a barometer of overall virtual memory activity.
However, more knowledge can be gained by examining the other fields
in this report. The Red Hat Enterprise Linux kernel marks all pages as either active or
inactive. As the names imply, active pages are currently in use in some
manner (as process or buffer pages, for example), while inactive pages
are not. This example report shows that the list of active pages (the
activepg field) averages approximately
660MB.
The remainder of the fields in this report concentrate on the
inactive list — pages that, for one reason or another, have not
recently been used. The inadtypg field
shows how many inactive pages are dirty
(modified) and may need to be written to disk. The
inaclnpg field, on the other hand,
shows how many inactive pages are clean
(unmodified) and do not need to be written to disk.
The inatarpg field represents the
desired size of the inactive list. This value is calculated by the
Linux kernel and is sized such that the inactive list remains large
enough to act as a pool for page replacement purposes.
For additional insight into page status (specifically, how often
pages change status), use the sar -R report. Here is
a sample report:
Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com) 07/22/2003
12:00:01 AM frmpg/s shmpg/s bufpg/s campg/s
12:10:00 AM -0.10 0.00 0.12 -0.07
12:20:00 AM 0.02 0.00 0.19 -0.07
…
08:50:01 PM -3.19 0.00 0.46 0.81
Average: 0.01 0.00 -0.00 -0.00
|
The statistics in this particular sar report are
unique, in that they may be positive, negative, or zero. When positive,
the value indicates the rate at which pages of this type are
increasing. When negative, the value indicates the rate at which pages
of this type are decreasing. A value of zero indicates that pages of
this type are neither increasing or decreasing.
In this example, the last sample shows slightly over three pages per
second being allocated from the list of free pages (the
frmpg/s field) and nearly 1 page per
second added to the page cache (the
campg/s field). The list of pages used
as buffers (the bufpg/s field) gained
approximately one page every two seconds, while the shared memory page
list (the shmpg/s field) neither gained
nor lost any pages.
