Use sar to monitor resources in Linux

---

This article presents 10 ways that you can use the System Activity Report (sar) command to monitor resources in Linux®.

1. Display the collective usage of all CPUs

Run the following command to display a summary of CPU usage:

sar -u

This command outputs the cumulative, real-time resource use of all CPUs.

You can add more parameters to this command. For example, adding 1 3 at the end of the command as shown in the following example reports for every 1 second a total of 3 times:

sar -u 1 3

The output is similar to the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:27:32 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
01:27:33 PM       all      0.00      0.00      0.00      0.00      0.00    100.00
01:27:34 PM       all      0.25      0.00      0.25      0.00      0.00     99.50
01:27:35 PM       all      0.75      0.00      0.25      0.00      0.00     99.00
Average:          all      0.33      0.00      0.17      0.00      0.00     99.50

The last field, %idle, indicates the CPU load.

The command has the following variations:

• sar -u: Displays CPU usage for the current day.
• sar -u 1 3: Displays real-time CPU usage every 1 second for 3 times.
• sar -u ALL: Same as the sar -u command, but displays additional fields.
• sar -u ALL 1 3: Same as the sar -u 1 3 command, but displays additional fields.
• sar -u -f /var/log/sa/sa10: Displays CPU usage for the tenth day of the month from the sa10 file.

2. Display CPU usage by CPU or core

Run the following command to display CPU usage by CPU or core:

sar -P ALL 1 1

The -P ALL flag indicates that you want to display statistics for all of the individual cores.

In the CPU column of the following example output, 0, 1, 2, and 3 are CPU core numbers:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:34:12 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
01:34:13 PM       all     11.69      0.00      4.71      0.69      0.00     82.90
01:34:13 PM         0     35.00      0.00      6.00      0.00      0.00     59.00
01:34:13 PM         1     22.00      0.00      5.00      0.00      0.00     73.00
01:34:13 PM         2      3.00      0.00      1.00      0.00      0.00     96.00
01:34:13 PM         3      0.00      0.00      0.00      0.00      0.00    100.00

If you want to display statistics for only the second core, use the -P 1 flag (because core numbers start from 0), as shown in the following example:

sar -P 1 1 1

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:36:25 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
01:36:26 PM         1      8.08      0.00      2.02      1.01      0.00     88.89

You might also find the following variations helpful:

• sar -P ALL: Displays CPU usage for each core for the current day.
• sar -P ALL 1 3: Displays real-time CPU usage of each core every 1 second for 3 times.
• sar -P 1: Displays CPU usage for core number 1 for the current day.
• sar -P 1 1 3: Displays real-time CPU usage for core number 1, every 1 second for 3 times.
• sar -P ALL -f /var/log/sa/sa10: Displays CPU usage for each core for the tenth day of the month from the sa10 file.

3. Display free and used memory

Run the following command to display free and used memory:

sar -r

If you add 1 3 to the command as shown in the following example, the output reports for every 1 second a total of 3 times:

sar -r 1 3

The output looks like the following example, in which kbmemfree is free memory and kbmemused is used memory:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

07:28:06 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact
07:28:07 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204
07:28:08 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204
07:28:09 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204
Average:      6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204

This command has the following variations:

• sar -r
• sar -r 1 3
• sar -r -f /var/log/sa/sa10

4. Display swap space use

Run the following command to display the amount of swap space that is in use:

sar -S

If you add 1 3 to the end of the command as shown in the following example, the output reports for every 1 second a total of 3 times:

sar -S 1 3

The output is similar to the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

07:31:06 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
07:31:07 AM   8385920         0      0.00         0      0.00
07:31:08 AM   8385920         0      0.00         0      0.00
07:31:09 AM   8385920         0      0.00         0      0.00
Average:      8385920         0      0.00         0      0.00

Note: If the kbswpused and %swpused are at 0, then your system is not swapping.

This command has the following variations:

• sar -S
• sar -S 1 3
• sar -S -f /var/log/sa/sa10

Additional swap space commands

You might also find the following swap space commands helpful:

• Use sar -R to identify the number of memory pages that the system has freed, used, and cached.
• Use sar -H to identify the Linux® HugePages (in kilobytes (KB)) that are in use and available.
• Use sar -B to generate paging statistics such as the number of KB paged in (and out) from disk per second.
• Use sar -W to generate page swap statistics, such as page swap in (and out) per second.

5. Display I/O activities

Run the following command to display overall input and output (I/O) activities:

sar -b

The output from this command displays I/O statistics. If you add 1 3 to the end of the command as shown in the following example, the output reports for every 1 second a total of 3 times:

sar -b 1 3

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:56:28 PM       tps      rtps      wtps   bread/s   bwrtn/s
01:56:29 PM    346.00    264.00     82.00   2208.00    768.00
01:56:30 PM    100.00     36.00     64.00    304.00    816.00
01:56:31 PM    282.83     32.32    250.51    258.59   2537.37
Average:       242.81    111.04    131.77    925.75   1369.90

The output includes the following fields:

• tps: Transactions per second (both read and write)
• rtps: Read transactions per second
• wtps: Write transactions per second
• bread/s: Bytes read per second
• bwrtn/s: Bytes written per second

This command has the following variations:

• sar -b
• sar -b 1 3
• sar -b -f /var/log/sa/sa10

Note: Use sar -v to display the number of inode handlers, file handlers, and pseudo-terminals that the system is using.

6. Display the I/O activities of individual block devices

Run the following command to display the I/O activities of individual block devices:

sar -d

To identify activities by individual block device (such as a specific mount point, logical unit number (LUN), or partition), use sar -d, as shown in the following example:

sar -d 1 1

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:59:45 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:59:46 PM    dev8-0      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.10
01:59:46 PM    dev8-1      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.10
01:59:46 PM dev120-64      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.62
01:59:46 PM dev120-65      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.62
01:59:46 PM  dev120-0      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.30
01:59:46 PM  dev120-1      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.30
01:59:46 PM dev120-96      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.91
01:59:46 PM dev120-97      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.91

In this example, the values in the DEV column represent specific block devices. For example, dev53-1 means a block device with 53 as major number, and 1 as minor number.

You can use the pretty print (-p) flag as shown in the following example to make the column display the actual device name (such as sda, sda1, or sdb1):

sar -p -d 1 1

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:59:45 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:59:46 PM       sda      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.10
01:59:46 PM      sda1      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.10
01:59:46 PM      sdb1      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.62
01:59:46 PM      sdc1      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.62
01:59:46 PM      sde1      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.30
01:59:46 PM      sdf1      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.30
01:59:46 PM      sda2      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.91
01:59:46 PM      sdb2      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.91

This command has the following variations:

• sar -d
• sar -d 1 3
• sar -d -f /var/log/sa/sa10
• sar -p -d

7. Display the context switch per second

Run the following command to display the context switch per second:

sar -w

This command reports the total number of processes created per second and the total number of context switches per second. If you add 1 3 to the end of the command as shown in the following example, it reports for every 1 second a total of 3 times:

sar -w 1 3

The output is similar to the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

08:32:24 AM    proc/s   cswch/s
08:32:25 AM      3.00     53.00
08:32:26 AM      4.00     61.39
08:32:27 AM      2.00     57.00

This command has the following variations:

• sar -w
• sar -w 1 3
• sar -w -f /var/log/sa/sa10

8. Display the run queue and load average

Run the following command to display the run queue and load average:

sar -q

This command reports the run queue size and load average for the past minute, past 5 minutes, and past 15 minutes. If you add 1 3 to the end of the command as shown in the following example, it reports for every 1 second a total of 3 times:

sar -q 1 3

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

06:28:53 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
06:28:54 AM         0       230      2.00      3.00      5.00         0
06:28:55 AM         2       210      2.01      3.15      5.15         0
06:28:56 AM         2       230      2.12      3.12      5.12         0
Average:            3       230      3.12      3.12      5.12         0

Note: The blocked column displays the number of tasks that are currently blocked and waiting for I/O operations to complete.

This command has the following variations:

• sar -q
• sar -q 1 3
• sar -q -f /var/log/sa/sa10

9. Display network statistics

Run the following command to display network statistics:

sar -n

This command reports various network statistics, such as the number of packets received (transmitted) through the network card or the number of packet failures that occurred. If you add 1 3 to the end of the command, it reports for every 1 second a total of 3 times.

You can also add a keyword to the command, as shown in the following example:

sar -n KEYWORD

KEYWORD can be one of the following options:

• DEV: Displays vital statistics for network devices
• EDEV: Display failure statistics for network devices
• NFS: Displays Network File System (NFS) client activities
• NFSD: Displays NFS server activities
• SOCK: Displays the sockets that are in use for IPv4
• IP: Displays IPv4 network traffic
• EIP: Displays IPv4 network errors
• ICMP: Displays ICMPv4 network traffic
• EICMP: Displays ICMPv4 network errors
• TCP: Displays TCPv4 network traffic
• ETCP: Displays TCPv4 network errors
• UDP: Displays UDPv4 network traffic
• SOCK6, IP6, EIP6, ICMP6, and UDP6: Displays IPv6 network information
• ALL: Displays all of the preceding information

The following example uses the DEV keyword:

sar -n DEV 1 1

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:11:13 PM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
01:11:14 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:11:14 PM      eth0    342.57    342.57  93923.76 141773.27      0.00      0.00      0.00
01:11:14 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10. Report sar data by start time

Run the following command to report sar data by start time:

sar -s

When you view historical sar data from the /var/log/sa/saXX file by using the sar -f option, the output displays all of the sar data for that specific day beginning at 12:00 a.m.

You can use the -s hh:mi:ss option to specify the start time. For example, if you specify sar -s 10:00:00, the output from the command displays the sar data beginning at 10 a.m. (instead of the default of midnight).

You can combine the -s flag with other sar flags. For example, to report the load average on the 26th of the current month starting from 10 a.m., combine the -q and -s flags as shown in the following example:

sar -q -f /var/log/sa/sa23 -s 10:00:01

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

10:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:10:01 AM         0       127      2.00      3.00      5.00         0
10:20:01 AM         0       127      2.00      3.00      5.00         0
...
11:20:01 AM         0       127      5.00      3.00      3.00         0
12:00:01 PM         0       127      4.00      2.00      1.00         0

You cannot specify an end time. Instead, you need to use the head command. For example, if you want to start from 10 a.m. and see 7 entries, you pipe the preceding output to head -n 10, as shown in the following example:

sar -q -f /var/log/sa/sa23 -s 10:00:01 | head -n 10

The output looks like the following example:

Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

10:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:10:01 AM         0       127      2.00      3.00      5.00         0
10:20:01 AM         0       127      2.00      3.00      5.00         0
10:30:01 AM         0       127      3.00      5.00      2.00         0
10:40:01 AM         0       127      4.00      2.00      1.00         2
10:50:01 AM         0       127      3.00      5.00      5.00         0
11:00:01 AM         0       127      2.00      1.00      6.00         0
11:10:01 AM         0       127      1.00      3.00      7.00         2