Linux Out-of-Memory Killer

Every Linux® distribution has the Out-of-Memory (OOM) Killer process included
in it, but what is it? Simply put, this is the server’s self-preservation
process. To fully understand what that means, consider how Linux allocates memory.

Linux memory allocation

The Linux kernel allocates memory on an on-demand basis for all applications
currently running on the server. Because this generally happens upfront,
applications usually don’t use all the assigned memory. This allows the
kernel to over-commit memory, making memory more efficient. This over-commitment
allows the kernel to commit more memory than is actually physically
available. Typically, this is not an issue. The problem occurs when too many
applications start using the memory allotted to them at once. The server
runs the risk of crashing because it ran out of memory. To prevent the
server from reaching that critical state, the kernel also contains a process
known as the OOM Killer. The kernel uses this process to start killing
non-essential processes so the server can remain operational.

While you might think this should not be a problem, the OOM Killer
kills processes that the server has deemed non-essential, not the user. For
example, the two applications the OOM Killer usually kills first are Apache® and
MySQL® because they use a large amount of memory. Anyone with a website
immediately knows that is a big problem. If the OOM Killer kills either of
those, a website often crashes immediately.

Why was a specific process killed?

When trying to find out why the OOM killer killed an application or process,
you can look for a few things that can help reveal how and why the
process was killed. The first place to look is in the syslog by running
the following command:

$ grep -i kill /var/log/messages*
host kernel: Out of Memory: Killed process 5123 (exampleprocess)

You should get output similar to the preceding example. The capital K in
Killed tells you that the process was killed with a -9 signal, and this
typically is a good indicator that the OOM Killer is to blame.

Additionally, you can run the following command to check the server’s high
and low memory statistics:

$ free -lh

The -l switch shows high and low memory statistics, and the -h switch puts
the output into gigabytes for easier human readability. You can change this
to the -m switch if you prefer the output in megabytes. An added benefit of
this command is it gives you the Swap memory usage information as well.
One caveat is that the free command provides only a snapshot of this moment,
so you need to check it multiple times to get an idea of what is happening.

Fortunately, the vmstat command obtains memory output over a period
of time, and it even has an option for an easy-to-read table:

$ vmstat -SM 10 20

The preceding command outputs system memory information twenty times at 10-second
intervals. That is what the 10 and 20 mean in the preceding example. You can change
both of these numbers to fit a frequency and total that better
suits your needs. The -S switch displays the output in a table
format, and the -M switch shows the output in megabytes. Use this command to
show what is actively going on throughout the time parameters you
specify.

Another good tool to use is, of course, the top command. top orders output by
the CPU variable by default, but if you click shift + M after running the top
command, you can get real-time updates for memory usage instead of CPU usage.

Configure the OOM Killer

Because the OOM Killer is a process, you can configure it to fit your needs better.
In fact, the OOM Killer already has several configuration options baked in that allow
server administrators and developers to choose how they want the OOM Killer process to
behave when faced with a memory-is-getting-dangerously-low situation. Keep in
mind that these options can vary depending on factors such as environment and
running applications.

As with anything involving changing configurations, it is always
better to test proposed changes in a development or staging environment
before making those changes in a live production environment. This way,
you know how the system reacts to those changes. Finally, even if you’re
confident of your plan, always make a backup before making any changes.
For the following configuration options, you must be the root user.

Option 1: Reboot

The first option involves editing the sysctl configuration
(/etc/sysctl.conf), which allows your changes to persist between reboots:

sysctl vm.panic_on_oom=1
sysctl kernel.panic=X
echo “vm.panic_on_oom=1” >> /etc/sysctl.conf
echo “kernel.panic=X” >> /etc/sysctl.conf

The X in the preceding command is the number of seconds you want the system to
wait before it reboots.

In most situations, it's not feasible to reboot every time the
system gets critically low on memory. While this approach might be necessary
for some situations, most do not need or warrant an entire system reboot
to address the issue.

Option 2: Protect or sacrifice processes

This particular option requires a more fine-honed approach. You can either
(a) protect certain processes by making them less likely to be killed by
the OOM Killer or (b) set certain processes to be more likely to be
killed. You can accomplish this with the following commands:

echo -15 > /proc/(PID)/oom_adj			(less likely)
echo 10 > /proc/(PID)/oom_adj			(more likely)

Replace the (PID) placeholder in the sample command with the particular process’s
ID (or PID) you are interested in. To protect or sacrifice a process, you need to find
the parent process (the original). Use the following command to locate the PPID
(or parent process ID), where you replace the process with your process (such as Apache,
MySQL, and so on):

pstree -p | grep "process" | head -1

You can see that this option is a little better than the nuclear option of an
entire system reboot. However, what if you have a process that is crucial
and cannot be killed?

Option 3: Exempt a process

This option comes with a cautionary note. Exempting processes can, in
some circumstances, cause unintended behavior changes, which largely
depend on the system and resource configurations. If the kernel cannot
kill a process using a large amount of memory, it will start killing other
available processes. This can include processes that also might be important
operating system processes. The system could potentially go down
completely as a result. Suffice it to say, use this option with extreme
caution.

Because the valid range for OOM Killer adjustments is between -16 and +15, a
setting of -17 exempts a process entirely because it falls outside
the scope of acceptable integers for the OOM Killer’s adjustment scale. The
general rule is: the higher the numerical value, the more likely a process
is picked to be killed. Therefore, the command to completely exempt a
process is:

echo -17 > /proc/(PID)/oom_adj

Option 4: The risky option

Warning: Rackspace does not recommend this for production environments.

If reboots and protecting, sacrificing, or exempting processes just aren’t good
enough, there is the final, risky option: the disable OOM Killer completely option.

This option can cause any of the following results:

• a serious kernel panic
• a system hang-up
• a full system crash

Why? It prevents the server from keeping itself from running out of resources.
If you disable the OOM Killer completely, then nothing protects the server from
running out of memory. Use extreme restraint and caution when considering this option.

To exercise this option, run the following command:

sysctl vm.overcommit_memory=2
echo “vm.overcommit_memory=2” >> /etc/sysctl.conf

Now that you've learned about the OOM Killer, you know how to
tailor the process to your individual environment and system needs. As a
general rule, exercise caution whenever you edit kernel
processes. The OOM Killer is no exception to that rule.