We earn commissions when you shop through the links below, at no extra cost to you. Our guides and technical recommendations are written independently of these relationships: a commission never decides what we recommend, how we diagnose a problem or what we publish, and the content does not depend on it. Affiliate disclosure

Linux · Intermediate · 15 min

Find what the Linux OOM killer killed

Find what the Linux OOM killer killed and why: read Killed process in the kernel log, tell a cgroup limit from a full machine, and fix the cause.

A process killed for memory leaves no log of its own, only Killed and exit status 137. This runbook reads the kernel log to find the victim, tells a memory limit from a machine that ran out, ranks what was using memory at that moment, and explains how the killer chooses. Commands were run on Ubuntu 26.04 with kernel 6.6 and systemd 259, and the RHEL-family log locations on AlmaLinux 10.

See it run

A short replay of real output from commands in this guide, captured on a WSL2 Ubuntu test machine (run as root; the memory hog ran in a throwaway scope and service limited to 100 MB, so the rest of the machine was not touched). The typing is replayed, and the host name, account and MAC addresses are redacted. Your output will differ, and a test machine is not a production server.

Text version of this demo
ops@web-01:~$ systemd-run --scope -q -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py; echo "exit status: $?"
bash: line 8:  1006 Killed                     systemd-run --scope -q -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py
exit status: 137

ops@web-01:~$ sudo journalctl -k -g 'Killed process' -o short-iso --no-pager | tail -n 1
2026-10-31T15:39:13+02:00 web-01 kernel: Memory cgroup out of memory: Killed process 88532 (python3) total-vm:118480kB, anon-rss:102016kB, file-rss:6656kB, shmem-rss:0kB, UID:0 pgtables:272kB oom_score_adj:0

ops@web-01:~$ sudo journalctl -k -g 'oom-kill:' -o cat --no-pager | tail -n 1 | grep -oE 'constraint=[A-Z_]+|task_memcg=[^,]+|task=[^,]+'
constraint=CONSTRAINT_MEMCG
task_memcg=/system.slice/run-rf5afbb2d22f747829e3b1767042114b6.scope
task=python3

ops@web-01:~$ sudo systemd-run -q --unit=ob-hog -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py; sleep 3; systemctl status ob-hog --no-pager | sed -n 1,4p
× ob-hog.service - [systemd-run] /usr/bin/python3 /tmp/ob-hog.py
     Loaded: loaded (/run/systemd/transient/ob-hog.service; transient)
  Transient: yes
     Active: failed (Result: oom-kill) since Sat 2026-10-31 15:39:14 EET; 2s ago

ops@web-01:~$ journalctl -u ob-hog -o cat --no-pager | grep -E 'OOM|oom-kill' | tail -n 2
ob-hog.service: The kernel OOM killer killed some processes in this unit.
ob-hog.service: Failed with result 'oom-kill'.

When Linux runs out of memory, the kernel's OOM (out-of-memory) killer ends one process with SIGKILL to free memory. The process gets no chance to log anything, so the application log just stops, the shell prints Killed, and the exit status is 137. The evidence is in the kernel log: which process died, which limit was hit, and what else was using memory at that moment. This runbook reads that evidence and works out whether to fix the process, the limit or the machine.

This is the guide for one of the six incident classes in the Linux server troubleshooting runbook, which sets out the order of work they all share.

Commands and output were captured on an Ubuntu 26.04 test machine (systemd 259) with a small Python script that keeps allocating memory. The kills were made safe on purpose: the script ran in a scope and a service limited to 100 MB, and the one machine-wide kill was triggered by hand on a process marked as the preferred victim. The RHEL-family notes were checked on 2026-11-08 on AlmaLinux 10.2 (a RHEL 10 rebuild, systemd 257), with the limited kills only.

0. Rules of engagement

  1. Read the log before you restart anything. The kernel log in RAM (dmesg) does not survive a reboot. The journal only keeps it if it is persistent (section 1).
  2. Find what used the memory before you add more. A leak will fill a bigger machine too, only later.
  3. Do not exempt processes from the killer to make the problem go away. If the kernel cannot kill the leaking process, it will kill something else, or the machine will stall.

1. Find the kill in the kernel log

# Every OOM kill in the kernel log of the current boot, with readable times
sudo journalctl -k -b -g 'Killed process' -o short-iso --no-pager
# Real output from the test machine: one kill inside a memory limit (a cgroup), one machine-wide kill
2026-10-31T15:35:16+02:00 web-01 kernel: Memory cgroup out of memory: Killed process 85101 (python3) total-vm:118480kB, anon-rss:102016kB, file-rss:6912kB, shmem-rss:0kB, UID:0 pgtables:272kB oom_score_adj:0
2026-10-31T15:35:21+02:00 web-01 kernel: Out of memory: Killed process 85110 (python3) total-vm:405308kB, anon-rss:392320kB, file-rss:6656kB, shmem-rss:0kB, UID:0 pgtables:836kB oom_score_adj:1000

The process name is in brackets, and anon-rss is roughly the memory it had in use when it died.

  • No journal access? sudo dmesg -T | grep -i 'killed process' reads the same messages from the kernel buffer. The -T times are calculated, and can be wrong after a suspend and resume. Trust the journal times.
  • The kill was before the last reboot? journalctl -k -b -1 reads the previous boot, but only if the journal is persistent (/var/log/journal/ exists). Without that, the evidence is gone. Create the directory now so the next one is kept.
  • RHEL, AlmaLinux, Rocky: on the AlmaLinux 10.2 test machine /var/log/journal/ did not exist, so the journal lived in RAM only (/run/log/journal/). There, rsyslog keeps the kernel messages across reboots: sudo grep 'Killed process' /var/log/messages found the same line as journalctl -k.
  • Nothing found, and the process still vanished? Look for a user-space killer: journalctl -u systemd-oomd (on by default on Ubuntu desktops; a separate package, not installed, on AlmaLinux 10) or earlyoom. They kill earlier, on memory pressure, and log in their own unit, not in the kernel log.

2. Was it the whole machine or a limit?

The first words of the message say it, and the oom-kill: line just before it gives the detail.

# The constraint and the cgroup of the last kill
sudo journalctl -k -g 'oom-kill:' -o cat --no-pager | tail -n 1 | grep -oE 'constraint=[A-Z_]+|task_memcg=[^,]+|task=[^,]+'
# Real output from the test machine: the kill inside the service limited to 100 MB
constraint=CONSTRAINT_MEMCG
task_memcg=/system.slice/ob-hog.service
task=python3
Messageconstraint=What ran outWhat to look at
Memory cgroup out of memory: Killed processCONSTRAINT_MEMCGA memory limit: a systemd MemoryMax=, a container --memory, a Kubernetes limitThe limit and the process inside it (section 4). The rest of the machine may have had plenty of memory
Out of memory: Killed processCONSTRAINT_NONE, with global_oomRAM and swap of the whole machineEverything that used memory at that moment (section 3)

task_memcg (and oom_memcg for the limit that was hit) names the cgroup, which tells you the systemd unit or the container. /system.slice/ob-hog.service is a service; a path with docker-<id>.scope is a container, and kubepods is a Kubernetes Pod.

One more line is easy to misread: python3 invoked oom-killer names the process that asked for memory when none was left. That is not necessarily the one that was killed, and not necessarily the one that used the most. The victim is only in Killed process.

3. See what else was using memory

With every kill, the kernel writes a table of the processes in the affected cgroup (or the whole machine) and their memory. It is the best record you will get of what was using memory, because it was taken at the moment of the kill.

# The kernel's process table at the time of a kill, sorted by resident memory (rss, in 4 KiB pages)
sudo journalctl -k --since 'YYYY-MM-DD HH:MM:SS' --until 'YYYY-MM-DD HH:MM:SS' -o cat --no-pager | grep -E '^\[ *[0-9]+\]' | tr -d '[]' | sort -k5,5nr | head -n 5

Use a window of a few seconds around the kill time from section 1. The columns are pid uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name.

# Real output from the test machine around the machine-wide kill
   6589  1000  6589 21768435  1922609 45514752  2705339             0 python
  85110     0 85110   101327    99744   856064        0          1000 python3
  41281     0 41281   754228    11506   577536     2176          -500 dockerd
    293  1000   293   219084     9603   405504     5240             0 weston
    623     0   623   689532     7238   471040     1375          -999 containerd

This shows how the killer chooses. The biggest user by far was python (PID 6589): 1,922,609 pages, about 7.3 GiB, plus about 10 GiB in swap (swapents). The killer still chose python3 (PID 85110), which used about 390 MiB, because its oom_score_adj was 1000, which makes it the first choice. On a machine where nobody has set the adjustments, the biggest process is usually the one killed. So when a small process was killed, check its oom_score_adj and the biggest rows in this table: the real cause is often a different process.

The same messages include a Mem-Info: block and Free swap / Total swap lines. On the test machine they showed 12 GiB of swap still free during the manual kill. In a real machine-wide OOM, free swap is usually close to zero.

4. A service or a container hit its own limit

systemd records the result on the unit.

# Real output from the test machine (trimmed): systemctl status and the journal of a service started with MemoryMax=100M
× ob-hog.service - [systemd-run] /usr/bin/python3 /tmp/ob-hog.py
     Active: failed (Result: oom-kill) since Sat 2026-10-31 15:35:16 EET; 3s ago
ob-hog.service: The kernel OOM killer killed some processes in this unit.
ob-hog.service: Main process exited, code=killed, status=9/KILL
ob-hog.service: Failed with result 'oom-kill'.
# The memory limits of a unit and how much it used at its peak
systemctl show UNIT -p MemoryMax -p MemoryHigh -p MemoryPeak -p Result
  • The limit is too low for normal work: raise it in a drop-in (sudo systemctl edit UNIT, then [Service] and MemoryMax=), not in the vendor unit file.
  • The use keeps growing until it hits any limit: that is a leak or an unbounded cache. Raising the limit only moves the kill later. Fix the application, or cap its cache, queue or worker count.
  • MemoryHigh= slows a unit down and reclaims its memory before the hard limit, so it gives a warning stage instead of a sudden kill.
  • Containers: docker inspect --format '{{.State.OOMKilled}}' CONTAINER prints true after a kill inside the container's limit. In Kubernetes the reason is OOMKilled, see Troubleshoot a Kubernetes Pod that keeps restarting. The exit codes are explained in Container exit codes 137, 139 and 143.

From a shell, a process killed this way looks like this:

# Real output from the test machine: the script in a 100 MB scope, run from a script
bash: line 8:  1006 Killed                     systemd-run --scope -q -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py
exit status: 137

137 is 128 + 9: the process was killed by signal 9 (SIGKILL). The OOM killer is the most common source, but not the only one. Only the kernel log confirms it.

5. How the killer chooses, and how to steer it

Every process has an oom_score, mostly its share of memory, plus an oom_score_adj from -1000 to 1000 that you can set.

# The processes the killer would choose first, with their score, adjustment and resident memory in KiB
ps -eo pid,oom,oomadj,rss,comm --sort=-oom | head -n 6

The oom column is /proc/PID/oom_score. On current kernels it is scaled to 0–1333: a process that uses almost no memory and has no adjustment scores 666 (on the test machine, an idle shell did), each oom_score_adj point adds about two thirds of a point, and memory use adds the rest. Higher means the killer picks it sooner.

  • Protect a critical service a little, not completely: OOMScoreAdjust=-500 in its unit. Docker sets -500 for dockerd and containerd uses -999, as the table in section 3 shows.
  • -1000 exempts a process completely. Keep it for processes without which the machine cannot recover. An exempt process that leaks takes everything else down with it.
  • Mark disposable work as the first choice with a positive value, for example OOMScoreAdjust=500 for batch jobs.
  • vm.overcommit_memory: 0 (the default) lets programs reserve more memory than exists, using a heuristic. 2 refuses reservations above CommitLimit in /proc/meminfo, so programs get allocation errors instead of being killed later. Most software expects the default. Check yours with sysctl vm.overcommit_memory. It was 1 (always allow) on the test machine.
  • vm.panic_on_oom=1 reboots instead of killing a process. It only makes sense where a fast reboot by a watchdog is better than a half-working machine.

6. Verify

  • The process runs through its normal peak without a new Killed process line: sudo journalctl -k -b -g 'Killed process' --no-pager.
  • For a service, systemctl show UNIT -p MemoryPeak stays clearly below MemoryMax, and Result=success.
  • For a machine-wide OOM, free -h shows available memory and some free swap at the busiest time of day, not only right after a restart.

7. Rollback and prevention

  • Rollback: undo a changed limit or OOMScoreAdjust with sudo systemctl revert UNIT (it removes drop-ins), then sudo systemctl restart UNIT. A sysctl change goes back with sudo sysctl -w NAME=OLD_VALUE and by removing its file in /etc/sysctl.d/.
  • Give every service and container a memory limit, so a leak kills only the leaking process, not a random neighbor.
  • Make the journal persistent on servers (/var/log/journal/), or the evidence disappears with the reboot that follows a machine-wide OOM.
  • Alert before the kill: watch available memory and memory pressure (/proc/pressure/memory), as described in Check Linux CPU, memory, and disk pressure, and alert on new oom-kill events.
  • Add swap with care. A little swap gives the kernel room for memory that is rarely used. A lot of swap on a slow disk can turn a quick kill into a machine that is too slow to log in to.

References

Official reference

docs.kernel.org ↗

Technical steps are original explanations; consult the upstream documentation for version-specific details.

Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.

Found this useful?

Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.