Linux · Intermediate · 15 min
Find what the Linux OOM killer killed
Find what the Linux OOM killer killed and why: read Killed process in the kernel log, tell a cgroup limit from a full machine, and fix the cause.
A process killed for memory leaves no log of its own, only Killed and exit status 137. This runbook reads the kernel log to find the victim, tells a memory limit from a machine that ran out, ranks what was using memory at that moment, and explains how the killer chooses. Commands were run on Ubuntu 26.04 with kernel 6.6 and systemd 259, and the RHEL-family log locations on AlmaLinux 10.
On this page
See it run
A short replay of real output from commands in this guide, captured on a WSL2 Ubuntu test machine (run as root; the memory hog ran in a throwaway scope and service limited to 100 MB, so the rest of the machine was not touched). The typing is replayed, and the host name, account and MAC addresses are redacted. Your output will differ, and a test machine is not a production server.
Text version of this demo
ops@web-01:~$ systemd-run --scope -q -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py; echo "exit status: $?"
bash: line 8: 1006 Killed systemd-run --scope -q -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py
exit status: 137
ops@web-01:~$ sudo journalctl -k -g 'Killed process' -o short-iso --no-pager | tail -n 1
2026-10-31T15:39:13+02:00 web-01 kernel: Memory cgroup out of memory: Killed process 88532 (python3) total-vm:118480kB, anon-rss:102016kB, file-rss:6656kB, shmem-rss:0kB, UID:0 pgtables:272kB oom_score_adj:0
ops@web-01:~$ sudo journalctl -k -g 'oom-kill:' -o cat --no-pager | tail -n 1 | grep -oE 'constraint=[A-Z_]+|task_memcg=[^,]+|task=[^,]+'
constraint=CONSTRAINT_MEMCG
task_memcg=/system.slice/run-rf5afbb2d22f747829e3b1767042114b6.scope
task=python3
ops@web-01:~$ sudo systemd-run -q --unit=ob-hog -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py; sleep 3; systemctl status ob-hog --no-pager | sed -n 1,4p
× ob-hog.service - [systemd-run] /usr/bin/python3 /tmp/ob-hog.py
Loaded: loaded (/run/systemd/transient/ob-hog.service; transient)
Transient: yes
Active: failed (Result: oom-kill) since Sat 2026-10-31 15:39:14 EET; 2s ago
ops@web-01:~$ journalctl -u ob-hog -o cat --no-pager | grep -E 'OOM|oom-kill' | tail -n 2
ob-hog.service: The kernel OOM killer killed some processes in this unit.
ob-hog.service: Failed with result 'oom-kill'.When Linux runs out of memory, the kernel's OOM (out-of-memory) killer ends one process with SIGKILL to free memory. The process gets no chance to log anything, so the application log just stops, the shell prints Killed, and the exit status is 137. The evidence is in the kernel log: which process died, which limit was hit, and what else was using memory at that moment. This runbook reads that evidence and works out whether to fix the process, the limit or the machine.
This is the guide for one of the six incident classes in the Linux server troubleshooting runbook, which sets out the order of work they all share.
Commands and output were captured on an Ubuntu 26.04 test machine (systemd 259) with a small Python script that keeps allocating memory. The kills were made safe on purpose: the script ran in a scope and a service limited to 100 MB, and the one machine-wide kill was triggered by hand on a process marked as the preferred victim. The RHEL-family notes were checked on 2026-11-08 on AlmaLinux 10.2 (a RHEL 10 rebuild, systemd 257), with the limited kills only.
0. Rules of engagement
- Read the log before you restart anything. The kernel log in RAM (
dmesg) does not survive a reboot. The journal only keeps it if it is persistent (section 1). - Find what used the memory before you add more. A leak will fill a bigger machine too, only later.
- Do not exempt processes from the killer to make the problem go away. If the kernel cannot kill the leaking process, it will kill something else, or the machine will stall.
1. Find the kill in the kernel log
# Every OOM kill in the kernel log of the current boot, with readable times
sudo journalctl -k -b -g 'Killed process' -o short-iso --no-pager# Real output from the test machine: one kill inside a memory limit (a cgroup), one machine-wide kill
2026-10-31T15:35:16+02:00 web-01 kernel: Memory cgroup out of memory: Killed process 85101 (python3) total-vm:118480kB, anon-rss:102016kB, file-rss:6912kB, shmem-rss:0kB, UID:0 pgtables:272kB oom_score_adj:0
2026-10-31T15:35:21+02:00 web-01 kernel: Out of memory: Killed process 85110 (python3) total-vm:405308kB, anon-rss:392320kB, file-rss:6656kB, shmem-rss:0kB, UID:0 pgtables:836kB oom_score_adj:1000The process name is in brackets, and anon-rss is roughly the memory it had in use when it died.
- No journal access?
sudo dmesg -T | grep -i 'killed process'reads the same messages from the kernel buffer. The-Ttimes are calculated, and can be wrong after a suspend and resume. Trust the journal times. - The kill was before the last reboot?
journalctl -k -b -1reads the previous boot, but only if the journal is persistent (/var/log/journal/exists). Without that, the evidence is gone. Create the directory now so the next one is kept. - RHEL, AlmaLinux, Rocky: on the AlmaLinux 10.2 test machine
/var/log/journal/did not exist, so the journal lived in RAM only (/run/log/journal/). There, rsyslog keeps the kernel messages across reboots:sudo grep 'Killed process' /var/log/messagesfound the same line asjournalctl -k. - Nothing found, and the process still vanished? Look for a user-space killer:
journalctl -u systemd-oomd(on by default on Ubuntu desktops; a separate package, not installed, on AlmaLinux 10) orearlyoom. They kill earlier, on memory pressure, and log in their own unit, not in the kernel log.
2. Was it the whole machine or a limit?
The first words of the message say it, and the oom-kill: line just before it gives the detail.
# The constraint and the cgroup of the last kill
sudo journalctl -k -g 'oom-kill:' -o cat --no-pager | tail -n 1 | grep -oE 'constraint=[A-Z_]+|task_memcg=[^,]+|task=[^,]+'# Real output from the test machine: the kill inside the service limited to 100 MB
constraint=CONSTRAINT_MEMCG
task_memcg=/system.slice/ob-hog.service
task=python3| Message | constraint= | What ran out | What to look at |
|---|---|---|---|
Memory cgroup out of memory: Killed process | CONSTRAINT_MEMCG | A memory limit: a systemd MemoryMax=, a container --memory, a Kubernetes limit | The limit and the process inside it (section 4). The rest of the machine may have had plenty of memory |
Out of memory: Killed process | CONSTRAINT_NONE, with global_oom | RAM and swap of the whole machine | Everything that used memory at that moment (section 3) |
task_memcg (and oom_memcg for the limit that was hit) names the cgroup, which tells you the systemd unit or the container. /system.slice/ob-hog.service is a service; a path with docker-<id>.scope is a container, and kubepods is a Kubernetes Pod.
One more line is easy to misread: python3 invoked oom-killer names the process that asked for memory when none was left. That is not necessarily the one that was killed, and not necessarily the one that used the most. The victim is only in Killed process.
3. See what else was using memory
With every kill, the kernel writes a table of the processes in the affected cgroup (or the whole machine) and their memory. It is the best record you will get of what was using memory, because it was taken at the moment of the kill.
# The kernel's process table at the time of a kill, sorted by resident memory (rss, in 4 KiB pages)
sudo journalctl -k --since 'YYYY-MM-DD HH:MM:SS' --until 'YYYY-MM-DD HH:MM:SS' -o cat --no-pager | grep -E '^\[ *[0-9]+\]' | tr -d '[]' | sort -k5,5nr | head -n 5Use a window of a few seconds around the kill time from section 1. The columns are pid uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name.
# Real output from the test machine around the machine-wide kill
6589 1000 6589 21768435 1922609 45514752 2705339 0 python
85110 0 85110 101327 99744 856064 0 1000 python3
41281 0 41281 754228 11506 577536 2176 -500 dockerd
293 1000 293 219084 9603 405504 5240 0 weston
623 0 623 689532 7238 471040 1375 -999 containerdThis shows how the killer chooses. The biggest user by far was python (PID 6589): 1,922,609 pages, about 7.3 GiB, plus about 10 GiB in swap (swapents). The killer still chose python3 (PID 85110), which used about 390 MiB, because its oom_score_adj was 1000, which makes it the first choice. On a machine where nobody has set the adjustments, the biggest process is usually the one killed. So when a small process was killed, check its oom_score_adj and the biggest rows in this table: the real cause is often a different process.
The same messages include a Mem-Info: block and Free swap / Total swap lines. On the test machine they showed 12 GiB of swap still free during the manual kill. In a real machine-wide OOM, free swap is usually close to zero.
4. A service or a container hit its own limit
systemd records the result on the unit.
# Real output from the test machine (trimmed): systemctl status and the journal of a service started with MemoryMax=100M
× ob-hog.service - [systemd-run] /usr/bin/python3 /tmp/ob-hog.py
Active: failed (Result: oom-kill) since Sat 2026-10-31 15:35:16 EET; 3s ago
ob-hog.service: The kernel OOM killer killed some processes in this unit.
ob-hog.service: Main process exited, code=killed, status=9/KILL
ob-hog.service: Failed with result 'oom-kill'.# The memory limits of a unit and how much it used at its peak
systemctl show UNIT -p MemoryMax -p MemoryHigh -p MemoryPeak -p Result- The limit is too low for normal work: raise it in a drop-in (
sudo systemctl edit UNIT, then[Service]andMemoryMax=), not in the vendor unit file. - The use keeps growing until it hits any limit: that is a leak or an unbounded cache. Raising the limit only moves the kill later. Fix the application, or cap its cache, queue or worker count.
MemoryHigh=slows a unit down and reclaims its memory before the hard limit, so it gives a warning stage instead of a sudden kill.- Containers:
docker inspect --format '{{.State.OOMKilled}}' CONTAINERprintstrueafter a kill inside the container's limit. In Kubernetes the reason isOOMKilled, see Troubleshoot a Kubernetes Pod that keeps restarting. The exit codes are explained in Container exit codes 137, 139 and 143.
From a shell, a process killed this way looks like this:
# Real output from the test machine: the script in a 100 MB scope, run from a script
bash: line 8: 1006 Killed systemd-run --scope -q -p MemoryMax=100M -p MemorySwapMax=0 python3 /tmp/ob-hog.py
exit status: 137137 is 128 + 9: the process was killed by signal 9 (SIGKILL). The OOM killer is the most common source, but not the only one. Only the kernel log confirms it.
5. How the killer chooses, and how to steer it
Every process has an oom_score, mostly its share of memory, plus an oom_score_adj from -1000 to 1000 that you can set.
# The processes the killer would choose first, with their score, adjustment and resident memory in KiB
ps -eo pid,oom,oomadj,rss,comm --sort=-oom | head -n 6The oom column is /proc/PID/oom_score. On current kernels it is scaled to 0–1333: a process that uses almost no memory and has no adjustment scores 666 (on the test machine, an idle shell did), each oom_score_adj point adds about two thirds of a point, and memory use adds the rest. Higher means the killer picks it sooner.
- Protect a critical service a little, not completely:
OOMScoreAdjust=-500in its unit. Docker sets -500 fordockerdand containerd uses -999, as the table in section 3 shows. - -1000 exempts a process completely. Keep it for processes without which the machine cannot recover. An exempt process that leaks takes everything else down with it.
- Mark disposable work as the first choice with a positive value, for example
OOMScoreAdjust=500for batch jobs. vm.overcommit_memory: 0 (the default) lets programs reserve more memory than exists, using a heuristic. 2 refuses reservations aboveCommitLimitin/proc/meminfo, so programs get allocation errors instead of being killed later. Most software expects the default. Check yours withsysctl vm.overcommit_memory. It was 1 (always allow) on the test machine.vm.panic_on_oom=1reboots instead of killing a process. It only makes sense where a fast reboot by a watchdog is better than a half-working machine.
6. Verify
- The process runs through its normal peak without a new
Killed processline:sudo journalctl -k -b -g 'Killed process' --no-pager. - For a service,
systemctl show UNIT -p MemoryPeakstays clearly belowMemoryMax, andResult=success. - For a machine-wide OOM,
free -hshows available memory and some free swap at the busiest time of day, not only right after a restart.
7. Rollback and prevention
- Rollback: undo a changed limit or
OOMScoreAdjustwithsudo systemctl revert UNIT(it removes drop-ins), thensudo systemctl restart UNIT. A sysctl change goes back withsudo sysctl -w NAME=OLD_VALUEand by removing its file in/etc/sysctl.d/. - Give every service and container a memory limit, so a leak kills only the leaking process, not a random neighbor.
- Make the journal persistent on servers (
/var/log/journal/), or the evidence disappears with the reboot that follows a machine-wide OOM. - Alert before the kill: watch available memory and memory pressure (
/proc/pressure/memory), as described in Check Linux CPU, memory, and disk pressure, and alert on newoom-killevents. - Add swap with care. A little swap gives the kernel room for memory that is rarely used. A lot of swap on a slow disk can turn a quick kill into a machine that is too slow to log in to.
References
Official reference
Technical steps are original explanations; consult the upstream documentation for version-specific details.
Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.
Found this useful?
Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.