We earn commissions when you shop through the links below, at no extra cost to you. Our guides and technical recommendations are written independently of these relationships: a commission never decides what we recommend, how we diagnose a problem or what we publish, and the content does not depend on it. Affiliate disclosure

Linux · Intermediate · 25 min

Linux server troubleshooting runbook

Linux troubleshooting strategy and commands for RHEL and Ubuntu: capture evidence, classify the symptom, run read-only checks, make one fix, verify, roll back.

Most Linux incidents are one of six classes, and the order of work is the same for all of them: evidence, classification, one narrow change, verification. This page is that order of work and nothing more: the rules, a symptom table that points to the guide for each class, a baseline and a snapshot script to run first, the rules for a change, a verification script, a rollback table and a change-note template. The checks and the fixes for a single symptom are in its guide. Commands target RHEL 8/9 and Ubuntu 22.04/24.04; run changes on a disposable machine first.

This is the method that sits underneath the per-symptom guides, and it holds no diagnosis of any one symptom: capture evidence, classify the symptom, make the narrowest change at the owning source, verify with a second signal, and write a change note with a rollback. The checks, the deep dives and the fixes for each class are in that class's guide, linked from the table in section 1. It targets RHEL 8/9 and Ubuntu 22.04/24.04 hosts, including Azure VMs and AKS or k3s nodes.

The order of work for every Linux incident1. Evidence (snapshot before any restart (section 3)) Then 2. Classify (match the symptom to a class (section 1)) Then 3. Read-only checks (baseline, then the guide for the class (section 2)) Then 4. Narrow change (at the owning source, with a rollback (section 4)) Then 5. Verify (original check plus a second signal (section 5)). Then it starts again: signals disagree: back to the evidence.Evidencesnapshot beforeany restart(section 3)Classifymatch the symptomto a class(section 1)Read-onlychecksbaseline, thenthe guide for theclass (section 2)Narrow changeat the owningsource, with arollback (section4)Verifyoriginal checkplus a secondsignal (section5)signals disagree: back to the evidence
Evidence comes before any change, and a change is only finished when a second signal agrees with the original check. If the signals disagree, the diagnosis was wrong: return to the evidence.

How to use this page

Read section 0 once. Before you change anything, run the baseline (section 2) and the snapshot script (section 3), then find the symptom in section 1 and follow its link to the guide for that class. Come back here for the rules that hold for every class: how to change, verify and roll back (sections 4 and 5) and how to write the change note (section 6). Start here when you do not yet know which class you are in; when you do, go straight to the guide.

The tools behind the checks have their own references: journalctl, ss, getent and dig, df and du. To practice the method on a disposable VM in order, follow the Linux troubleshooting learning path.

0. Rules of engagement

  1. Evidence first. Capture state before any restart, kill, resize, reboot or rule change. A restart destroys the best evidence you have (memory state, open FDs, connection tables, in-memory logs). Run the snapshot script in section 3 first.
  2. Read-only before change. Everything in sections 1 to 3 is read-only unless marked [CHANGE]. State changes only happen in section 4, under a change note (section 6) with a rollback.
  3. Know your access path before you touch network or firewall. If you are on SSH, a bad firewall/sshd/DNS change locks you out. On Azure, confirm you have a second path first: Serial Console, az vm run-command, Bastion, or a Run Command extension.
  4. Secrets. Do not cat env files, unit files with Environment=, /proc/PID/environ, or ps command lines into tickets or chat. Reference variable names only. Redact tokens, passwords, internal hostnames and IPs before sharing output. systemctl show, journalctl and ss -p output can contain arguments with secrets.
  5. Time-stamp everything. Use UTC (date -u +%FT%TZ) in notes. One reading proves nothing; take two samples separated by a known interval.
  6. Second signal. Never conclude from one tool. Confirm with a second, independent source (for example ss plus an external probe, df -h plus df -i, resolvectl plus dig).
  7. Do not, blind: kill -9 a process you have not identified; rm -rf anything under /var, /home or /; systemctl restart a failing unit in a loop; iptables -F / nft flush ruleset / systemctl stop firewalld; setenforce 0; edit /etc/resolv.conf by hand on a host where it is managed; chmod -R / chown -R on system paths; resize or reboot a host with unexplained pressure.
  8. Fix at the owning source. The edit belongs in Git/Ansible/Terraform/cloud-init/Helm, not only on the host. A hand-edit that is not backported will be reverted on the next run.
  9. Privilege. Diagnosis needs read access (journalctl needs systemd-journal or adm group; ss -p and lsof need root for other users' processes). Use sudo per command, not a root shell, and keep the session logged.

1. Classify the symptom

Match the first observable symptom to the most likely cause class, then open the guide in the last column; it has the read-only checks, the deep dive and the fix for that class. Always run the baseline in section 2 first; it takes under a minute.

Symptom or errorLikely cause classFirst read-only checkGuide
High load average, slow shell, timeoutsCPU saturation, run-queue, or IO wait (load counts D-state tasks)uptime, vmstat 1 5, top -b -n1CPU, memory and disk pressure
Out of memory: Killed process in kernel logOOM killer, cgroup memory limit, leakjournalctl -k -g oom, free -m, dmesg -TFind what the OOM killer killed
Heavy swapping, host unresponsiveMemory pressure, swap thrashvmstat 1 5 (si/so), cat /proc/pressure/memoryCPU, memory and disk pressure
No space left on device, but df -h shows free spaceInode exhaustion, deleted-but-open files, reserved blocks, quotadf -i, lsof +L1Out of disk space
No space left on device, df -h shows 100%Block exhaustion (logs, journal, containers, core dumps)df -h, du -xh --max-depth=1Out of disk space
Job for X.service failed, start-limit-hitConfig error, permissions, dependency, port conflict, SELinux/AppArmor denialsystemctl status X, journalctl -u X -bService fails to start
Unit flaps, Result: exit-code / signal / oom-killCrash loop, OOM, wrong Type=systemctl show X -p Result,ExecMainStatus,NRestartsService fails to start
ssh: connect to host ... port 22: Connection refusedNothing listening, wrong port, host firewall REJECTss -tlnp, systemctl status sshd (from console; the name sshd is valid on RHEL and also resolves on Ubuntu 26.04, where the unit is ssh)SSH connection refused
ssh: ... Connection timed outPacket drop: NSG, firewall DROP, routingip route get, nc -vz HOST 22SSH connection refused
Permission denied (publickey)Auth/key/permissions/SELinux context, not a network issuessh -vvv, server journalctl -u sshdSSH permission denied (publickey)
Could not resolve host, Temporary failure in name resolutionLocal resolver config, upstream unreachable, record missingresolvectl status (failed on the AlmaLinux 10.2 test machine, where systemd-resolved was not running; use cat /etc/resolv.conf there), dig @RESOLVER NAMEDNS resolution failures
Works by IP, fails by name; only some names failSearch domain, split DNS, stale cache, NXDOMAIN vs SERVFAILgetent hosts NAME vs dig NAMEDNS resolution failures
Port reachable locally, not remotelyHost firewall, cloud NSG, listener bound to 127.0.0.1ss -tlnp, firewall listingFirewall access
Connection works from host A, not host BSource-specific rule, routing, NSG, asymmetric pathip route get, tcpdump (bounded)Firewall access

2. Baseline: the Linux troubleshooting commands to run first

All commands in this section are read-only.

# Baseline: time, kernel and OS, system state, failed units, CPU, memory and disk
date -u +%FT%TZ; hostnamectl --static; uptime
uname -r; cat /etc/os-release | grep -E '^(NAME|VERSION_ID)='
systemctl is-system-running; systemctl --failed --no-legend
nproc; free -m; df -hT -x tmpfs -x devtmpfs; df -i -x tmpfs -x devtmpfs

Read it as follows: load average vs nproc (sustained load above core count means saturation, but Linux load also counts uninterruptible IO waiters, so high load with idle CPU points at IO); is-system-running returning degraded means at least one failed unit; df -hT shows filesystem type, which matters for the fix (xfs cannot shrink, ext4 reserves 5% by default).

3. Read-only snapshot script

Run this before any change and attach the archive to the ticket after review. It writes to a mode-700 directory with mode-600 files, collects no environment variables and no process arguments (ps prints command names only), and makes no change to the host. Journal excerpts can still contain sensitive strings: review before sharing. Optional arguments: a unit name and a DNS name to test.

#!/usr/bin/env bash
# linux-snapshot.sh - read-only evidence collection for Linux troubleshooting.
# Usage: sudo ./linux-snapshot.sh [UNIT] [DNS_NAME]
# Writes to /var/tmp/snapshot-HOST-TIMESTAMP (mode 700, files mode 600). No changes to the host.
set -u
umask 077

UNIT="${1:-}"
DNS_NAME="${2:-}"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
OUT="/var/tmp/snapshot-$(hostname -s)-${TS}"
mkdir -p "$OUT" && chmod 700 "$OUT" || { echo "cannot create $OUT" >&2; exit 1; }

# run LABEL CMD... : capture stdout+stderr, never abort on a failing command.
run() {
  local label="$1"; shift
  { echo "### $label"; echo "### cmd: $*"; echo "### at: $(date -u +%FT%TZ)"; "$@" 2>&1; echo "### rc=$?"; } > "$OUT/${label}.txt"
}

run os          cat /etc/os-release
run kernel      uname -a
run uptime      uptime
run time        timedatectl
run cpu         nproc
run mem         free -m
run vmstat      vmstat 1 5
run psi_cpu     cat /proc/pressure/cpu
run psi_mem     cat /proc/pressure/memory
run psi_io      cat /proc/pressure/io
run top_cpu     bash -c "ps -eo pid,user,pcpu,pmem,rss,stat,comm --sort=-pcpu | head -20"
run top_mem     bash -c "ps -eo pid,user,pcpu,pmem,rss,stat,comm --sort=-rss | head -20"
run dstate      bash -c "ps -eo pid,user,stat,wchan:30,comm | awk '\$3 ~ /^D/'"
run df_bytes    df -hT
run df_inodes   df -i
run mounts      findmnt
run lsblk       lsblk -f
run deleted     bash -c "lsof +L1 2>/dev/null | head -30"
run journal_du  journalctl --disk-usage
run oom         journalctl -k --since "-24h" -g "Out of memory|oom-kill|Killed process" --no-pager
run failed      systemctl --failed --no-pager
run listeners   ss -tulnp
run routes      ip -br route
run addrs       ip -br addr
run resolv_link readlink -f /etc/resolv.conf
run dns_status  bash -c "resolvectl status 2>/dev/null | head -60"
run fw_active   bash -c "systemctl is-active firewalld ufw nftables 2>&1"
run nft         bash -c "nft list ruleset 2>/dev/null | head -200"
run sshd_eff    bash -c "sshd -T 2>/dev/null | grep -Ei '^(port|listenaddress|permitrootlogin|passwordauthentication|pubkeyauthentication|allowusers|allowgroups)'"
run selinux     bash -c "getenforce 2>/dev/null || aa-status 2>/dev/null | head -5"

if [ -n "$UNIT" ]; then
  run unit_show   systemctl show "$UNIT" -p ActiveState,SubState,Result,ExecMainStatus,NRestarts,FragmentPath,DropInPaths
  run unit_status systemctl status "$UNIT" -l --no-pager
  run unit_log    journalctl -u "$UNIT" -b -n 200 --no-pager
fi

if [ -n "$DNS_NAME" ]; then
  run dns_getent  getent hosts "$DNS_NAME"
  run dns_dig     dig "$DNS_NAME" +noall +answer +comments
fi

find "$OUT" -type f -exec chmod 600 {} +
tar -C /var/tmp -czf "${OUT}.tar.gz" "$(basename "$OUT")" && chmod 600 "${OUT}.tar.gz"
echo "Snapshot: ${OUT}.tar.gz  (review before sharing; contains hostnames and IPs)"

Notes: the script depends only on standard tools (vmstat, ps, ss, ip from procps-ng and iproute/iproute2). lsof, dig (package bind-utils on RHEL, bind9-dnsutils on Ubuntu 26.04; the older name dnsutils is not the package there) and nft may be missing; the capture then records a non-zero rc when the tool itself is the last command. Several captures end in | head, and rc= is the status of the last stage: an empty capture can show rc=0. In our run dns_status was empty with rc=0 on AlmaLinux because resolvectl status failed (The name is not activatable) and head was the last stage, so read the capture contents, not only rc. The snapshot and verification scripts ran unchanged on an AlmaLinux 10.2 and an Ubuntu 26.04 test machine. The dstate capture is the evidence for IO-wait incidents: it lists tasks in uninterruptible sleep and the kernel function they are blocked in. Delete snapshots from /var/tmp once they are attached to the ticket; do not leave them there for days.

4. Fix safely

Every command in a fix is [CHANGE]. Do not start until the snapshot (section 3) is archived, a change note (section 6) is open, and you have a second access path. Take the fix for your class from its guide, with the options, the commands and what each one risks. Prefer the option that contains the blast radius, and make the same change in the owning source (Ansible, Terraform, cloud-init, Git) in the same change. These rules hold for every class:

  • Do not run setenforce 0, disable AppArmor profiles, stop firewalld/ufw, or flush the nft ruleset to "test" a theory.
  • Do not open a port to 0.0.0.0/0, or add AllowUsers *, PermitRootLogin yes, or PasswordAuthentication yes to fix an access problem.
  • Do not chmod 777 or chown -R system or home paths; use namei -l and fix the specific bit.
  • Do not kill -9 or restart before the snapshot is archived.
  • Do not delete under /var/lib (databases, containers, kubelet) by hand.
  • Do not hand-edit /etc/resolv.conf, netplan, or NetworkManager files on a host managed by cloud-init or config management without updating the source.
  • Do not reboot a host to clear a symptom you have not explained; a reboot erases kernel ring buffer, process state and tmpfs.

5. Verification and rollback

Verification repeats the original failing check, adds a second independent signal, and then waits. Everything here is read-only. Each guide names the second signal and the soak time for its class; the script below is the check that works across classes.

User-facing confirmation is part of verification: run the application's own health check or a real transaction, not only a port test.

# Read-only post-change check, saves nothing sensitive. Pass UNIT, MOUNT, HOST, PORT.
set -u
UNIT="${1:?unit}"; MOUNT="${2:?mount}"; HOST="${3:?host}"; PORT="${4:?port}"
date -u +%FT%TZ
systemctl is-active "$UNIT"
systemctl show "$UNIT" -p Result,NRestarts,ExecMainStatus
df -h "$MOUNT"; df -i "$MOUNT"
uptime
journalctl -k --since '-30min' -g 'oom-kill|Out of memory' --no-pager | tail -5
nc -vz -w 5 "$HOST" "$PORT"   # Ncat on RHEL, OpenBSD netcat on Ubuntu: the messages differ, check the exit code
getent hosts "$HOST"

5.1 Rollback

ChangeRollback triggerRollback actionNot reversible
set-property or drop-in on a unitService errors, latency up, OOM at the new capRemove drop-in, systemctl daemon-reload, restart in a windowWork lost by a restart
journald retentionNeeded older entriesRestore previous drop-in, restart journaldVacuumed journal entries
File deletion or cleanupApplication errors afterwardsRestore from backup or snapshotDeleted data without a backup
Disk or partition growFilesystem errorsRestore from the pre-change Azure disk snapshotShrinking xfs (not supported); partition table mistakes
sshd configNew login fails, or monitoring shows failuresRestore saved sshd_config, sshd -t, reload from the still-open sessionA lockout with no second path (needs Serial Console or az vm run-command)
Firewall ruleUnexpected exposure or lost accessfirewalld: --permanent --remove-rich-rule plus reload; ufw: ufw deleteConnections dropped during reload
Resolver or netplan changeName resolution worseRestore previous file from backup, netplan try first, or re-apply NetworkManager profileCached bad answers until TTL expiry
Rebootn/an/aKernel ring buffer, tmpfs, in-memory state, process evidence

Before any config edit take a copy: sudo cp -a FILE FILE.bak-$(date -u +%Y%m%dT%H%MZ). Prefer netplan try (Ubuntu) with its timed auto-revert for network changes; verify on your version that it is available in your image. On Azure, take a disk snapshot (az snapshot create) before resize or bulk deletion.

6. Change note template

Use this for every [CHANGE] step, in the lab and in production. In the lab it doubles as the learning record: it forces you to state the question, the evidence and the rollback before you act. Remove hostnames, usernames, addresses and tokens before sharing.

# Change note: SHORT TITLE

- Date/time (UTC):
- Operator:
- Host / environment:
- Ticket:

## Symptom and impact
What is failing, since when, who is affected.

## Evidence (read-only)
- Snapshot archive: (name, stored where)
- Key readings: (command -> result, with timestamps)
- Second signal:

## Cause (confirmed / suspected)
One sentence. Mark suspected causes as suspected.

## Change
- Owning source (Git/Ansible/Terraform path):
- Exact commands or diff:
- Options considered and why rejected:

## Risk
Lockout / data loss / evidence loss / blast radius.

## Rollback
- Trigger:
- Steps:
- Not reversible:

## Verification
- Original check repeated: (result)
- Second signal: (result)
- Soak time and result:

## Follow-up
Prevention item, alert, ticket for the owning team.

7. Prevention

Each guide ends with the guardrails for its own class: alerts, configuration in code and monitoring, so that the next occurrence is caught earlier or cannot happen. They are in CPU, memory and disk pressure, service start failures, out of disk space, SSH, DNS and firewall access. Whatever the class, the follow-up line of the change note should name one guardrail and who owns it.

8. References

Related pages on opsbreakfix.com: the hands-on labs directory and resources.

Official reference

man7.org ↗

Technical steps are original explanations; consult the upstream documentation for version-specific details.

Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.

Found this useful?

Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.