Linux · Practice · 60-90 min
Linux troubleshooting practice: 8 broken servers
Eight Linux troubleshooting exercises on a disposable server: one script breaks it, you find and fix the cause, and it checks your fix. Ubuntu and RHEL.
Eight broken servers to fix on your own: a small script breaks a disposable machine, you get only the symptom a user would report, and the same script tells you when it is really fixed. Hints and full solutions with real output are at the end. Every exercise was run on Ubuntu 26.04 and AlmaLinux 10.2.
On this page
Reading about troubleshooting is not the same as doing it at 3 a.m. with nothing but a symptom. This page gives you eight broken servers to fix on your own. A small script breaks a disposable machine in a specific way, you get only the symptom a user would report, and the same script tells you when it is really fixed. Hints and full solutions, with real output, are at the end, so you can try first.
Every exercise was run on Ubuntu 26.04 and AlmaLinux 10.2 with systemd. The script works on both families and on Debian, RHEL and Rocky Linux; it picks apt or dnf itself.
0. Rules of engagement
- Use a machine you can delete. A local VM, a WSL2 distribution with systemd, or a small cloud server. The script installs packages, creates users and units, mounts a small loop filesystem and deliberately breaks things. Do not run it on a server that matters.
- Read the script before you run it as root. It is about 250 lines of plain bash, one block per exercise, and every block has a matching
reset. - Work from evidence. The point is the diagnosis, not the fix. Before you change anything, write down what you saw and what you think is wrong. That is the habit the troubleshooting runbook is built on.
- Clean up when you are done.
obf-lab.sh restoreremoves everything the script created and installed (section 8).
1. Prepare a test machine
You need one Linux machine that you are happy to break and throw away. You do not need to install nginx, SSH, cron or anything else by hand: the script's prepare command checks the machine and installs every package the exercises use, and restore removes them again at the end.
What the machine needs:
| Requirement | Why | How to check |
|---|---|---|
| Ubuntu or Debian (apt), or RHEL, AlmaLinux, Rocky (dnf) | the script installs packages with apt or dnf | cat /etc/os-release |
| systemd running as PID 1 | six of the eight exercises are systemd services and timers | ps -p 1 -o comm= prints systemd |
root, or a user that can run sudo | the script installs packages and creates users | sudo true asks for your password and returns |
| internet access | to download the script and the packages | curl -sI https://opsbreakfix.com |
about 1 GB of RAM and 1 GB free on / | packages, a small loop filesystem, a memory-limited worker | prepare checks both |
| nothing important on it | the script breaks things on purpose | you decide |
It will not work in a normal Docker container (no systemd as PID 1), on WSL1, on macOS directly, or on a shared server you do not own.
Pick one of these three ways to get a machine.
Option A: WSL2 on Windows (free, about 5 minutes)
Open PowerShell and create a separate distribution just for the lab, so your everyday WSL stays untouched. The --name option needs WSL 2.4 or newer (wsl --version shows it; wsl --update updates it).
# PowerShell on Windows: a new Ubuntu 26.04 named obf-lab
wsl --install -d Ubuntu-26.04 --name obf-lab# Real output
Downloading: Ubuntu 26.04 LTS
Installing: Ubuntu 26.04 LTS
Distribution successfully installed. It can be launched via 'wsl.exe -d obf-lab'It opens a terminal and asks for a new user name and password: that user can use sudo. Later, wsl -d obf-lab opens it again. Check that systemd is running:
# Inside the new distribution
ps -p 1 -o comm=# Real output
systemdThe current Ubuntu 26.04 and AlmaLinux 10 images have systemd on by default. If you see init instead, turn it on and restart the distribution:
# Only if the check above did not print systemd
printf '[boot]
systemd=true
' | sudo tee -a /etc/wsl.conf# PowerShell: stop WSL, then open obf-lab again
wsl --shutdown
wsl -d obf-lab- RHEL, AlmaLinux, Rocky:
wsl --install -d AlmaLinux-10 --name obf-labworks the same way; your user is in thewheelgroup, sosudoworks.wsl --list --onlineshows every distribution you can install.
When you are finished, delete the whole thing from PowerShell. This erases everything inside it:
# PowerShell: delete the lab distribution and everything in it
wsl --unregister obf-lab# Real output
The operation completed successfully.Option B: a small cloud server (a few cents for an afternoon)
Create the smallest Ubuntu 26.04 or AlmaLinux 10 server your provider offers (1 GB of RAM is enough), add your SSH key while creating it, and log in:
# From your own computer; the provider shows the IP address
ssh root@203.0.113.10Cloud images run systemd, so there is nothing else to set up. Delete the server in the provider's panel when you are done: that is cheaper and cleaner than any cleanup command.
Option C: a local virtual machine
Any hypervisor works (VirtualBox, VMware, Hyper-V, UTM on a Mac, virt-manager on Linux). Install Ubuntu Server 26.04 or AlmaLinux 10 (minimal) from the official ISO, give it 1 GB of RAM or more and 10 GB of disk, and take a snapshot before you start, so you can roll back in one click.
Check the machine and install the packages
Download the script (section 2 shows how to read it first), then run prepare. It checks systemd, free disk space and RAM, installs the packages and confirms that every tool the exercises and their solutions use is present:
# On the test machine
curl -fsSLO https://opsbreakfix.com/assets/obf-lab.sh
sudo bash obf-lab.sh prepare# Real output on a fresh Ubuntu 26.04 test machine; AlmaLinux 10.2 prints the same lines
obf-lab.sh 2026-11-11.3: preparing a debian machine
ok systemd is running (PID 1: systemd)
ok 977262 MB free on /
ok 11960 MB of RAM
... installing packages (a minute or two the first time)
ok tools: curl python3 ss ps journalctl ssh sshd crontab mkfs.ext4 less
READY: start with 'sudo bash obf-lab.sh break 1'. When you are done, 'sudo bash obf-lab.sh restore' removes all of it.A FAIL line tells you what to fix, and nothing is installed until all checks pass. Forgot sudo? The script says so: obf-lab: run as root: sudo bash obf-lab.sh prepare.
What prepare installs (each break would install its own part anyway; prepare just does it all up front, so no exercise stops halfway for a download):
| Used by | Ubuntu, Debian | RHEL, AlmaLinux, Rocky |
|---|---|---|
| Exercise 1 (web server, and the process that blocks it) | nginx, curl, python3 | nginx, curl, python3 |
| Exercise 2 (SSH) | openssh-server, openssh-client | openssh-server, openssh-clients |
| Exercise 3 (cron) | cron | cronie |
| Exercise 4 (loop filesystem) | e2fsprogs | e2fsprogs |
Diagnosis (ss, ps, less) | iproute2, procps, less | iproute, procps-ng, less |
| Exercises 5 to 8 | nothing extra: systemd only | nothing extra |
On a fresh Ubuntu 26.04 machine prepare took 15 seconds.
2. Get the script
If you ran prepare in section 1, you already have the script; skip the curl line. Read it before you run anything else as root: less opens it, the arrow keys scroll, / searches and q quits.
# Download, read, then list the exercises (on the test machine)
curl -fsSLO https://opsbreakfix.com/assets/obf-lab.sh
less obf-lab.sh
sudo bash obf-lab.sh list# Real output
1 The website is down: nginx will not start
2 "Permission denied (publickey)" for the deploy user
3 The nightly report job runs, but no report appears
4 "No space left on device", but df shows free space
5 The server is slow, and nobody started anything
6 A worker service keeps restarting
7 The app service fails right after it starts
8 Backups stopped after the last rebootThe loop for every exercise is the same:
# Break, investigate, fix, then let the script judge
sudo bash obf-lab.sh break 1
sudo bash obf-lab.sh check 1 # NOT YET: ... until it is really fixed
sudo bash obf-lab.sh reset 1 # undo everything the exercise createdStuck or lost? Two escape hatches:
# Apply the reference fix for one exercise; it prints every command it runs
sudo bash obf-lab.sh solve 1
# Put the machine back exactly as it was before the first break
sudo bash obf-lab.sh restoresolve is for when you are stuck: read the commands it prints, then try the next exercise on your own. restore resets every exercise, disables the services the script enabled and removes the packages it installed, using a list of installed packages it saved before the first break. On both test machines the package list and the enabled services were identical before the exercises and after restore.
check tests the outcome a user cares about, not the command you ran. For example, exercise 1 asks whether port 80 answers with nginx, not whether you stopped a particular process.
3. The exercises
Each one starts with what a user or a monitor would tell you. Try it without the hints first. If you get stuck, the linked guide has the method.
| # | The report | Difficulty | If you get stuck |
|---|---|---|---|
| 1 | "The website is down." nginx will not start. | Easy | A service that fails to start |
| 2 | "I can't log in as deploy: Permission denied (publickey)." | Medium | SSH permission denied (publickey) |
| 3 | "The nightly report job runs, but there is no report." | Medium | Cron and systemd timers |
| 4 | "No space left on device", but df -h shows the disk almost empty. | Medium | Server out of disk space |
| 5 | "The server is slow, and nobody started anything." | Easy | Find and stop a process safely |
| 6 | "The worker service keeps restarting." | Medium | Find what the OOM killer killed |
| 7 | "The app starts, then fails a second later." | Easy | Service logs with journalctl |
| 8 | "Backups stopped after the last reboot." | Easy | Manage services with systemctl |
Two of them have a trap: the obvious first fix is not the right one. Say which two, before you read the solutions.
4. Hints
Read one hint at a time, only when you are stuck.
- Something else got to port 80 first. Which process owns a listening port?
- The key, the
authorized_keysfile and the.sshdirectory are all fine. sshd is picky about one more directory, and it says so in its log. - cron does run the job every minute; its log proves it. Look at the file it runs.
df -hcounts bytes. Something else can run out.- Find the process, then find out who started it before you kill anything.
systemctl statusshows aResult. What does it say, and which setting in the unit causes it?- The unit runs as a different user. What does that user need to write to?
systemctl list-timersandsystemctl is-enabledanswer different questions.
5. Solutions
Each solution shows the evidence first, then the narrow fix. All output is real, from the Ubuntu 26.04 test machine unless it says AlmaLinux.
Exercise 1: port 80 is taken
nginx failed to start. The journal says it could not bind, and ss shows who holds the port:
# Who is listening on port 80, and which unit started it?
ss -ltnp 'sport = :80'
systemctl status obf-legacy-ui --no-pager -n 0# Real output (shortened)
LISTEN 0 5 0.0.0.0:80 0.0.0.0:* users:(("python3",pid=759,fd=3))
● obf-legacy-ui.service - [systemd-run] /usr/bin/python3 -m http.server 80 --bind 0.0.0.0 --directory /var/emptyA leftover "legacy UI" holds the port. Stop it through its unit, not with kill, so systemd does not restart it, then start nginx:
# Free the port the supported way, then start the real service
sudo systemctl stop obf-legacy-ui
sudo systemctl start nginxcheck 1 then said FIXED: nginx is running and answers on port 80. In real life, also find out why the other program was there and disable it, or it will be back after the next reboot.
Exercise 2: the home directory is too open (a trap)
The key login fails with Permission denied (publickey,password). The client error says nothing useful; the server log does:
# The reason is only in sshd's log (the unit is ssh on Ubuntu, sshd on RHEL)
sudo journalctl -u ssh --since -5min | grep -i refused
ls -ld /home/deploy# Real output
Authentication refused: bad ownership or modes for directory /home/deploy
drwxrwxrwx 3 deploy deploy 4096 Nov 11 10:45 /home/deploysshd's StrictModes refuses keys when the home directory is writable by others, because anyone could replace .ssh. The trap is to regenerate keys or to fix .ssh permissions, which were already right. The fix is the home directory:
# Group and others must not be able to write the home directory
sudo chmod 755 /home/deploy- RHEL, AlmaLinux, Rocky: the same fix; read the log with
journalctl -u sshd. Tested on AlmaLinux 10.2.
Exercise 3: the script is not executable
cron's log proves the job runs every minute:
# Real output from journalctl -u cron (crond on RHEL)
Nov 11 10:46:01 lab CRON[1231]: (root) CMD (/usr/local/bin/obf-report)
Nov 11 10:47:01 lab CRON[1272]: (root) CMD (/usr/local/bin/obf-report)But the file it runs cannot be executed:
# Real output from ls -l /usr/local/bin/obf-report
-rw-r--r-- 1 root root 68 Nov 11 10:45 /usr/local/bin/obf-reportcron starts the job, the shell fails to execute it, and the error goes to a mail nobody reads. Make the script executable and wait for the next minute:
# The job line was fine; the file it points to was not executable
sudo chmod 755 /usr/local/bin/obf-reportExercise 4: out of inodes
touch fails although there are 26 MB free. df -i counts files instead of bytes:
# Real output from df -h and df -i on /srv/obf-cache
/dev/loop0 28M 224K 26M 1% /srv/obf-cache
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/loop0 2048 2048 0 100% /srv/obf-cacheEvery file needs an inode, and this filesystem has none left. Find the directory with the most files:
# Count files per directory on this filesystem only
sudo find /srv/obf-cache -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -n | tail -2# Real output
2034 /srv/obf-cache/sessionsThousands of empty session files. Delete them, and in real life fix whatever stopped cleaning them up:
# Remove the stale session files
sudo find /srv/obf-cache/sessions -type f -deleteExercise 5: a CPU hog started by a unit
# Top CPU consumers, then which unit owns the top one
ps -eo pid,%cpu,cmd --sort=-%cpu | head -3
systemctl status 1360 --no-pager -n 0 | head -1# Real output
PID %CPU CMD
1360 99.6 /usr/bin/sh -c while :; do :; done
1357 98.7 /usr/bin/sh -c while :; do :; done
● obf-indexer-2.service - [systemd-run] /usr/bin/sh -c "while :; do :; done"systemctl status PID tells you which unit a process belongs to. Two "indexer" units burn a core each. Stop them through systemd, which also prevents a restart:
# Stop the units, not just the processes
sudo systemctl stop obf-indexer-1 obf-indexer-2Exercise 6: the memory limit kills the worker (a trap)
# Real output from systemctl status obf-worker and its journal
Active: activating (auto-restart) (Result: oom-kill) since Wed 2026-11-11 10:48:34 EET; 813ms ago
obf-worker.service: The kernel OOM killer killed some processes in this unit.
obf-worker.service: Main process exited, code=killed, status=9/KILL
obf-worker.service: Failed with result 'oom-kill'.The machine has plenty of memory; the unit has MemoryMax=32M and the worker needs 64 MB. The trap is to add swap or more RAM, which changes nothing, because the limit is the unit's. Raise it with a drop-in, so the original unit file stays untouched:
# A drop-in overrides one setting and survives package updates
sudo mkdir -p /etc/systemd/system/obf-worker.service.d
printf '[Service]\nMemoryMax=128M\n' | sudo tee /etc/systemd/system/obf-worker.service.d/memory.conf
sudo systemctl daemon-reload
sudo systemctl restart obf-workerWhether 128M is right is a sizing question; in production, measure the real peak first (systemctl show obf-worker -p MemoryPeak).
Exercise 7: the service user cannot write its log
# Real output from systemctl status obf-app and ls -l
× obf-app.service - obf-lab app (writes a line to its log every 5 seconds)
Active: failed (Result: exit-code) since Wed 2026-11-11 10:48:47 EET; 2s ago
-rw-r--r-- 1 root root 0 Nov 11 10:48 /var/log/obf-app/app.logThe unit runs as obfapp, and the log file belongs to root with mode 644. journalctl -u obf-app also shows the shell's Permission denied line. Give the file to the user that writes it, then restart:
# The writer must own (or be able to write) its log file
sudo chown obfapp: /var/log/obf-app/app.log
sudo systemctl restart obf-appDo not "fix" it with chmod 777 or by running the service as root.
Exercise 8: the timer was never enabled
# Real output from systemctl list-timers obf-backup.timer --all and is-enabled
NEXT LEFT LAST PASSED UNIT ACTIVATES
0 timers listed.
disabledThe timer and its service exist, but nobody enabled the timer. Someone started it by hand, so it worked until the reboot. enable makes it start at boot, --now also starts it now:
# Start now and after every reboot
sudo systemctl enable --now obf-backup.timerThe two traps were exercises 2 and 6.
6. Verify a fix the way a user would
obf-lab.sh check N tests the outcome, not your command: the page answers on port 80, the key login works, the report file keeps growing, a file can be created, the CPU is idle, the worker stays up without restarts, the app writes its log, the timer is enabled. In real incidents, do the same: check what the user reported, from where they see it, and check it again a few minutes later.
# The script's verdict for every exercise (FIXED or NOT YET, with the reason)
for n in 1 2 3 4 5 6 7 8; do sudo bash obf-lab.sh check $n; done7. Where to practice next
These eight exercises cover the most common single faults. Real incidents stack two or three of them, and the hard part is choosing what to look at first. Three ways to go further:
- Free: break your own test machine. Pick a guide from the table above, read only its symptom table, and reproduce one cause before you read the fix.
- A server you can destroy: a small cloud server for a few hours costs very little and behaves like production (real network, real disk, a public IP to attack your own firewall rules). Delete it when you are done.
- Graded labs: if you want exercises that are set up for you and checked automatically, on more topics than one script can break, a lab platform does that. It is not necessary for these eight; it helps when you want volume and a curriculum, for example for the LFCS exam.
8. Rollback and clean up
reset N removes what one exercise created (users, units, the loop filesystem, cron entries, test files); packages stay installed. restore also removes the packages and disables the services the script added, so the machine is as it was before the first break. Packages you installed yourself after that first break would be removed too, so run restore before you install anything of your own.
# Back to how the machine was before the first break
sudo bash obf-lab.sh restoreOr throw the whole machine away, which is even cleaner: wsl --unregister obf-lab in PowerShell for WSL (section 1), delete the server in the provider's panel for a cloud server, or roll back to the snapshot for a virtual machine.
Related guides
- Linux server troubleshooting runbook: the method behind all eight exercises.
- Linux Troubleshooting and Operations learning path: guided lessons in order, if you want to learn each class before you test yourself.
- Find what the Linux OOM killer killed: exercise 6 in depth, including a whole machine running out.
Sources
- obf-lab.sh, the script used on this page (plain bash, readable before you run it).
- sshd_config(5), StrictModes: https://man7.org/linux/man-pages/man5/sshd_config.5.html
- systemd.resource-control(5), MemoryMax: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html
- systemd.timer(5): https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html
Official reference
Technical steps are original explanations; consult the upstream documentation for version-specific details.
Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.
Found this useful?
Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.