SRE · Intermediate · 14 min
Monitor a restic backup from a systemd timer
Send start, success and failure pings from a systemd service, an OnFailure= unit and a timer, with TimeoutStartSec and a mount guard, tested with restic 0.18.1.
A backup that never runs cannot raise a failure alert. This runbook is the systemd approach: a service and a timer send the pings to a heartbeat check, so a disabled timer, a switched-off host or an unmounted source raises an alert too. The cron and Bash approach is in the use case linked on this page. Commands were run with restic 0.18.1 and systemd on Ubuntu 26.04, with a local listener standing in for the monitor.
On this page
- Rules of engagement
- Two approaches to the same check
- Two kinds of failure
- Create the check
- The service, the failure unit and the timer
- Drill 1: a normal run
- Drill 2: a loud failure
- Drill 3: the empty folder that "succeeds"
- Drill 4: the silent stop
- After the first week
- Verify, rollback and prevention
- Common mistakes
- References
See it run
A short replay of real output from commands in this guide, captured on a WSL2 Ubuntu test machine (restic 0.18.1, systemd service and timer run as root, a local listener standing in for the monitor, a 10 second timer for the demo). The typing is replayed, and the host name, account and MAC addresses are redacted. Your output will differ, and a test machine is not a production server.
Text version of this demo
ops@web-01:~$ systemctl start restic-backup; tail -n 2 pings.log
01:23:46 /ping/demo-check/start
01:23:47 /ping/demo-check
ops@web-01:~$ echo wrong-password > .pw; systemctl start restic-backup 2>/dev/null; echo "unit exit: $?"; cp .pw.good .pw; sleep 1; tail -n 2 pings.log; journalctl -u restic-backup -n 5 -o cat | grep status=
unit exit: 1
01:23:47 /ping/demo-check/start
01:23:48 /ping/demo-check/fail
restic-backup.service: Main process exited, code=exited, status=12/n/a
ops@web-01:~$ umount data; restic backup data | tail -n 2; echo "exit status: ${PIPESTATUS[0]}"
processed 0 files, 0 B in 0:00
snapshot a5a75342 saved
exit status: 0
ops@web-01:~$ systemctl start restic-backup 2>/dev/null; echo "unit exit: $?"; sleep 1; tail -n 2 pings.log; mount --bind disk data
unit exit: 1
01:23:50 /ping/demo-check/start
01:23:50 /ping/demo-check/fail
ops@web-01:~$ systemctl enable --now restic-backup.timer 2>/dev/null; sleep 12; systemctl list-timers restic-backup.timer | head -n 2; ./hc-check.sh 10 15
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2026-11-01 01:24:10 EET 6s Sun 2026-11-01 01:24:00 EET 2s ago restic-backup.timer restic-backup.service
UP: last success ping 2 s ago
ops@web-01:~$ systemctl disable --now restic-backup.timer 2>/dev/null; systemctl list-timers restic-backup.timer | head -n 2; sleep 28; ./hc-check.sh 10 15
NEXT LEFT LAST PASSED UNIT ACTIVATES
DOWN: last success ping 31 s ago (period 10 s + grace 15 s)A failure alert only fires when the backup runs and fails. The outages that cost data are quieter: the timer was never enabled after a reinstall, the host was switched off for a month, a typo in the schedule keeps the unit from loading, or the disk did not mount and restic backed up an empty folder and reported success. None of these produces an error. The fix is a dead man's switch: the job reports to a monitor after every good run, and the monitor raises the alarm when the reports stop. That use case explains the idea, push against pull, the grace period and the restore test. This runbook does not repeat it. It covers what the idea needs on a systemd host.
It wires a restic backup to the check with a systemd service, an OnFailure= unit and a timer, then breaks it four ways and shows what the monitor sees. Commands and output were captured on an Ubuntu 26.04 test machine with restic 0.18.1 and systemd. A small local listener stood in for the monitor, so nothing was sent to a real service. The ping paths used (/start, /fail, the plain URL, and /<exit status>) are the ones Healthchecks.io documents; check them against the current documentation of the monitor you choose.
0. Rules of engagement
- Keep the ping URL secret. Anyone who has it can report a failed backup as successful. Put it in a file only root can read, not in the unit file.
- Host the monitor somewhere else. A dead server cannot send you an alert about itself.
- Send the success ping only after restic exits 0. A partial snapshot (exit code 3) and a wrong password (exit code 12) are failures.
1. Two approaches to the same check
This runbook starts the backup from a systemd service and timer and lets the unit send the pings. The use case Backup monitoring with heartbeat checks starts it from cron and a Bash script and lets the script send them, with restic's exit status and log tail in the ping. Both feed the same kind of heartbeat check. They differ in who starts the job and who reports the result.
| cron and a script (the use case) | systemd service and timer (this runbook) | |
|---|---|---|
| Starts the backup | a crontab line | a timer unit with OnCalendar= |
| Sends the pings | the script: /start, then $HC_URL/$status | the unit: ExecStartPre= for start, ExecStartPost= for success, a second unit started by OnFailure= for failure |
| What the failure alert carries | the exit status and restic's log tail | a failure ping; the reason is in journalctl |
| A hung run | the /start ping lets the monitor notice it | the same, and TimeoutStartSec= ends it and triggers OnFailure= |
| An empty or unmounted source | restic exits 0, so the script needs its own check | ExecStartPre= with mountpoint -q (drill 3) |
| Silent causes it catches | cron stopped, host off, a crontab line removed | timer disabled, host off, a schedule typo |
Use the one that matches how your backups already run, and do not run both on the same job.
2. Two kinds of failure
| Kind | Examples | What catches it |
|---|---|---|
| Loud: the job runs and fails | wrong password (exit 12), repository missing (exit 10), repository locked (exit 11), unreadable files (exit 3), disk full, network down | a failure ping from the unit |
| Silent: the job does not run, or "succeeds" with nothing to do | timer disabled or never enabled, unit masked, host off, schedule typo, a hung run, an empty source | missing pings at the monitor |
Use both. The failure ping gives a fast alert. The missing success ping is the only signal for the silent cases.
3. Create the check
Create one check for the backup, with a period, a grace period and a notification channel you read. The use case Backup monitoring with heartbeat checks has the steps, in its section on the schedule, time zone and grace period. Two numbers are specific to a systemd timer: the period is the interval of OnCalendar=, and the grace has to cover the slowest normal run plus the timer's RandomizedDelaySec, for example 2 hours for a 15 minute delay. Copy the check's ping URL into the root-only file in section 4.
4. The service, the failure unit and the timer
# Root-only file with the repository, the password file and the ping URL
sudo install -d -m 700 /etc/restic
sudo tee /etc/restic/env >/dev/null <<'EOF'
RESTIC_REPOSITORY=/mnt/backup/repo
RESTIC_PASSWORD_FILE=/etc/restic/password
HC_URL=https://hc-ping.com/YOUR-CHECK-UUID
EOF
sudo chmod 600 /etc/restic/env# /etc/systemd/system/restic-backup.service
[Unit]
Description=restic backup of /srv/data
OnFailure=restic-backup-failed.service
[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
Environment=RESTIC_CACHE_DIR=%C/restic
TimeoutStartSec=2h
# Tell the monitor the run started; the leading "-" keeps a monitor outage from blocking the backup.
ExecStartPre=-/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${HC_URL}/start
# Refuse to back up a folder whose disk is not mounted.
ExecStartPre=/usr/bin/mountpoint -q /srv/data
ExecStart=/usr/bin/flock -n /run/restic-backup.lock /usr/bin/restic backup --quiet /srv/data
# Runs only when the backup exited 0.
ExecStartPost=-/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${HC_URL}# /etc/systemd/system/restic-backup-failed.service: started by OnFailure= when the unit above fails
[Unit]
Description=Tell the monitor that the restic backup failed
[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
ExecStart=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null ${HC_URL}/fail# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Run the restic backup
[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15m
Persistent=true
[Install]
WantedBy=timers.targetWhat each setting does:
Type=oneshotandExecStartPost=: the success ping runs only afterExecStart=exited 0. Any other exit status fails the unit and triggersOnFailure=.TimeoutStartSec=2h:systemd.servicedisables the start timeout forType=oneshotunits by default, so a hung backup would run forever. With a timeout, a stuck run fails and sends the failure ping.- The
-beforecurlinExecStartPre=andExecStartPost=: a failed ping is recorded but does not fail the unit. Otherwise a monitor outage would stop your backups. flock -n: a second run exits at once with status 1 while the first still holds the lock, instead of waiting or running in parallel.mountpoint -q: see drill 3.Persistent=true: if the host was off at 03:00, the timer runs the backup when it starts again.
Check the schedule before you trust it. A typo can keep a timer from loading:
# Print how systemd reads the schedule and when it fires next
systemd-analyze calendar '*-*-* 03:00:00'# Real output from the test machine: a valid schedule, and one that does not parse
Normalized form: *-*-* 03:00:00
Next elapse: Sun 2026-11-01 03:00:00 EET
(in UTC): Sun 2026-11-01 01:00:00 UTC
Failed to parse calendar specification '*-*-* 3am': Invalid argumentThen load and start it:
# Load the new units, start the timer, and confirm that it has a next run
sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
systemctl list-timers restic-backup.timerThe lab used the same units with a 10 second OnCalendar=*:*:0/10, AccuracySec=1s and a mounted folder in place of /srv/data, so each drill fits on a screen. The default AccuracySec is one minute, which blurs a timer that fires every 10 seconds.
5. Drill 1: a normal run
# Start one run by hand and read the last two pings; at a real monitor, open the check page instead
sudo systemctl start restic-backup.service
tail -n 2 pings.log# Real output from the test machine: a start ping, then a success ping
01:23:46 /ping/demo-check/start
01:23:47 /ping/demo-checkThe start ping lets a monitor that supports it measure how long each run takes and notice a run that starts and never finishes.
6. Drill 2: a loud failure
The password file held a wrong password for one run:
# Real output from the test machine: the unit failed, the monitor got a failure ping, and the journal names restic's exit status 12
unit exit: 1
01:23:47 /ping/demo-check/start
01:23:48 /ping/demo-check/fail
restic-backup.service: Main process exited, code=exited, status=12/n/aThe same path covers a missing repository (restic exit status 10, observed on the test machine) and a locked repository (exit status 11, from the restic documentation). A second run started while the first still held the lock failed with exit status 1 and sent the same failure ping.
7. Drill 3: the empty folder that "succeeds"
# Unmount the disk, then back up the now empty mount point with restic directly
umount /srv/data
restic backup /srv/data | tail -n 2; echo "exit status: ${PIPESTATUS[0]}"# Real output from the test machine: restic backed up nothing and exited 0
processed 0 files, 0 B in 0:00
snapshot a5a75342 saved
exit status: 0restic does not treat an empty source as an error, so a plain success ping would have gone out every night with nothing behind it. The mountpoint -q line in the unit makes the service fail instead:
# Real output from the test machine: the same unmounted folder, now run through the unit
unit exit: 1
01:23:50 /ping/demo-check/start
01:23:50 /ping/demo-check/failIf the source is not a mount point, a check on the content works as well, for example ExecStartPre=/bin/sh -c 'test -n "$(ls -A /srv/data)"'. Leave it out for a source that can legitimately be empty.
8. Drill 4: the silent stop
The timer was running, then it was switched off, the way a reinstall or a careless systemctl disable would do it:
# Real output from the test machine: the timer is listed and the last success ping is 2 seconds old
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2026-11-01 01:24:10 EET 6s Sun 2026-11-01 01:24:00 EET 2s ago restic-backup.timer restic-backup.service
UP: last success ping 2 s ago# Real output from the test machine: no timer is listed, no job fails, and after the grace period the check goes DOWN
NEXT LEFT LAST PASSED UNIT ACTIVATES
DOWN: last success ping 31 s ago (period 10 s + grace 15 s)Nothing failed here, so no failure ping could have been sent. The UP and DOWN lines come from a three-line script that stood in for the monitor in the lab. A real monitor does this comparison for you and sends the alert.
9. After the first week
- Alert on the other side too. The monitor should email you when pings stop. Test that once by disabling the timer on purpose and waiting out the grace period.
- Retention and integrity get their own checks.
restic forget --pruneandrestic checkrun as separate units with separate checks, because a backup that works says nothing about whether old snapshots are being removed or whether the repository still reads cleanly. - A ping proves the job ran, not that the data restores. Schedule a restore test as well and watch it with its own check; the use case above covers that second check, and Test a backup restore before you need it has the procedure.
- Expect an occasional false alarm. If the network is down at the moment of the success ping, the monitor sees a missing ping. The
--retry 3helps, and a grace period of a few hours absorbs the rest.
10. Verify, rollback and prevention
- Verify:
systemctl list-timers restic-backup.timershows a next run, the monitor shows a success ping from the last night, and one forced failure (a temporary wrong password) produced a failure alert. - Rollback:
sudo systemctl disable --now restic-backup.timerstops the schedule. Pause the check at the monitor first, or it will alert. The backup data and the repository are not touched. - Prevention: keep the ping URL in the root-only file, put a mount check in front of every backup of a mounted folder, set a timeout on every long-running oneshot unit, and run
systemd-analyze calendarandsystemd-analyze verifyon every unit file you change.
11. Common mistakes
- Alerting only on failures. A disabled timer, a host that was switched off, a schedule typo or an empty source produces silence, not an error, so the monitor must also expect a success ping on time.
- Sending the success ping before restic exits 0. A partial snapshot (exit code 3) and a wrong password (exit code 12) are failures.
- Running both approaches, cron with Bash and systemd units, on the same job.
- Letting a second run start while the first holds the repository lock.
flock -nmakes the second run exit at once with status 1. - Adding a content check for a source that can legitimately be empty. Leave that check out for such a source.
- Trusting the setup without forcing the failures. Check that a temporary wrong password produced a failure alert, and that switching the timer off raises an alert after the grace period.
- Sharing the ping URL. Anyone who has it can report a failed backup as successful.
References
Official reference
Technical steps are original explanations; consult the upstream documentation for version-specific details.
Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.
Found this useful?
Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.