Use cases · Published 2026-11-01
Backup Monitoring with Heartbeat Checks: Dead Man's Switch, Exit-Status Alerts and Restore Tests
Push-based backup monitoring with restic and Healthchecks: exit-status alerts with the log, a dead man's switch for missed runs, and restore tests.
The problem: silent failure
A backup system can fail in two ways. In the first, the job runs and fails: the error is in a log nobody reads. In the second, the job never runs at all: the cron daemon stopped, the server is off, someone commented out the crontab line during maintenance. That leaves no log, no error and no email. Most backup setups only report the first kind, if anything. The second kind is found when someone needs a restore.
This use case builds monitoring for both kinds, with two numbers in mind:
- RPO (recovery point objective): how much data you can afford to lose. With a nightly backup it is about 24 hours, but only if every night's backup succeeds. A backup that silently stopped two weeks ago means an RPO of two weeks.
- RTO (recovery time objective): how long a restore may take. Only a restore test tells you the real value.
Everything below was run on a test machine: restic 0.18.1 for the backup, a self-hosted Healthchecks 5.0.3 in Docker as the monitor, and Mailpit as a local mail server to catch the alert emails. The check UUIDs, keys and passwords shown are throwaway values from that setup.
Architecture: push-based monitoring and the dead man's switch
There are two ways to watch a scheduled job:
| Approach | How it works | Catches a job that never runs? |
|---|---|---|
| Pull (polling) | The monitoring system connects to the server and checks something, such as the age of the newest snapshot | Yes, if the check itself is right, but it needs access to the server or the repository |
| Push (heartbeat) | The job sends a signal (a "ping") to the monitor when it starts and when it ends | Yes: the monitor alerts when an expected ping does not arrive |
The push approach is called a heartbeat check or dead man's switch: the alarm goes off unless someone keeps confirming that everything is fine. The monitor knows the schedule and a grace period. If no successful ping arrives within schedule plus grace, it raises an alert. The job does not need to be alive to trigger it. That is the point.
The pieces in this setup:
| Component | Role |
|---|---|
| cron | Starts the backup script on schedule |
backup.sh | Runs restic, sends a start ping, then a ping with restic's real exit status and the log tail |
| Healthchecks | Holds the check (schedule, time zone, grace period), receives the pings, decides up or down |
| Alert channel | Email here. Healthchecks also supports webhooks, Slack, Microsoft Teams and many others |
| Restore test | A second, independent check that the backup can actually be restored |
The check: schedule, time zone and grace period
The check is created once, through the Healthchecks web UI or its API, with the same cron expression as the job:
# Create the check through the Healthchecks API (v3): nightly at 02:00 Sofia time, one hour of grace, all alert channels
curl -s http://localhost:8000/api/v3/checks/ -H "X-Api-Key: $HC_API_KEY" \
-d '{"name":"nightly-backup","slug":"nightly-backup","schedule":"0 2 * * *","tz":"Europe/Sofia","grace":3600,"channels":"*","tags":"backup restic"}'# Real output from the test setup (fields picked with jq)
{
"name": "nightly-backup",
"slug": "nightly-backup",
"status": "new",
"schedule": "0 2 * * *",
"tz": "Europe/Sofia",
"grace": 3600,
"ping_url": "http://localhost:8000/ping/cf91d713-53b6-458d-8161-fe91d9a2902c"
}- Time zone: set it on the check. Without it, a cron expression is read as UTC, and the check expects the job one or two hours off.
- Grace period: longer than the slowest normal run plus scheduler delay. A backup that sometimes takes 40 minutes needs more than 40 minutes of grace, or every slow night is a false alarm.
- The ping URL is a secret of sorts: anyone who has it can mark the job as successful. Keep it in the same place as other credentials of the server.
The job: report the real exit status
#!/usr/bin/env bash
# Nightly backup: report start, then the real restic exit status, to the heartbeat check.
set -u
HC_URL="http://localhost:8000/ping/cf91d713-53b6-458d-8161-fe91d9a2902c"
export RESTIC_REPOSITORY=/tmp/uc/repo RESTIC_PASSWORD_FILE=/tmp/uc/.restic-pw
log=$(mktemp)
curl -fsS -m 10 --retry 5 -o /dev/null "$HC_URL/start"
restic backup --quiet /tmp/uc/data >"$log" 2>&1
status=$?
curl -fsS -m 10 --retry 5 -o /dev/null --data-binary @"$log" "$HC_URL/$status"
rm -f "$log"
exit "$status"# The crontab line: the same schedule as the check
0 2 * * * /usr/local/bin/backup.shThe design choices that matter:
$HC_URL/$statussends the exit status itself. Healthchecks treats0as success and any other number as failure. A script that only pings on success hides a whole class of errors behind "no ping", which the check also catches, but only after the grace period and without the reason.- restic exit status 3 is a failure here. restic documents
3as "backup could not read some source data": the snapshot is saved but incomplete. Many wrappers only treat1as an error and call this a success. /startlets the monitor measure the duration and notice a job that starts and then hangs.- The log goes with the ping (
--data-binary @"$log", up to 100 kB by default), so the alert says what went wrong. - No
set -e, andcurlfailures do not stop the backup. The monitor being down must never prevent the backup from running.
Failure modes, tested
1. The job runs and succeeds
# Real output from the test setup
=== run 1 (all files readable)
exit status: 0The check goes up. Nothing is sent.
2. The job runs and fails
One file was made unreadable for the backup user, the classic "permission changed after a deploy" case:
# Real output from the test setup
=== run 2 (one file unreadable)
exit status: 3The check went down at once, without waiting for the grace period, and the alert email arrived two seconds later:
# Real output from the test setup: the alert email (trimmed)
The check "nightly-backup" has gone down.
Reason: received a failure signal.
Schedule: 0 2 * * *
Time zone: Europe/Sofia
Last ping type: Exit status 3
Last Ping Body:
---------------
error: open /tmp/uc/data/config/signing-key.pem: permission denied
Warning: at least one source file could not be readThe alert names the exit status and carries restic's own error line, so whoever reads it knows where to look before logging in.
3. The job never runs: the dead man's switch
A second check, demo-every-minute, expected a ping every 60 seconds with 60 seconds of grace. It got one ping and then nothing, like a job whose server went down:
# Real output from the test setup
11:33:29 pinged
up last_ping=2026-11-01T11:33:29+00:00 next_ping=2026-11-01T11:34:29+00:00
11:35:30 down
down last_ping=2026-11-01T11:33:29+00:00 next_ping=null# Real output from the test setup: the alert emails in Mailpit
2026-11-01T11:35:30.379Z DOWN | demo-every-minute
2026-11-01T11:33:16.167Z DOWN | nightly-backupThe alert came exactly at period plus grace, two minutes after the last ping, with no action from the job. This is the case that no amount of error handling inside the script can cover.
4. Recovery
After fixing the file permissions, the next run succeeded and the monitor sent a recovery notice:
# Real output from the test setup
=== run 3 (fixed)
exit status: 0
2026-11-01T11:35:40.42Z UP | nightly-backup5. The monitor itself is down
With the Healthchecks container stopped, the backup still ran and saved a snapshot. Only the pings failed:
# Real output from the test setup
curl: (7) Failed to connect to localhost port 8000 after 0 ms: Could not connect to server
curl: (7) Failed to connect to localhost port 8000 after 1 ms: Could not connect to server
exit status: 0That is the right behavior for the backup, but it shows the weak spot: a monitor that is down sends no alerts at all. Run the monitor somewhere other than the machines it watches. Use the hosted service, or a self-hosted instance on separate infrastructure, and have something external check the monitor's own /health endpoint.
Variant: a systemd timer instead of cron
If your backups are started by a systemd timer, the same check works with the pings sent from the unit instead of a script: ExecStartPre= for /start, ExecStartPost= for success, and a second unit started by OnFailure= for /fail. The runbook Monitor a restic backup from a systemd timer shows the units, a TimeoutStartSec= for hung runs, a mountpoint -q guard for the empty-folder case and a drill for a switched-off timer.
Restore tests: the second check
A heartbeat proves that the backup job ran and what it reported. It does not prove that the data can be restored. Schedule a restore test as its own job, with its own heartbeat check: restore the latest snapshot to a clean folder, compare it with checksums recorded before the backup, measure the time, and ping with the result. The procedure, with real output, is in Test a backup restore before you need it. The measured restore time is your real RTO.
Checklist
- One heartbeat check per scheduled job, with the job's own schedule and time zone.
- Grace period longer than the slowest normal run.
- The script sends
startand the real exit status, with the log tail. - Non-zero means failure, including restic's
3. - Monitoring failures never stop the job.
- The monitor runs outside the infrastructure it watches, and is itself monitored.
- A separate, scheduled restore test with its own check.
- Alerts go to a channel someone actually reads, and an
UPnotice closes the loop.
References
More from the blog
- SSH Port 22 Open on GCP vs Azure: 12 Hours of Logs · Use cases, 2026-11-16
- What Happens When You Expose SSH to the Internet · Use cases, 2026-11-13
- Run an AI Coding Agent Against Infrastructure Safely · Use cases, 2026-11-05
This article is educational information, not incident-response or legal advice. Verify package status with your operating system, hosting, or runtime vendor.