SRE · Beginner · 12 min
Test a backup restore before you need it
Test a backup restore before you need it: catch partial backups by exit code, restore to a clean folder, compare checksums, verify the repository.
A backup job that reports success only proves that it ran. This runbook catches a partial backup by its exit code, restores to a clean folder, compares the result with checksums recorded before the backup, verifies the repository, and measures the restore time. Commands were run with restic 0.18.1; tar, rsync and database restores are covered too.
On this page
See it run
A short replay of real output from commands in this guide, captured on a WSL2 Ubuntu test machine (restic 0.18.1, local repository, throwaway data and password). The typing is replayed, and the host name, account and MAC addresses are redacted. Your output will differ, and a test machine is not a production server.
Text version of this demo
ops@web-01:~$ restic backup -q data; echo "exit status: $?" error: ~/demo/backup-lab/data/config/signing-key.pem: open data/config/signing-key.pem: permission denied Warning: at least one source file could not be read exit status: 3 ops@web-01:~$ restic snapshots --compact ID Time Host Tags Size -------------------------------------------------------- f761666f 2026-10-31 20:47:08 web-01 585.967 KiB -------------------------------------------------------- 1 snapshots ops@web-01:~$ restic restore latest --target restore restoring snapshot f761666f of [~/demo/backup-lab/data] at 2026-10-31 20:47:08.348703065 +0200 EET by ops@web-01 to restore Summary: Restored 7 files/dirs (585.967 KiB) in 0:00 ops@web-01:~$ cd restore/data && sha256sum -c --quiet ../../manifest.sha256 sha256sum: ./config/signing-key.pem: No such file or directory ./config/signing-key.pem: FAILED open or read sha256sum: WARNING: 1 listed file could not be read ops@web-01:~$ restic check --read-data 2>&1 | tail -n 1 no errors were found ops@web-01:~$ RESTIC_PASSWORD_FILE= RESTIC_PASSWORD=wrong-password restic snapshots; echo "exit status: $?" Fatal: wrong password or no key found exit status: 12
A backup job that runs every night and reports success tells you one thing: the job ran. It does not tell you whether every file was read, whether the data can be restored, how long that takes, or whether anyone still has the password. You find out during an incident, which is the worst moment. A restore test answers all of it in a few minutes: restore to a clean place, compare with what should be there, and time it.
Commands and output were captured with restic 0.18.1 (from the Ubuntu 26.04 archive) and a local repository, with three photos, a config file, and a key file that the backup user could not read. The same steps work with tar or rsync backups (section 7). Database restores are described in section 6 but were not run here.
0. Rules of engagement
- Restore to a new, empty folder or a separate machine. Never restore over the live data to test.
- Use the credentials you would have in a disaster. If the repository password only exists on the server you are backing up, the backup dies with the server.
- Treat the restored data as sensitive. It holds the same secrets as production. Delete it after the test.
1. A backup that "worked"
# Real output from the test machine: restic saved a snapshot, and exited with status 3
error: ~/demo/backup-lab/data/config/signing-key.pem: open data/config/signing-key.pem: permission denied
Warning: at least one source file could not be read
exit status: 3# Real output from the test machine: the snapshot list looks normal
ID Time Host Tags Size
--------------------------------------------------------
f761666f 2026-10-31 20:47:08 web-01 585.967 KiB
--------------------------------------------------------
1 snapshotsThe snapshot exists, and it is missing a file. restic documents exit code 3 as "backup could not read some source data". A wrapper script or cron job that only treats 1 as failure, or only checks that a snapshot appeared, reports success every night.
# In the backup script: anything but 0 is a failure, including 3 (partial snapshot)
restic backup /srv/data
status=$?
[ "$status" -eq 0 ] || { echo "backup failed with exit status $status" >&2; exit "$status"; }Common reasons for unreadable files: the backup runs as a user without access to some files, files are removed while the backup reads them, or a network mount is not mounted at that moment.
2. Restore to a clean folder
# The latest snapshot into an empty folder; restic recreates the original path below the target
restic restore latest --target /tmp/restore-test# Real output from the test machine
restoring snapshot f761666f of [~/demo/backup-lab/data] at 2026-10-31 20:47:08.348703065 +0200 EET by ops@web-01 to restore
Summary: Restored 7 files/dirs (585.967 KiB) in 0:00"Restored 7 files/dirs" is not proof of a complete restore. It counts what was in the snapshot, not what should have been.
3. Compare with what should be there
The strongest comparison is a list of checksums recorded before the backup, kept next to the backup or in the backup itself:
# Before the backup: record a checksum for every file
(cd /srv/data && find . -type f -exec sha256sum {} + | sort -k2) > /srv/manifest.sha256
# After the restore: check every file against that list
cd /tmp/restore-test/srv/data && sha256sum -c --quiet /srv/manifest.sha256# Real output from the test machine: the file that was never backed up
sha256sum: ./config/signing-key.pem: No such file or directory
./config/signing-key.pem: FAILED open or read
sha256sum: WARNING: 1 listed file could not be readWith --quiet, only problems are printed. Silence means every file was restored with the same content. Without a manifest, compare the restored tree with the live data (diff -rq /srv/data /tmp/restore-test/srv/data), keeping in mind that live data has changed since the backup.
4. Check the repository itself
A complete snapshot can still sit on damaged storage. restic can verify that every stored pack is readable and matches its checksum:
# Read and verify all data in the repository (use --read-data-subset=10% for large repositories)
restic check --read-data# Real output from the test machine
no errors were foundrestic check without --read-data only checks the structure, which is faster. Reading a different 10 percent subset every week covers the whole repository over time.
5. Can you still open it?
# Real output from the test machine: the repository with the wrong password
Fatal: wrong password or no key found
exit status: 12There is no recovery from a lost restic password: the data is encrypted with it. Store the password (and the repository location and credentials) in a password manager or vault that does not depend on the backed-up machine, and test the restore with that copy.
6. Databases: restore into a scratch database
Copying the files of a running database can give you a backup that does not start, because the files were copied at different moments. Back up with the database's own tool, then restore the dump into a scratch database and query it. These are the usual commands; they were not run on the test machine:
- PostgreSQL:
pg_dump -Fc DBNAME > db.dump, thencreatedb scratch && pg_restore -d scratch db.dump. - MySQL and MariaDB:
mysqldump --single-transaction DBNAME > db.sql, then load it into an empty database. - SQLite:
sqlite3 app.db ".backup app-backup.db", thensqlite3 app-backup.db 'PRAGMA integrity_check;'.
A query that proves something real (row counts of the main tables, the newest order's timestamp) is better than "it loaded without errors".
7. tar and rsync backups
- tar:
tar -tzf backup.tar.gz | headlists the contents.tar -xzf backup.tar.gz -C /tmp/restore-testrestores into a clean folder. GNU tar exits with1when files changed while being read, and2on fatal errors such as unreadable files. - rsync:
rsync -a --dry-run --checksum --itemize-changes /srv/data/ BACKUP/data/lists every file that differs between the source and the copy, without copying anything.
8. Measure, then schedule
- Restore time: run the restore with
time. If a full restore takes six hours, that is your recovery time, whatever the plan says. - Data age: the newest snapshot's time is the most data you can lose. The
Timecolumn ofrestic snapshotsshows it. - Schedule the restore test like the backup: a weekly script that restores the latest snapshot (or a sample of files) to a temporary folder, checks the manifest, measures the time, deletes the folder, and exits non-zero on any problem. Give it its own heartbeat check, separate from the backup's, and let the script ping it with the result. It then alerts when the test fails or does not run at all, and a green backup check cannot hide a restore test that has stopped. The full setup, with exit-status alerts and a dead man's switch, is in the use case Backup monitoring with heartbeat checks (cron and a script). If a systemd timer starts the backup, Monitor a restic backup from a systemd timer does the same with units.
9. Verify, rollback and prevention
- Verify: the last restore test restored every file in the manifest,
restic checkfound no errors, the restore time is written down, and the restore test's own check shows a ping from its last scheduled run. - Rollback: a restore test changes nothing in production. Delete the test folder afterwards (
rm -rf /tmp/restore-test), because it holds production data. - Prevention: keep at least one copy off the machine and off the site, make the backup credentials unable to delete old snapshots where the storage allows it (append-only or object lock), review the exit codes of the backup job (not only its last line), and give the restore test its own heartbeat check so it cannot stop unnoticed.
10. Common mistakes
- Restoring over the live data to test. Restore to a new, empty folder or a separate machine.
- Keeping the repository password only on the server you are backing up. If the credentials you would have in a disaster do not work, the backup dies with the server.
- Leaving the restored data on disk. It holds the same secrets as production; delete it after the test.
- Reading "Restored 7 files/dirs" as proof of a complete restore. It counts what was in the snapshot, not what should have been, so compare the restore with the source to find the file that was never backed up.
- Ignoring files the backup could not read. Common reasons are a backup user without access to some files, files removed while the backup reads them, and a network mount that is not mounted at that moment.
References
Official reference
Technical steps are original explanations; consult the upstream documentation for version-specific details.
Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.
Found this useful?
Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.