We earn commissions when you shop through the links below, at no extra cost to you. Our guides and technical recommendations are written independently of these relationships: a commission never decides what we recommend, how we diagnose a problem or what we publish, and the content does not depend on it. Affiliate disclosure

SRE · Beginner · 12 min

Test a backup restore before you need it

Test a backup restore before you need it: catch partial backups by exit code, restore to a clean folder, compare checksums, verify the repository.

A backup job that reports success only proves that it ran. This runbook catches a partial backup by its exit code, restores to a clean folder, compares the result with checksums recorded before the backup, verifies the repository, and measures the restore time. Commands were run with restic 0.18.1; tar, rsync and database restores are covered too.

See it run

A short replay of real output from commands in this guide, captured on a WSL2 Ubuntu test machine (restic 0.18.1, local repository, throwaway data and password). The typing is replayed, and the host name, account and MAC addresses are redacted. Your output will differ, and a test machine is not a production server.

Text version of this demo
ops@web-01:~$ restic backup -q data; echo "exit status: $?"
error: ~/demo/backup-lab/data/config/signing-key.pem: open data/config/signing-key.pem: permission denied
Warning: at least one source file could not be read
exit status: 3

ops@web-01:~$ restic snapshots --compact
ID        Time                 Host  Tags    Size
--------------------------------------------------------
f761666f  2026-10-31 20:47:08  web-01          585.967 KiB
--------------------------------------------------------
1 snapshots

ops@web-01:~$ restic restore latest --target restore
restoring snapshot f761666f of [~/demo/backup-lab/data] at 2026-10-31 20:47:08.348703065 +0200 EET by ops@web-01 to restore
Summary: Restored 7 files/dirs (585.967 KiB) in 0:00

ops@web-01:~$ cd restore/data && sha256sum -c --quiet ../../manifest.sha256
sha256sum: ./config/signing-key.pem: No such file or directory
./config/signing-key.pem: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read

ops@web-01:~$ restic check --read-data 2>&1 | tail -n 1
no errors were found

ops@web-01:~$ RESTIC_PASSWORD_FILE= RESTIC_PASSWORD=wrong-password restic snapshots; echo "exit status: $?"
Fatal: wrong password or no key found
exit status: 12

A backup job that runs every night and reports success tells you one thing: the job ran. It does not tell you whether every file was read, whether the data can be restored, how long that takes, or whether anyone still has the password. You find out during an incident, which is the worst moment. A restore test answers all of it in a few minutes: restore to a clean place, compare with what should be there, and time it.

Commands and output were captured with restic 0.18.1 (from the Ubuntu 26.04 archive) and a local repository, with three photos, a config file, and a key file that the backup user could not read. The same steps work with tar or rsync backups (section 7). Database restores are described in section 6 but were not run here.

What a restore test proves1. Restore (the latest snapshot into an empty folder) [is the data there?] Then 2. Compare (with checksums recorded before the backup) [is it correct?] Then 3. Check the repository (restic check --read-data) [is the storage healthy?] Then 4. Open it for real (with the password and tools you have today) [can you still get in?] Then 5. Time it (the restore, start to end) [how long will it take?].Restorethe latestsnapshot into anempty folderis the data there?Comparewith checksumsrecorded beforethe backupis it correct?Check therepositoryrestic check--read-datais the storagehealthy?Open it forrealwith the passwordand tools youhave todaycan you still getin?Time itthe restore,start to endhow long will ittake?
Each step answers a different question. A backup job that reports success answers none of them.

0. Rules of engagement

  1. Restore to a new, empty folder or a separate machine. Never restore over the live data to test.
  2. Use the credentials you would have in a disaster. If the repository password only exists on the server you are backing up, the backup dies with the server.
  3. Treat the restored data as sensitive. It holds the same secrets as production. Delete it after the test.

1. A backup that "worked"

# Real output from the test machine: restic saved a snapshot, and exited with status 3
error: ~/demo/backup-lab/data/config/signing-key.pem: open data/config/signing-key.pem: permission denied
Warning: at least one source file could not be read
exit status: 3
# Real output from the test machine: the snapshot list looks normal
ID        Time                 Host  Tags    Size
--------------------------------------------------------
f761666f  2026-10-31 20:47:08  web-01          585.967 KiB
--------------------------------------------------------
1 snapshots

The snapshot exists, and it is missing a file. restic documents exit code 3 as "backup could not read some source data". A wrapper script or cron job that only treats 1 as failure, or only checks that a snapshot appeared, reports success every night.

# In the backup script: anything but 0 is a failure, including 3 (partial snapshot)
restic backup /srv/data
status=$?
[ "$status" -eq 0 ] || { echo "backup failed with exit status $status" >&2; exit "$status"; }

Common reasons for unreadable files: the backup runs as a user without access to some files, files are removed while the backup reads them, or a network mount is not mounted at that moment.

2. Restore to a clean folder

# The latest snapshot into an empty folder; restic recreates the original path below the target
restic restore latest --target /tmp/restore-test
# Real output from the test machine
restoring snapshot f761666f of [~/demo/backup-lab/data] at 2026-10-31 20:47:08.348703065 +0200 EET by ops@web-01 to restore
Summary: Restored 7 files/dirs (585.967 KiB) in 0:00

"Restored 7 files/dirs" is not proof of a complete restore. It counts what was in the snapshot, not what should have been.

3. Compare with what should be there

The strongest comparison is a list of checksums recorded before the backup, kept next to the backup or in the backup itself:

# Before the backup: record a checksum for every file
(cd /srv/data && find . -type f -exec sha256sum {} + | sort -k2) > /srv/manifest.sha256
# After the restore: check every file against that list
cd /tmp/restore-test/srv/data && sha256sum -c --quiet /srv/manifest.sha256
# Real output from the test machine: the file that was never backed up
sha256sum: ./config/signing-key.pem: No such file or directory
./config/signing-key.pem: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read

With --quiet, only problems are printed. Silence means every file was restored with the same content. Without a manifest, compare the restored tree with the live data (diff -rq /srv/data /tmp/restore-test/srv/data), keeping in mind that live data has changed since the backup.

4. Check the repository itself

A complete snapshot can still sit on damaged storage. restic can verify that every stored pack is readable and matches its checksum:

# Read and verify all data in the repository (use --read-data-subset=10% for large repositories)
restic check --read-data
# Real output from the test machine
no errors were found

restic check without --read-data only checks the structure, which is faster. Reading a different 10 percent subset every week covers the whole repository over time.

5. Can you still open it?

# Real output from the test machine: the repository with the wrong password
Fatal: wrong password or no key found
exit status: 12

There is no recovery from a lost restic password: the data is encrypted with it. Store the password (and the repository location and credentials) in a password manager or vault that does not depend on the backed-up machine, and test the restore with that copy.

6. Databases: restore into a scratch database

Copying the files of a running database can give you a backup that does not start, because the files were copied at different moments. Back up with the database's own tool, then restore the dump into a scratch database and query it. These are the usual commands; they were not run on the test machine:

  • PostgreSQL: pg_dump -Fc DBNAME > db.dump, then createdb scratch && pg_restore -d scratch db.dump.
  • MySQL and MariaDB: mysqldump --single-transaction DBNAME > db.sql, then load it into an empty database.
  • SQLite: sqlite3 app.db ".backup app-backup.db", then sqlite3 app-backup.db 'PRAGMA integrity_check;'.

A query that proves something real (row counts of the main tables, the newest order's timestamp) is better than "it loaded without errors".

7. tar and rsync backups

  • tar: tar -tzf backup.tar.gz | head lists the contents. tar -xzf backup.tar.gz -C /tmp/restore-test restores into a clean folder. GNU tar exits with 1 when files changed while being read, and 2 on fatal errors such as unreadable files.
  • rsync: rsync -a --dry-run --checksum --itemize-changes /srv/data/ BACKUP/data/ lists every file that differs between the source and the copy, without copying anything.

8. Measure, then schedule

  • Restore time: run the restore with time. If a full restore takes six hours, that is your recovery time, whatever the plan says.
  • Data age: the newest snapshot's time is the most data you can lose. The Time column of restic snapshots shows it.
  • Schedule the restore test like the backup: a weekly script that restores the latest snapshot (or a sample of files) to a temporary folder, checks the manifest, measures the time, deletes the folder, and exits non-zero on any problem. Give it its own heartbeat check, separate from the backup's, and let the script ping it with the result. It then alerts when the test fails or does not run at all, and a green backup check cannot hide a restore test that has stopped. The full setup, with exit-status alerts and a dead man's switch, is in the use case Backup monitoring with heartbeat checks (cron and a script). If a systemd timer starts the backup, Monitor a restic backup from a systemd timer does the same with units.

9. Verify, rollback and prevention

  • Verify: the last restore test restored every file in the manifest, restic check found no errors, the restore time is written down, and the restore test's own check shows a ping from its last scheduled run.
  • Rollback: a restore test changes nothing in production. Delete the test folder afterwards (rm -rf /tmp/restore-test), because it holds production data.
  • Prevention: keep at least one copy off the machine and off the site, make the backup credentials unable to delete old snapshots where the storage allows it (append-only or object lock), review the exit codes of the backup job (not only its last line), and give the restore test its own heartbeat check so it cannot stop unnoticed.

10. Common mistakes

  • Restoring over the live data to test. Restore to a new, empty folder or a separate machine.
  • Keeping the repository password only on the server you are backing up. If the credentials you would have in a disaster do not work, the backup dies with the server.
  • Leaving the restored data on disk. It holds the same secrets as production; delete it after the test.
  • Reading "Restored 7 files/dirs" as proof of a complete restore. It counts what was in the snapshot, not what should have been, so compare the restore with the source to find the file that was never backed up.
  • Ignoring files the backup could not read. Common reasons are a backup user without access to some files, files removed while the backup reads them, and a network mount that is not mounted at that moment.

References

Official reference

restic.readthedocs.io ↗

Technical steps are original explanations; consult the upstream documentation for version-specific details.

Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.

Found this useful?

Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.