Blog
Backups fail quietly. Send a heartbeat only after a verified run, check size and duration, automate restore tests and route failures to the person who owns them.
To monitor backup jobs reliably, have the backup script ping a heartbeat monitor only after it has verified its own output: the archive is readable, the size is plausible and the offsite copy exists. If the ping does not arrive within the expected window, you get alerted. Add a scheduled restore test with its own heartbeat, and route both alerts to the person who owns the data, not to whoever happens to be on call at 3 a.m.
Most backup failures produce no error anyone sees. Cron mails output to a local mailbox nobody reads. The job runs, exits 0 and writes a file, but the file is useless. These silent failures usually come in a few shapes:
pg_dump app | gzip > dump.gz, the exit code is gzip's. If pg_dump dies halfway, the shell still reports success and you get a truncated archive.An HTTP check on your website catches none of this. You need a check that expects a signal from the job itself and treats the absence of that signal as the failure. That is what heartbeat monitoring does, and if the basic mechanism is new to you, the article on heartbeat monitoring for cron jobs covers it. This guide focuses on what makes backups different: a backup that ran is not the same as a backup you can restore.
set -euo pipefail, and wrap long steps in timeout.The most common mistake is pinging at the start of the script, or pinging unconditionally with ; curl ... at the end of a crontab line. Both turn the monitor into a "cron is alive" check. A backup heartbeat should mean "a usable backup exists offsite", not "the script started".
The pattern is verify-then-ping: every check is a command that exits non-zero on failure, set -e aborts the script at the first failure, and the ping is unreachable unless everything before it succeeded. Here is a PostgreSQL example uploading to S3:
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR=/var/backups/db
FILE="$BACKUP_DIR/app-$(date +%F).sql.gz"
MIN_BYTES=50000000 # set from your own known-good backups
START=$(date +%s)
# fail instead of hanging forever
timeout 2h pg_dump --format=plain app | gzip > "$FILE"
# 1. archive is readable
gzip -t "$FILE"
# 2. dump reached the end (pg_dump writes this line last)
zcat "$FILE" | tail -n 5 | grep -q "PostgreSQL database dump complete"
# 3. size is plausible: absolute floor and max 20% shrink vs previous run
SIZE=$(stat -c %s "$FILE")
PREV=$(ls -1t "$BACKUP_DIR"/app-*.sql.gz | sed -n 2p)
PREV_SIZE=$(stat -c %s "$PREV" 2>/dev/null || echo 0)
[ "$SIZE" -ge "$MIN_BYTES" ]
[ "$PREV_SIZE" -eq 0 ] || [ "$SIZE" -ge $((PREV_SIZE * 80 / 100)) ]
# 4. offsite copy exists and has the same size
aws s3 cp "$FILE" "s3://example-backups/db/"
REMOTE=$(aws s3api head-object --bucket example-backups \
--key "db/$(basename "$FILE")" --query ContentLength --output text)
[ "$REMOTE" -eq "$SIZE" ]
echo "backup ok size=$SIZE duration=$(( $(date +%s) - START ))s"
# 5. only now: tell the monitor
curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL" > /dev/nullA few details matter here. pipefail makes the dump pipeline fail if pg_dump fails, which fixes the masked exit code. The completion marker check catches dumps that were cut off without an error. curl -f treats HTTP errors as failures, and --retry 3 keeps a brief network blip on the backup host from causing a false alarm.
Keep the log on disk so the alert has context when it fires:
30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1The same idea applies to other tools. With restic or borg, run the backup, then restic check or borg check on the latest snapshot, then ping. With file-level rsync jobs, compare file counts or total bytes against a floor before pinging.
Size and duration are the two cheapest signals that something changed, and both are easy to reason about.
Use two rules together. An absolute floor catches the empty or near-empty file: set it well below your smallest known-good backup, for example half of it. A relative rule catches partial backups: data grows slowly, so a backup that is 20% smaller than yesterday's deserves a look. Pick the percentage from your own history. A database with heavy nightly cleanup needs a looser rule than an append-only one.
Do not alert on growth. Large jumps can be worth a log line, but they rarely mean the backup is broken.
Duration is handled in two places. The timeout wrapper turns a hang into a failure, so the script exits and never pings. The heartbeat grace period is the outer safety net for everything else, including the case where the host is down and the script never starts.
A practical way to size the grace period: if the nightly job starts at 02:30 and normally takes 40 minutes, set the period to 24 hours and the grace period to around 2 hours. A run that drifts to 90 minutes passes quietly; a run that has not reported by mid-morning alerts. If duration keeps creeping up week over week, that is a capacity signal worth tracking in the log, before it becomes a timeout.
Every check so far proves the file is plausible. Only a restore test proves you can get your data back. It also tests things the backup script cannot see: whether the restore tooling version still matches, whether the encryption key is available, and how long a real restore takes.
Automate it on a separate host or container, weekly is a reasonable starting point, and give it its own heartbeat with a 7-day period:
#!/usr/bin/env bash
set -euo pipefail
LATEST=$(aws s3 ls s3://example-backups/db/ | sort | tail -n 1 | awk '{print $4}')
aws s3 cp "s3://example-backups/db/$LATEST" /tmp/restore.sql.gz
dropdb --if-exists restore_test
createdb restore_test
zcat /tmp/restore.sql.gz | psql -q -v ON_ERROR_STOP=1 restore_test
# sanity checks on real data
USERS=$(psql -tA restore_test -c "select count(*) from users")
[ "$USERS" -gt 1000 ]
FRESH=$(psql -tA restore_test -c \
"select max(created_at) > now() - interval '26 hours' from orders")
[ "$FRESH" = "t" ]
dropdb restore_test
curl -fsS -m 10 --retry 3 "$RESTORE_HEARTBEAT_URL" > /dev/nullThe freshness query is the important one. It catches the backup that has been restoring perfectly for weeks, just from a replica that stopped replicating. Pick a table that receives writes every day and a threshold that matches your backup schedule plus some slack.
Once or twice a year, also do a manual restore drill with the runbook in hand. The automated test proves the data is there; the drill proves a person under pressure can find the credentials, the bucket and the commands.
A missed backup is serious but rarely urgent in the way a down website is. Your data is still there; you have lost one restore point. Treat it accordingly:
db-backup prod-db-1 nightly 02:30, and point to the log path in the monitor description.Before relying on any of this, break it once on purpose: set MIN_BYTES above the real size, run the job and confirm the alert arrives where you expect it.
If you want a hosted option, m0nitor has push-based heartbeat monitors: each job gets a unique URL to ping, and an alert fires when the expected ping does not arrive within the period plus grace period. The heartbeat monitor setup docs describe the configuration. Alerts can go to email, Slack, Teams, SMS and other channels, and alert workflows can route a backup monitor to a different channel than your website checks or call a webhook. Any heartbeat service, or a self-hosted one, works with the scripts above as long as it alerts on a missing ping.
At the end, after all verification steps have passed. A ping at the start only proves the scheduler fired. Some services also support a start signal to measure duration, but the success ping must come last and must be skipped when any check fails.
The pattern is the same. In PowerShell, run the backup and checks with $ErrorActionPreference = 'Stop' and call Invoke-WebRequest to the heartbeat URL at the end. In a Kubernetes CronJob, add the ping as the final command in the container, so it only runs if the earlier commands exit 0.
Automate a restore test at least weekly for important databases, and run a manual restore drill with your runbook once or twice a year. Daily automated restores make sense if the data changes fast and the restore fits comfortably in the window.
No. Pipelines can hide a failed dump behind a successful compression step, and a dump with missing permissions can exit cleanly with partial data. Check integrity, a completion marker, a minimum size and the offsite copy before treating the run as successful.