Blog

How to monitor backup jobs: verified heartbeat pings, size and duration checks, restore tests and the right alerts

Backups fail quietly. Send a heartbeat only after a verified run, check size and duration, automate restore tests and route failures to the person who owns them.

To monitor backup jobs reliably, have the backup script ping a heartbeat monitor only after it has verified its own output: the archive is readable, the size is plausible and the offsite copy exists. If the ping does not arrive within the expected window, you get alerted. Add a scheduled restore test with its own heartbeat, and route both alerts to the person who owns the data, not to whoever happens to be on call at 3 a.m.

Why backup jobs fail silently

Most backup failures produce no error anyone sees. Cron mails output to a local mailbox nobody reads. The job runs, exits 0 and writes a file, but the file is useless. These silent failures usually come in a few shapes:

  • The job never runs. The server was rebuilt and the crontab was not migrated, the systemd timer is disabled, or the host itself is gone.
  • The job runs but the output is broken. In pg_dump app | gzip > dump.gz, the exit code is gzip's. If pg_dump dies halfway, the shell still reports success and you get a truncated archive.
  • The output is valid but incomplete. Credentials changed and the dump user now sees only some tables, or a misconfigured exclude pattern skips the uploads directory. The file exists, it is just much smaller.
  • The local copy is fine, the offsite copy is not. The upload token expired, the bucket policy changed, or the disk filled up and the sync step was never reached.
  • The job hangs. A lock, a stalled network mount or a slow remote keeps it running forever. No failure, no success, no new backup.

An HTTP check on your website catches none of this. You need a check that expects a signal from the job itself and treats the absence of that signal as the failure. That is what heartbeat monitoring does, and if the basic mechanism is new to you, the article on heartbeat monitoring for cron jobs covers it. This guide focuses on what makes backups different: a backup that ran is not the same as a backup you can restore.

How to monitor backup jobs step by step

  1. Create a heartbeat monitor for each backup job, with the period matching the schedule (for example 24 hours for a nightly dump).
  2. Set the grace period to the job's normal duration plus a buffer, so a slow night does not alert but a hung job does.
  3. Make the script fail loudly: set -euo pipefail, and wrap long steps in timeout.
  4. Verify the output inside the script: archive integrity, completion marker, minimum size and size relative to the previous run.
  5. Verify the offsite copy, not just the local file.
  6. Send the heartbeat ping as the very last line, only if every check passed.
  7. Schedule an automated restore test with its own, separate heartbeat.
  8. Route both monitors to the owner of the backups, with escalation if a second run is missed.

Send the heartbeat only after a verified successful run

The most common mistake is pinging at the start of the script, or pinging unconditionally with ; curl ... at the end of a crontab line. Both turn the monitor into a "cron is alive" check. A backup heartbeat should mean "a usable backup exists offsite", not "the script started".

The pattern is verify-then-ping: every check is a command that exits non-zero on failure, set -e aborts the script at the first failure, and the ping is unreachable unless everything before it succeeded. Here is a PostgreSQL example uploading to S3:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR=/var/backups/db
FILE="$BACKUP_DIR/app-$(date +%F).sql.gz"
MIN_BYTES=50000000          # set from your own known-good backups
START=$(date +%s)

# fail instead of hanging forever
timeout 2h pg_dump --format=plain app | gzip > "$FILE"

# 1. archive is readable
gzip -t "$FILE"

# 2. dump reached the end (pg_dump writes this line last)
zcat "$FILE" | tail -n 5 | grep -q "PostgreSQL database dump complete"

# 3. size is plausible: absolute floor and max 20% shrink vs previous run
SIZE=$(stat -c %s "$FILE")
PREV=$(ls -1t "$BACKUP_DIR"/app-*.sql.gz | sed -n 2p)
PREV_SIZE=$(stat -c %s "$PREV" 2>/dev/null || echo 0)
[ "$SIZE" -ge "$MIN_BYTES" ]
[ "$PREV_SIZE" -eq 0 ] || [ "$SIZE" -ge $((PREV_SIZE * 80 / 100)) ]

# 4. offsite copy exists and has the same size
aws s3 cp "$FILE" "s3://example-backups/db/"
REMOTE=$(aws s3api head-object --bucket example-backups \
  --key "db/$(basename "$FILE")" --query ContentLength --output text)
[ "$REMOTE" -eq "$SIZE" ]

echo "backup ok size=$SIZE duration=$(( $(date +%s) - START ))s"

# 5. only now: tell the monitor
curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL" > /dev/null

A few details matter here. pipefail makes the dump pipeline fail if pg_dump fails, which fixes the masked exit code. The completion marker check catches dumps that were cut off without an error. curl -f treats HTTP errors as failures, and --retry 3 keeps a brief network blip on the backup host from causing a false alarm.

Keep the log on disk so the alert has context when it fires:

30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

The same idea applies to other tools. With restic or borg, run the backup, then restic check or borg check on the latest snapshot, then ping. With file-level rsync jobs, compare file counts or total bytes against a floor before pinging.

Check backup size and duration, not just exit codes

Size and duration are the two cheapest signals that something changed, and both are easy to reason about.

Size thresholds

Use two rules together. An absolute floor catches the empty or near-empty file: set it well below your smallest known-good backup, for example half of it. A relative rule catches partial backups: data grows slowly, so a backup that is 20% smaller than yesterday's deserves a look. Pick the percentage from your own history. A database with heavy nightly cleanup needs a looser rule than an append-only one.

Do not alert on growth. Large jumps can be worth a log line, but they rarely mean the backup is broken.

Duration and grace period

Duration is handled in two places. The timeout wrapper turns a hang into a failure, so the script exits and never pings. The heartbeat grace period is the outer safety net for everything else, including the case where the host is down and the script never starts.

A practical way to size the grace period: if the nightly job starts at 02:30 and normally takes 40 minutes, set the period to 24 hours and the grace period to around 2 hours. A run that drifts to 90 minutes passes quietly; a run that has not reported by mid-morning alerts. If duration keeps creeping up week over week, that is a capacity signal worth tracking in the log, before it becomes a timeout.

Restore tests: the only proof a backup works

Every check so far proves the file is plausible. Only a restore test proves you can get your data back. It also tests things the backup script cannot see: whether the restore tooling version still matches, whether the encryption key is available, and how long a real restore takes.

Automate it on a separate host or container, weekly is a reasonable starting point, and give it its own heartbeat with a 7-day period:

#!/usr/bin/env bash
set -euo pipefail

LATEST=$(aws s3 ls s3://example-backups/db/ | sort | tail -n 1 | awk '{print $4}')
aws s3 cp "s3://example-backups/db/$LATEST" /tmp/restore.sql.gz

dropdb --if-exists restore_test
createdb restore_test
zcat /tmp/restore.sql.gz | psql -q -v ON_ERROR_STOP=1 restore_test

# sanity checks on real data
USERS=$(psql -tA restore_test -c "select count(*) from users")
[ "$USERS" -gt 1000 ]
FRESH=$(psql -tA restore_test -c \
  "select max(created_at) > now() - interval '26 hours' from orders")
[ "$FRESH" = "t" ]

dropdb restore_test
curl -fsS -m 10 --retry 3 "$RESTORE_HEARTBEAT_URL" > /dev/null

The freshness query is the important one. It catches the backup that has been restoring perfectly for weeks, just from a replica that stopped replicating. Pick a table that receives writes every day and a threshold that matches your backup schedule plus some slack.

Once or twice a year, also do a manual restore drill with the runbook in hand. The automated test proves the data is there; the drill proves a person under pressure can find the credentials, the bucket and the commands.

Alerting the right person when a backup fails

A missed backup is serious but rarely urgent in the way a down website is. Your data is still there; you have lost one restore point. Treat it accordingly:

  • Route to the owner. Backups belong to whoever runs the database or the infrastructure. Send the alert to that person or a team channel, not to a generic alerts room where it scrolls away.
  • Business hours for the first miss. One failed nightly backup can usually wait until morning. Paging someone at night for it trains people to ignore pages, a pattern described in detail in the guide to reducing alert fatigue.
  • Escalate on the second miss. Two consecutive failures, or a failed restore test, means you are running without a safety net. That should reach the current on-call person, with on-call schedules and do-not-disturb hours deciding who that is.
  • Put context in the alert. Name the job, host and schedule in the monitor name, for example db-backup prod-db-1 nightly 02:30, and point to the log path in the monitor description.

Before relying on any of this, break it once on purpose: set MIN_BYTES above the real size, run the job and confirm the alert arrives where you expect it.

Monitoring backup jobs with m0nitor

If you want a hosted option, m0nitor has push-based heartbeat monitors: each job gets a unique URL to ping, and an alert fires when the expected ping does not arrive within the period plus grace period. The heartbeat monitor setup docs describe the configuration. Alerts can go to email, Slack, Teams, SMS and other channels, and alert workflows can route a backup monitor to a different channel than your website checks or call a webhook. Any heartbeat service, or a self-hosted one, works with the scripts above as long as it alerts on a missing ping.

FAQ

Should the heartbeat ping happen at the start or the end of a backup job?

At the end, after all verification steps have passed. A ping at the start only proves the scheduler fired. Some services also support a start signal to measure duration, but the success ping must come last and must be skipped when any check fails.

How do I monitor backup jobs that run on Windows or in Kubernetes?

The pattern is the same. In PowerShell, run the backup and checks with $ErrorActionPreference = 'Stop' and call Invoke-WebRequest to the heartbeat URL at the end. In a Kubernetes CronJob, add the ping as the final command in the container, so it only runs if the earlier commands exit 0.

How often should I test restoring a backup?

Automate a restore test at least weekly for important databases, and run a manual restore drill with your runbook once or twice a year. Daily automated restores make sense if the data changes fast and the restore fits comfortably in the window.

Is an exit code of 0 enough to know a backup succeeded?

No. Pipelines can hide a failed dump behind a successful compression step, and a dump with missing permissions can exit cleanly with partial data. Check integrity, a completion marker, a minimum size and the offsite copy before treating the run as successful.

Product

  • Pricing
  • Features
  • Integrations
  • Security
  • Free network check

Features

  • Uptime monitoring
  • Status pages
  • Alert workflows
  • Incident management
  • Dependencies
  • AI assistant
  • Global infrastructure
  • On-call schedules
  • Enterprise SSO

Use cases

  • Websites for free
  • Cron jobs
  • SSL certificates
  • Databases
  • IP blacklists
  • GDPR compliance

Resources

  • What is a status page?
  • Documentation
  • API
  • MCP server
  • Monitoring glossary
  • Free tools
  • SLA calculator
  • SSL checker
  • DNS lookup
  • Blog
  • Changelog
  • About
  • Contact

Compare

  • Free uptime monitoring
  • UptimeRobot
  • Better Stack
  • Pingdom
  • StatusCake
  • Uptime Kuma
  • Hyperping
  • Atlassian Statuspage
  • Instatus
  • Cachet
  • Freshping