Heartbeat monitoring for cron jobs: catching the jobs that silently stop running

HTTP checks tell you whether a server answers. They tell you nothing about the nightly export that stopped running three weeks ago because someone rotated an SSH key, or the billing job that now exits immediately because a disk is full. Scheduled work fails in a way that produces no traffic, no error page and no log line anybody reads. This article covers how to instrument cron jobs, systemd timers and container schedulers so that silence itself becomes an alert.

Why failed scheduled jobs stay invisible

The classic failure modes of cron are all quiet ones. The crontab was lost during a host migration. The cron daemon is not running inside a minimal container image. The job is there, but PATH in the cron environment is shorter than in your login shell, so python3 is not found. Mail delivery for job output was never configured, so stderr goes nowhere. The job hangs on a network call and never returns, which means it also never fails.

None of these produce an event you can subscribe to. A job that never starts produces no error output, so the only reliable signal is the absence of a ping you expected to receive. That inversion is what heartbeat monitoring does: instead of waiting for a failure message, you wait for a success message and alert when it does not arrive in time.

How a heartbeat check works

A heartbeat check is a URL that your job calls when it finishes successfully. The monitoring side stores the time of the last call and knows two numbers: the expected period between calls and a grace period. If no call arrives within period plus grace, the check goes down and the alert fires. This is a dead man's switch, and the useful property is that it fails safe: if the host is off, the network is broken, the scheduler is gone or the script crashes before the final line, no ping arrives and you hear about it.

Three signals are worth sending, not one:

• start, sent before the work begins, so you can measure duration and detect jobs that are still running past their deadline
• success, sent only after the work completed and was verified
• failure, sent with the exit code and the tail of the output, so the alert carries context instead of just a name

The failure ping is a convenience, not the safety net. If your script segfaults, the failure ping may never be sent either. The missing success ping is what actually protects you.

Choosing the grace period

A grace period that is too short produces flapping alerts every time the database is slow. Too long and your daily report is six hours stale before anyone notices. Derive it from measured runtime rather than from feeling.

Take the slowest normal run you have observed over the last few weeks, add the scheduling jitter you tolerate, then add a margin for one retry. For a job scheduled daily that normally takes four minutes and has peaked at nine, a period of one day and a grace of thirty minutes is reasonable. For a job that runs every five minutes and takes ten seconds, a grace of five minutes means you tolerate exactly one skipped run before alerting, which is usually the right trade.

Two rules keep this honest. First, the grace period must be shorter than the business deadline of the job. If the export has to be in the warehouse by 08:00 and the job starts at 03:17, a grace of four hours is useless. Second, never set the grace period from the failure you had last week. If a run occasionally takes three times as long, fix the job or split it, do not widen the window until the check cannot fail.

Instrumenting a cron job

The naive version appends a curl call to the crontab line:

17 3 * * * /opt/app/bin/export && curl -fsS https://hb.example.com/ping/8f3c0c

This works and is better than nothing, but it has holes. The ping fails silently if DNS hiccups, there is no timeout so a hanging curl can block the next run, and you get no duration and no failure detail. A wrapper script fixes all of that in one place.

#!/usr/bin/env bash
# /usr/local/bin/hb-run  UUID  command [args...]
set -Eeuo pipefail

BASE="https://hb.example.com/ping/$1"
shift

ping() {
  curl -fsS -m 10 --retry 3 --retry-connrefused \
       -o /dev/null "$BASE$1" || true
}

ping "/start"

if out=$("$@" 2>&1); then
  ping ""
else
  code=$?
  printf '%s' "$out" | tail -c 10000 | \
    curl -fsS -m 10 --data-binary @- -o /dev/null "$BASE/fail/$code" || true
  printf '%s\n' "$out" >&2
  exit "$code"
fi

The flags matter. -f makes curl return a non zero status on HTTP errors instead of quietly writing an error page to stdout. -s suppresses the progress meter that would otherwise end up in cron mail. -m 10 caps the total time so a dead endpoint cannot stall the job. --retry 3 absorbs a single packet loss event. The trailing || true is deliberate: a monitoring outage must never turn a healthy job into a failed one.

The crontab entry then looks like this, with an explicit PATH because cron does not inherit your shell environment:

PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=""

17 3 * * * /usr/local/bin/hb-run 8f3c0c /opt/app/bin/export

Exit code zero is not the same as success

Plenty of batch jobs exit cleanly while doing nothing useful. A sync that finds an empty source directory, an export that writes a zero row CSV, a backup that produces a 400 byte file because the dump failed but the pipeline did not. The heartbeat is green and the data is gone.

Send the success ping only after a cheap sanity check on the output. In practice this is two or three lines:

rows=$(wc -l < /var/exports/orders.csv)
if [ "$rows" -lt 100 ]; then
  echo "export produced only $rows rows" >&2
  exit 3
fi

Pick the threshold from the known floor of normal operation, not from an average. If the smallest legitimate daily export in the last quarter had 900 rows, alerting below 100 is safe and still catches the empty file case.

Overlapping runs

When a job starts taking longer than its interval, cron happily starts a second copy. Two importers writing the same table is a worse incident than a skipped run. Guard the job with flock:

*/5 * * * * /usr/bin/flock -n /var/lock/import.lock /usr/local/bin/hb-run 4b19ff /opt/app/bin/import

With -n, flock exits immediately with status 1 if the lock is held. No ping is sent for that run at all, which is exactly right: the previous run is still in progress, and if it finishes in time the success ping still arrives. If the stuck run never finishes, the grace period expires and you get the alert you want.

systemd timers

For timer units you do not need a wrapper around the command, because systemd already knows how the unit ended. Use ExecStopPost, which runs in both the success and the failure case and has $SERVICE_RESULT and $EXIT_STATUS in its environment:

[Service]
Type=oneshot
Environment=HB=https://hb.example.com/ping/2d71aa
ExecStartPre=-/usr/bin/curl -fsS -m 10 -o /dev/null ${HB}/start
ExecStart=/opt/app/bin/export
ExecStopPost=/usr/local/bin/hb-report

And the reporter:

#!/bin/sh
if [ "$SERVICE_RESULT" = "success" ]; then
  curl -fsS -m 10 -o /dev/null "$HB"
else
  curl -fsS -m 10 -o /dev/null "$HB/fail/${EXIT_STATUS:-1}"
fi

The leading dash on ExecStartPre means a failed ping does not abort the unit. Keep OnCalendar and the heartbeat period in sync, and remember that RandomizedDelaySec adds directly to the required grace.

Kubernetes CronJobs and application schedulers

In a Kubernetes CronJob the same wrapper works inside the container, with the heartbeat URL injected from a secret. Two settings deserve attention: concurrencyPolicy: Forbid plays the role of flock, and startingDeadlineSeconds decides whether a job that could not start on time is skipped entirely. If the control plane was unavailable at the scheduled minute and the deadline passes, the run is dropped without an event anybody sees, which is precisely the case a heartbeat catches.

For in process schedulers such as Celery beat, the Laravel scheduler or a Quartz trigger, instrument the task body rather than the process. A running scheduler with a task that silently stopped being registered is a common and very quiet failure.

Routing the alert

A missed heartbeat at 03:20 does not automatically justify waking somebody. Route by deadline, not by severity label. If the result is needed at 08:00 and the job can be rerun in twenty minutes, send a chat notification at night and escalate to a phone call at 06:30 if the check is still down. If the job is a replication heartbeat or a fraud screening run, page immediately.

Give every heartbeat check a short runbook in its description. Four lines are enough: what the job does, where its logs are, the exact command to rerun it manually, and whether a rerun is safe to repeat. The person acting on the alert at 03:20 should not have to reverse engineer a crontab.

Checklist before you call it done

1. Every scheduled job has a heartbeat, including the ones that run weekly or monthly. Rare jobs are the ones that rot.
2. The grace period is derived from measured runtime and is shorter than the business deadline.
3. Success is pinged after an output check, not just after exit code zero.
4. The ping has a timeout, retries, and cannot fail the job itself.
5. Concurrent runs are prevented with flock or concurrencyPolicy.
6. Each check has an owner, an escalation path and a four line runbook.
7. Deleting a job also deletes its check, so nobody learns to ignore a permanently red alert.

Done properly, heartbeat monitoring turns the hardest class of outage to detect into the easiest. You stop relying on someone noticing that a report looks stale, and start relying on a clock.