The nightly dump ran at 2 a.m. and exited 0. The file it wrote was 20 bytes of gzip header around nothing, because the database password had rotated a week earlier and pg_dump failed inside a pipe that nobody had told to care. Every night since had “succeeded.” The way to monitor database backups is to stop asking whether the job reported an error and start asking whether a good backup arrived.
Three Ways a Backup Fails Without an Error
Error alerts only catch the failures that announce themselves. Backups have three quieter ones:
- It never ran. The server was rebooted and the crontab lived in a user account that got removed. The scheduler moved hosts. Nothing ran, so nothing failed, so nothing emailed.
- It ran and hung. The dump is waiting on a lock, or the destination disk is full and the write is stalled. The process is alive, so the job hasn’t failed either.
- It “succeeded” with garbage. The exit code said 0, but the file is empty, truncated, or unreadable.
An email-on-failure setup catches none of these, and it has a fourth weakness: the alert is sent by the same machine that’s in trouble. If the backup host is down, so is the thing meant to tell you about it.
Flip the Logic: Expect a Ping
A heartbeat monitor (sometimes called a dead man’s switch) inverts the question. You tell it how often a backup should finish, and the job sends a small HTTP request when it does. If the request doesn’t arrive on time, the monitor raises the alarm from the outside.
That design catches all three failure modes. A job that never ran sends no ping. A job that hung sends a start signal but never a finish. And a job that produced garbage sends no ping either, as long as you put the ping after the checks, which is the whole trick. We cover the broader model in our guide to heartbeat vs cron vs uptime monitoring.
A Backup Script That Earns Its Ping
Here’s a PostgreSQL example. The same shape works for mysqldump, mongodump, or a managed provider’s export command.
#!/usr/bin/env bash
set -euo pipefail
PING="https://app.flarewarden.com/ping/YOUR-TOKEN"
OUT="/backups/app-$(date +%F).sql.gz"
fail() {
curl -fsS -m 10 --retry 3 --data-raw "$1" "$PING/fail" > /dev/null || true
exit 1
}
trap 'fail "backup script exited early on line $LINENO"' ERR
curl -fsS -m 10 --retry 3 "$PING/start" > /dev/null || true
pg_dump --no-owner app_production | gzip > "$OUT"
gzip -t "$OUT" || fail "gzip integrity check failed"
BYTES=$(stat -c %s "$OUT")
[ "$BYTES" -gt 1000000 ] || fail "backup is only $BYTES bytes"
curl -fsS -m 10 --retry 3 -H "Content-Type: application/json" \
-d "{\"message\":\"ok\",\"metadata\":{\"bytes\":\"$BYTES\"}}" \
"$PING/complete" > /dev/null
The lines that do the real work:
set -o pipefail. Without it, a pipeline’s exit status is the status of its last command. Bash’s own manual is explicit about this. Sopg_dump | gzipreports gzip’s success even whenpg_dumpdied on its first line. This single flag is the difference between the scenario in the opening paragraph and a 2 a.m. alert.gzip -t. The GNU gzip manual describes-tas a check of “the compressed file integrity.” It catches a truncated archive, which is what a full disk tends to leave behind.- A minimum size. Pick a floor well under your real backup size (the 1 MB here is a placeholder, not a recommendation) so an empty dump fails loudly. Revisit it when the database grows.
- The
/completeping comes last. Every check above it has to pass first. || trueon the start and fail pings. A monitoring hiccup should never be the reason a backup didn’t run. The complete ping is deliberately allowed to fail the script, because its absence is what alerts you.
stat -c is the GNU form. On macOS or BSD, use stat -f %z.
Configure the Monitor for How Backups Actually Behave
In FlareWarden, a cron monitor sits under the uptime monitor for the site it supports, so the backup’s health shows next to the site’s. The settings that matter for backups:
- Schedule. Use the same cron expression as the job, or a fixed interval.
- Grace period. Size it to your slowest normal run, not your average one. A backup that usually takes 12 minutes but takes 40 on the first of the month needs a grace period that covers the 40.
- Max run duration. This is what turns a hang into an alert. With the
/startping in place, a run that stays in Running past the limit is marked failed as a hung job. Without a start ping, a hang just looks like a late run, which you’d still catch, only later. - Severity. For backups, Notify Only usually fits. A failed backup is urgent for you and invisible to your customers, so it shouldn’t mark the site as degraded on a public status page.
The metadata field in the complete ping isn’t an alert. It records the backup size in the run history, which is where you’ll look when someone asks whether last Tuesday’s dump looked normal. The cron monitor docs list every field and limit, including the 10 KB body cap and the 1,000-character message limit on a fail ping.
A Backup You Haven’t Restored Is a Hope
Our position: the nightly ping proves a plausible file exists. It doesn’t prove you can get your data back. Only a restore proves that.
So add a second job, on a slower schedule (weekly or monthly), that restores the latest backup into a scratch database and runs one sanity query, such as a row count on a table that only ever grows. Give it its own cron monitor with its own ping at the end. Now you have two signals with different meanings:
| Monitor | Schedule | What a ping proves |
|---|---|---|
| Nightly backup | Daily | A dump ran, finished, passed integrity and size checks |
| Restore test | Weekly or monthly | The latest dump actually restores and contains real data |
The restore test is also the honest limit of everything above. A dump can pass gzip -t and a size check while missing a schema that was added last month, or while being taken from a replica that stopped replicating. The size floor catches catastrophes, not subtle drift.
Managed Databases Still Need This
If your provider takes snapshots for you, you don’t need the script, but the silence problem doesn’t go away. Plan changes, paused projects, and retention settings can all stop backups without anything you’d call an error. Our piece on Supabase cron job monitoring covers one version of this: Supabase may pause a Free Plan project after seven days of low activity, and a paused project runs no schedules at all.
The pattern is the same. Have something outside the provider check that the newest backup is recent, and ping a monitor only when it is. Logical dumps you take yourself, on your own schedule, are still worth having on top of provider snapshots, because they’re the copy you control.
How We Think About It
Backups are the textbook case for heartbeat monitoring because the cost is so lopsided. The ping costs one HTTP request a night. The failure it catches usually isn’t discovered until the day you need the backup, which is the worst possible day to discover it. If you only add one cron monitor anywhere, put it on the job that protects your data. Our Laravel scheduled task guide shows the same idea for apps that run backups through a framework scheduler.
Key Takeaways
- Monitor database backups by expecting a ping, not by waiting for an error. Skipped, hung, and empty backups don’t raise errors.
- Turn on
set -o pipefail, orpg_dump | gzipreports gzip’s success when the dump itself failed. - Put the success ping after the integrity and size checks, so a bad file never counts as a good night.
- Use a start ping plus a max run duration to catch a hung dump, and size the grace period to your slowest normal run.
- Schedule a restore test with its own monitor. A backup you’ve never restored is a hope, not a backup.
Want to hear about a missed backup the same night instead of the day you need it? Start monitoring free with FlareWarden: a cron monitor is one curl call, it shares the free plan’s 15-monitor pool, and no credit card is required.