Kubernetes documents its own uncertainty: a CronJob creates a Job “approximately once per execution time of its schedule,” because there are circumstances where two Jobs might be created, or no Job might be created. That sentence sits in the official docs, and it explains why the backup you scheduled at 3 a.m. sometimes just isn’t there in the morning.
A Schedule Is a Declaration, Not a Receipt
kubectl get cronjob prints the schedule you asked for and the last time a Job was created. Neither field tells you whether the run you expected last night happened.
That gap matters because the evidence expires. .spec.successfulJobsHistoryLimit defaults to 3 and .spec.failedJobsHistoryLimit defaults to 1, so a Job that ran every hour has already discarded the record of anything more than three runs ago. A CronJob that has been failing since Tuesday keeps exactly one failed Job to prove it. By the time somebody asks, the cluster has quietly composted the answer.
Five Documented Causes, in the Order Worth Checking
All five are in the Kubernetes CronJob documentation. None of them raise anything a human would notice.
.spec.suspendis stilltrue. Suspended executions remain scheduled but never start, and the docs are explicit that executions suspended during their scheduled time count as missed Jobs. Somebody paused it during an incident three weeks ago. Check this first, because it costs one command.concurrencyPolicy: Forbidplus a job that overruns. WithForbid, if the previous run hasn’t finished when the next is due, the new run is skipped. A job that gradually slows past its own interval starts eating its successors, and each skip is silent.- More than 100 missed schedules. The controller counts missed schedules since the last scheduled time, and above 100 it declines to start the Job and logs
too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew. Worth reading carefully: the docs say this applies to catch-up scheduling and does not mean the CronJob stops running for good. Plenty of blog posts claim it dies permanently. It doesn’t. startingDeadlineSecondsset too tight. A Job that misses its deadline is skipped, and Kubernetes counts it as a failed Job. Below 10 seconds the CronJob may not be scheduled at all, because the controller only wakes up every 10 seconds.- The time zone isn’t yours. With no
.spec.timeZone, the kube-controller-manager interprets the schedule in its local time zone, which is a property of the control plane rather than of you. PuttingCRON_TZinside.spec.scheduleisn’t supported and fails validation outright, so a schedule that looks time-zone-aware may never have been created.
Causes one and two account for most reports of a Kubernetes CronJob not running without any change to the manifest. Both are visible in kubectl describe cronjob <name> in under a minute.
The Cause You Cannot Configure Away
Numbers one through five are static. Fix them and they stay fixed.
The line quoted at the top of this article is not. Kubernetes says it tries to avoid creating two Jobs or none, and that it does not completely prevent either, which is why the same paragraph tells you to make your Jobs idempotent. That advice is about correctness under duplication, and it quietly concedes the other half: sometimes nothing runs, and the cluster is behaving as designed.
So a plan that ends at “the YAML is right now” is unfinished. Correct configuration lowers your miss rate. It does not reach zero, and nothing inside the cluster is obliged to tell you when a run went missing.
Ask Whether It Ran, Not Whether It Failed
Those are different questions. Polling the cluster answers the first one badly, because absence produces no object to query — the Job that was never created has no status field to read.
Inverting the check fixes that. The job reports in when it finishes, and silence past a deadline becomes the alert. In FlareWarden this is a cron monitor: you register the schedule you expect, the container pings a URL on success, and a ping that doesn’t arrive within the grace period moves the monitor to Late and fires a missed-run alert. The token in the URL is the credential, so there’s no API key to mount and nothing inbound to allow into the cluster.
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-invoice-export
spec:
schedule: "17 3 * * *"
timeZone: "Etc/UTC" # never rely on the control plane's local zone
concurrencyPolicy: Forbid
startingDeadlineSeconds: 3600
successfulJobsHistoryLimit: 5 # default of 3 is a very short memory
failedJobsHistoryLimit: 5 # default of 1 overwrites repeat failures
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: OnFailure
containers:
- name: export
image: registry.example.com/invoice-export:1.8
command:
- /bin/sh
- -c
- |
curl -fsS -m 10 "$PING_URL/start" || true
if ./export-invoices; then
curl -fsS -m 10 "$PING_URL/complete"
else
curl -fsS -m 10 "$PING_URL/fail"
exit 1
fi
env:
- name: PING_URL
valueFrom:
secretKeyRef:
name: flarewarden-ping
key: url
Three details earn their place. The /complete ping is inside the success branch, so it can’t report a win the job didn’t have. The /fail branch turns a crash into an immediate alert instead of one that waits out the grace period. And curl -f exits non-zero on an HTTP error, which stops a failed ping from looking like a delivered one.
Raising both history limits is free and worth doing regardless of what monitors the job. The defaults are tuned for a tidy cluster, not for the afternoon you need to reconstruct what happened.
Size the Grace Period to the Controller, Not the Cron Expression
Most heartbeat setups fail here, in the direction of being too strict. The controller checks every 10 seconds, admission and image pulls take time, and Forbid legitimately defers a run behind its predecessor. A two-minute grace period on an hourly job pages you for documented behaviour, and you have rebuilt alert fatigue with extra YAML.
Our position: 15 minutes minimum for anything hourly or faster, and 45 to 60 for daily jobs whose pods pull large images. You still learn about a genuinely skipped nightly export hours before anyone downstream notices missing invoices, which was the whole point. The same trade shows up when you monitor Laravel scheduled tasks or chase a GitHub Actions scheduled workflow that didn’t run: the platform’s own timing sloppiness sets the floor, not your ideal.
If a job’s duration matters as much as its completion, ping /start too and set a max run duration. A pod that hangs on a locked table then reports as a hung job rather than a missed one, which points at a different fix entirely.
Not Every CronJob Needs a Pager
Instrument all of them and you’ll train yourself to swipe the alerts away. The line we’d draw: watch the CronJobs that move money, write data something else reads later, publish artifacts other people consume, or send customer-facing mail. Leave the log rotators, cache warmers, and namespace tidiers alone, because their failure self-corrects on the next tick and costs nothing.
A metrics roll-up that skips a night is an inconvenience. A billing export that skips a night becomes a support queue. Cron and heartbeat monitoring is on every FlareWarden plan including the free one, so what you watch should come down to judgment rather than a licence tier.
Key Takeaways
- The cluster does not keep the evidence — history limits default to 3 successful and 1 failed Job, so proof of a missed or repeatedly failing run disappears quickly.
- Check
suspendandconcurrencyPolicyfirst — a paused CronJob and aForbidpolicy behind an overrunning job cause most silent misses, and both show up inkubectl describe. - The 100-missed-schedule error is not permanent — the controller declines that catch-up batch and logs it, but the CronJob keeps running, contrary to a lot of published advice.
- Kubernetes admits scheduling is approximate — the docs say two Jobs or no Job may be created, so correct manifests reduce misses without eliminating them.
- Only an outside expectation catches absence — a heartbeat that alerts on silence detects the run that was never created, which no query against the cluster can do.
Want to know the night a CronJob quietly skipped its run? Start monitoring free with FlareWarden — cron and heartbeat monitors on every plan, 15 monitors, no credit card required.