Most published incident runbook templates open with a step like “confirm the outage is real and determine scope”. That step exists because the alert that woke you can’t be trusted. Fix that, and the first third of your runbook deletes itself.
The First Steps Belong in Your Monitoring, Not Your Document
Open any runbook template and the opening moves are the same: is it actually down, is it just me, which regions, what did the server return, when did it start. Every one of those is a question a monitoring system can answer before it ever pages a human, and every one of them is dead weight when it sits in a document instead.
A page that arrives already carrying “unreachable from four regions since 02:14, HTTP 502 from all four” has done the triage. A page that says “site down” has handed you a research project at the worst possible hour. The difference is not documentation quality. It’s whether the alert was verified before it fired.
This is why we run failed checks past other regions before declaring an incident, and why false positive alerts are worth engineering against rather than documenting around. A team that has been burned by three bogus 3 a.m. pages writes a runbook whose first step is “check whether this is real”, and then follows that step during the one incident that was real, losing four minutes to it.
So: before adding a diagnostic step to a runbook, ask whether the alert could have carried the answer instead. Usually it could.
What’s Left Is Shorter Than You Think
Strip the verification steps and a small-team runbook has four parts:
- What the alert means. One sentence, in plain language, about which customer-facing thing is broken. Not the metric that tripped. “Checkout is failing” rather than “payments-api 5xx rate above threshold”.
- What to check first, with the command. The two or three most likely causes in the order they’re worth checking, each with the exact command or dashboard URL. Copy-pasteable, with the expected output written next to it so a half-awake person can tell success from failure.
- How to make it stop. The mitigation, which is frequently not the fix. Roll back, fail over, disable the feature flag, put up the maintenance page. Mark anything destructive.
- Who to wake and when. One name and one threshold. “If not resolved in 30 minutes, call Priya.”
That’s it. A runbook that fits on one screen gets followed. A twelve-section document with a compliance appendix gets skimmed for the command block and abandoned, which means the eleven other sections were work you did for nobody.
One Runbook Per Alert, Not Per System
The organizing mistake small teams make is writing “the database runbook” or “the API runbook”. Nobody experiences a database. They experience an alert at 2 a.m.
Write one runbook per alert that can page you, and put a link to it in the alert itself. The alert is the only artifact you’re guaranteed to be looking at during an incident, so if the runbook isn’t reachable from there, it doesn’t functionally exist. FlareWarden’s incident records keep the check history, the affected regions, and the failure reason together for exactly this reason: the context should travel with the notification instead of waiting in a wiki.
An immediate benefit of the per-alert rule is that it audits your alerting. If you can’t write a useful runbook for an alert, that alert probably shouldn’t page anyone. There’s no shame in demoting it to a daily digest, and it’s the single fastest way to cut the noise described in alert fatigue.
Runbooks Rot, and Yours Will Too
The uncomfortable part. A runbook is a snapshot of a system that keeps moving, and it starts decaying the moment you save it. The commands drift, the dashboard gets renamed, the service moves to a different host, and none of that generates a notification.
Two habits keep them alive, and both are cheap:
- Edit during the incident, not after. When a step is wrong at 3 a.m., fix that line before you close the laptop. The retro-scheduled cleanup does not happen; the thirty-second edit does.
- Note the date of last use. A runbook nobody has opened in a year is either for an incident that stopped happening or for one that will surprise you. Both are worth knowing.
We’d skip the quarterly review ritual entirely for a team under about ten people. It’s a process built for organizations that can staff it, and imposing it on a two-person team produces a calendar invite that gets declined four times and then deleted.
When You Don’t Need a Runbook at All
Runbooks earn their keep for incidents that are recurring, time-pressured, and handled by someone who didn’t build the system. Miss any one of those and the honest answer is to skip it.
If one person wrote the whole stack and is the only one who’ll ever respond, a runbook is a document they’re writing to themselves. Their time goes further into fixing the recurring failure, or into making the alert carry more context. If an incident has happened exactly once, you don’t have a pattern yet, you have an anecdote, and writing procedure around an anecdote is how teams end up with forty documents describing thirty-nine things that never recurred.
The genuinely good reason to write one anyway: you’re about to go on holiday and somebody else is holding the phone. That single constraint justifies more documentation than any maturity model will.
Where it does pay off, it pays off in recovery time rather than detection. Detection is a monitoring problem; the time between alert and resolution is where a good runbook shows up in the numbers.
Key Takeaways
- Delete the verification steps. “Is it really down” and “which regions” belong in a validated alert, not in a document a human reads at 2 a.m.
- Four parts is enough. What broke in customer terms, what to check with exact commands, how to stop the bleeding, and who to escalate to at what time.
- One runbook per alert, linked from the alert. If it isn’t reachable from the notification, it won’t be opened during the incident.
- An alert you can’t write a runbook for shouldn’t be paging you. Use the exercise as an audit and demote what fails it.
- Fix the runbook during the incident. The correction you make at 3 a.m. is the one that actually gets made.
Want alerts that arrive already triaged, so the runbook can skip straight to the fix? Start monitoring free with FlareWarden — cross-region validation before anything pages you, 15 monitors, no credit card required.