A probe in one city loses twenty seconds of clean network. Your site never noticed. Your phone did. That alert wasn’t slow or badly worded, it was factually untrue, and the person who got up to check it learned exactly one thing: the next alert probably isn’t real either. That lesson is the actual damage, and it outlasts the incident by months.
A Wrong Alert Costs More Than a Late One
Monitoring vendors compete on detection speed because it’s easy to put on a chart. The number that matters more is what fraction of your pages were real, because that’s what sets how fast a human moves when the next one arrives.
Detection latency and alert accuracy trade against each other, and most teams have the dial in the wrong place. Thirty extra seconds of confirmation delay costs you thirty seconds. A month of false positive downtime alerts costs you the response itself: people stop checking immediately, stop reading past the subject line, and eventually route the channel to a folder. The outage you actually needed to hear about then arrives into a system that has been trained to ignore it. We think optimizing for raw speed while tolerating wrong verdicts is a bad trade, and it’s the default in this industry.
Where They Actually Come From
Running probes continuously across regions teaches you that the causes cluster, and that almost none of them are about your site:
- The path, not the endpoint. Transient congestion or a bad route between one probe location and your origin. Your server is healthy and unreachable from exactly one place on earth.
- The probe machine itself. This one is uncomfortable to admit and important. A monitoring instance whose own host has stalled reports the world as broken, because from where it sits, the world is. Any fleet running on shared infrastructure hits this. If your monitoring can’t distinguish “the target is down” from “I am unwell,” it will page you for its own problems.
- Timeouts set below real tail latency. A configurable timeout is a threshold, and a threshold set at 5 seconds on an origin whose p99 is 6 seconds manufactures outages on schedule. This is the slow but not down problem inverted: instead of missing real slowness, you’re inventing fake downtime from it.
- Origins that refuse robots. Bot walls, WAF rules, and rate limits produce a failure that is really a policy decision. The site is up; it just declined to talk to a machine. Nothing about this looks different from an outage in a status code.
- Brittle content checks. A check asserting on a marketing string fails the day marketing edits the string. The page is fine. The assertion is stale.
- DNS, which deserves its own category. Resolution failures at one resolver look identical to origin failures from the outside, and they’re genuinely intermittent. We’ve covered why DNS produces invisible outages that are hard to attribute after the fact.
Notice how many of these are properties of the observer rather than the observed. A single vantage point cannot separate the two, in principle. It has one observation and no way to cross-examine it.
Quorum Is the Only Real Answer
If one probe can’t distinguish a local problem from a global one, ask more probes. That’s the whole idea, and it’s unglamorous enough that plenty of tools skip it.
When a check fails in FlareWarden, that failure isn’t a verdict, it’s a hypothesis. The failure triggers validation requests to other regions, and the verdict comes from what they collectively see: everyone reports up and the original failure is discarded as local, everyone reports down and you get an alert, results are mixed and the monitor is marked degraded rather than down. Checks run from 18 regions across 6 continents, and the confirmation threshold is yours to set — two regions by default, more if you want a higher bar before anyone gets woken.
The second half matters as much and gets less attention: unconfirmed blips shouldn’t rewrite your history. An incident that opens on a single unconfirmed failure and resolves before confirmation arrives stays out of the uptime calendar and out of the percentage, because counting it would mean publishing a downtime figure we don’t believe. Anything still open is always visible, since hiding an ongoing incident would be the worse error. Without that rule, a monitoring tool’s own network noise quietly becomes your customer-facing SLA number.
The Honest Counterargument
Quorum is not free, and two objections are fair.
It costs detection time. Confirming across regions adds seconds before an alert fires, and a genuine 20-second outage might resolve before the quorum completes, meaning you never hear about it. If you want the rawest, earliest possible signal and you’re prepared to triage the noise yourself, a single-region check with an aggressive timeout will beat any confirmation scheme on latency. That’s a real trade, not a straw man. Our position is that most teams overvalue those seconds and undervalue their own willingness to answer the phone in month six.
It can’t fix a hostile origin. Here’s the limit worth stating plainly: if an origin blocks automated traffic by pattern, several regions can all be blocked and all agree. Consensus reports down, confidently and wrongly. Quorum defends against uncorrelated noise; it’s useless against a correlated cause. The fix there is allowlisting the monitor or checking an endpoint built for the purpose, and no amount of cross-checking substitutes.
If You’re Living With Noisy Alerts Right Now
This works with any tool, ours or not:
- Find out which region reports the failures. If one location generates most of your alerts, you have a probe problem, not a site problem. Any tool that won’t tell you where a failure was observed is hiding the most useful field it has.
- Raise the timeout above your real p99, then look at what’s left. A large share of “downtime” is a threshold argument.
- Require two consecutive failures, or two locations, before paging. One of the two is usually a setting you already have.
- Allowlist your monitor at the WAF or CDN, and check whether your “outages” correlate with traffic spikes rather than deploys.
- Re-examine content assertions. Assert on something structural, not on copy someone edits.
- Compare alert windows against your deploy log. Restarts during deploys produce honest-looking failures that no one needs to be paged about.
Fixing the noise floor is what makes the rest of your alerting worth building. A team that believes its monitors responds in minutes; a team that doesn’t responds when the customer emails. That gap is the entire return on getting false positive downtime alerts under control, and it’s larger than any feature you could add on top of an alerting setup nobody trusts.
Key Takeaways
- Accuracy beats latency — thirty seconds of confirmation delay costs seconds; a month of false alarms costs you the human response permanently.
- Most false positives describe the observer — the network path, the probe host’s own health, and a timeout set too low, none of which are facts about your site.
- One vantage point cannot cross-examine itself — quorum across regions is the only structural fix, with a threshold you choose rather than inherit.
- Unconfirmed blips must not enter your history — a single-region failure that resolves before confirmation should never appear in a published uptime percentage.
- Consensus fails against correlated causes — an origin that blocks robots gets every region agreeing on the wrong answer; allowlist the monitor instead.
Tired of getting up for outages that never happened? Start monitoring free with FlareWarden — two regions confirm every failure before an alert leaves, 15 monitors, no credit card required.