Skip to main content

Service Degradation: The Third Answer Your Monitor Needs

Service degradation is the state most monitoring has no box for: what it means, the three answers a check returns, and where consensus stops helping.

FLAREWARDEN
FlareWarden Team
7 min read

A 403 from a bot wall and a 502 from a dying application look equally red on most dashboards. Only one of them is evidence your site is alive: the 403 means DNS resolved, TCP connected, TLS completed, and the origin chose to answer. Sorting those two apart is the whole problem underneath the word degradation, and most tooling has only two boxes to sort them into.

What Service Degradation Actually Means

Service degradation is when a service is still answering but no longer answering well enough: responses are slow, a fraction of requests fail, or part of what should come back is missing.

The concrete version is more useful than the definition. Your checkout page loads quickly for most visitors and takes eleven seconds for the ones whose session lands on the replica that’s rebuilding an index. Nothing is down. Nothing returns an error. Some fraction of those customers close the tab, and you have no idea how many.

That event has a start, an end, a customer impact, and no red on the status board. It is an outage in every sense that matters to the person trying to pay you, and in no sense that a pass/fail check can express.

The Three Answers, Not Two

An HTTP check gets back one of three meaningfully different things, and collapsing them into two is where the reporting goes wrong.

The first is a normal response: 2xx or 3xx. The service answered, and it answered the way it should. In our checker that’s up, with no further argument.

The second is a refusal. A 401, 403, 407, 418, 429, or 451 means the origin received your request, understood it, and declined it. That is a policy decision by a running server, not an availability failure, so we classify the whole set as up and flag the check as blocked rather than counting it against anyone’s uptime. We didn’t start there. Probing one host from eighteen regions inside the same second looks like a burst to any rate limiter, and we watched our own fan-out provoke 429s from a service that answered a single probe with a clean 200. Counting that as downtime would have published an outage we caused.

The third is everything else: a 5xx, a connection failure, a timeout. Nobody answered, or the thing that answered was broken. That’s down.

The reason this taxonomy belongs in a degradation article is that the middle category is where degradation hides. A service shedding load with 429s is degraded, not down, and a monitor that files it as an outage tells you the wrong thing twice: it exaggerates the incident and it corrupts the uptime figure you report afterward. We wrote about the specific case of a monitor being refused by a firewall separately, because it comes up constantly.

A Failing Part Doesn’t Have to Mean a Failing Site

The other half of the answer is severity, and it’s the half that vendors rarely expose.

FlareWarden’s sub-monitors (cron jobs, content checks, dependency checks) each carry one of three severity settings, and the setting decides what a failure does to the parent service:

SeverityWhat a failure does
CriticalParent monitor is marked down, high-priority alerts fire
DegradedParent monitor is marked degraded, standard alerts fire
Notify OnlyNotification is sent, parent status is untouched

That middle row is service degradation as an actual state rather than an adjective. Your nightly analytics export failing does not mean your storefront is down, and saying so would be a lie that shows up on your status page and in your uptime history. Setting it to degraded records the truth: something is broken, the site is serving, and a human should look before morning.

The position worth taking here: severity is a product decision, not a technical one. Ask what the customer loses when this specific piece breaks. If they lose nothing today, it isn’t critical, no matter how much the failure annoys you.

The Shapes Degradation Takes

Four patterns cover most of what you’ll actually see:

  • Slow but complete. Every request succeeds, p95 latency has tripled. The most common shape and the one pass/fail checks are blindest to.
  • Partially failing. Some percentage of requests error while the rest are fine. A single check has a matching percentage chance of noticing, which is to say it will usually miss.
  • Structurally incomplete. The page returns 200 and renders, but the product grid is empty because the search backend timed out. Status codes cannot see this. Content assertions can.
  • Regional. The service is healthy from three continents and broken from a fourth, usually a CDN or DNS problem rather than an origin one.

Each shape needs a different check to catch it, which is the honest reason “just monitor for degradation” is not actionable advice. Latency thresholds catch the first, content checks catch the third, and only multi-region probing catches the fourth.

Where Consensus Helps and Where It Stops

Multi-region checking is the standard recommendation for the regional shape, and it works. It also has a limit that nobody advertises, so here’s ours.

When a check fails, we re-test from other regions before declaring anything, and the rule is unanimous: a single region reporting the service reachable is enough to call the original failure a false alarm. That deliberately biases toward under-reporting. We would rather miss a brief regional blip than page someone at 3 a.m. about a router in Frankfurt.

The interesting case is what happens when every validating region gets refused instead of answered. A walled quorum has verified nothing in either direction, so we leave the monitor’s state exactly as it was rather than let regions that never saw the service vouch for it. An inconclusive answer recorded as inconclusive beats a confident wrong one, and this is the kind of edge that only shows up once you’re running the fleet.

Consensus tells you whether a failure is real. It tells you nothing about whether a success was good, which is why it doesn’t replace latency baselines. Anomaly detection and threshold alerting cover that ground, and they answer a different question than availability checking does.

What Monitoring Genuinely Cannot See

Synthetic checks watch the paths you told them to watch. Degradation that lives four steps into an authenticated funnel, in a query that only slows under a specific tenant’s data volume, or in a mobile client’s retry loop is invisible to them, and no amount of check frequency fixes that.

This is the case for keeping expectations calibrated rather than buying more probes. External monitoring answers “is it reachable and does it look right from out here” with high confidence and low cost. It does not answer “are all my users having a good time,” and a vendor who tells you otherwise is selling something. Real user monitoring answers the second question, at the price of only working once real users are already suffering.

The related trap is contractual. Most uptime SLAs define availability as reachability, so a service can be degraded for six hours and remain fully compliant. We went through what those guarantees actually promise in detail; the short version is that your customers’ definition and your contract’s definition were never the same document.


Key Takeaways

  • A refusal is not an outage. 401, 403, 407, 418, 429, and 451 prove the origin is serving and declining. Counting them as downtime inflates incidents and corrupts uptime history.
  • Degradation needs to be a state, not an adjective. Severity tiers let a failing component mark a service degraded instead of forcing a false choice between “down” and “fine”.
  • The four shapes need four different checks. Latency thresholds, error-rate sampling, content assertions, and multi-region probing each catch one and miss the others.
  • Consensus validates failures, not successes. Cross-region agreement kills false alarms; it says nothing about whether a passing check was fast enough.
  • Set severity by customer impact. If a broken component costs your customers nothing today, it is not critical, however irritating it is to you.

Want a monitor that can tell “degraded” from “down” instead of guessing? Start monitoring free with FlareWarden — 15 monitors, cross-region validation, and per-component severity on every incident, no credit card required.