Every system has failures it cannot page you about

Victoria Shinder20 September 2026, 21:404 min read

Monitoring is organised around errors, because errors are easy to define: something raised, something returned a status in the wrong range, something timed out. Every alerting system in ordinary use is a function of that definition. Which means there is an entire class of defect that is structurally invisible to it — the operation that succeeds, and is wrong anyway.

Cloudflare published a good example this month. TLS 1.3 asks the client to commit to a key agreement algorithm in its first packet; guess wrong and the server replies with a HelloRetryRequest and the handshake takes two round trips instead of one. Their measurement found that was happening on roughly half of origin connections. Not one of those handshakes failed. Nothing raised, nothing timed out, no status code was out of range. The protocol was working exactly as specified, and it had been costing a round trip on half the connections for years.

The shape of the class

Once you have the shape, you see it everywhere, and it is always the same shape: a correct result obtained through more work than necessary.

• A page that answers 200 with the front page for a URL that does not exist. Users see a page; crawlers see a duplicate; your error rate is zero.
• A cache that returns a stale-but-valid entry. Every request succeeds. The data is a day old.
• The N+1 query. One hundred and one successful queries where two would have done.
• A retry that succeeds on the second attempt, with the retry budget absorbing a dependency that is quietly half-broken.
• A fallback path that is being taken constantly because the primary is misconfigured. The fallback works. That is the problem.

Each of these is a success by the definition your alerting uses. Each is a real defect. And each survives precisely as long as nobody measures the thing that distinguishes it, because there is no page, no ticket, and no complaint — just a system that costs more than it should and gets slower as it grows.

Why thresholds do not find them

The instinctive answer is "alert on latency". It does not work, for a reason worth being precise about. These defects usually sit inside the normal latency distribution: an extra round trip is 40 ms on a link where the p99 already moves by 200. They are visible as a count, not a duration — how many handshakes retried, how many requests took the fallback, how many queries ran per request. Nobody puts a threshold on a count that is supposed to be nonzero, so nobody has the number at all.

The second reason is that these ratios drift rather than break. Nothing changed on the day the X25519 guess became wrong for a third of the internet; the population it described changed, slowly, while the default stayed correct-looking. A threshold catches a step. It never catches a slope.

What to do instead, concretely

Count the paths, not just the outcomes. For every branch your system can take to reach the same correct answer — retry, fallback, cache miss, slow path, second attempt — emit a counter, and put it next to the request count. The ratio is the instrument. You are not looking for it to be zero; you are looking to know what it is, so that a change in it is legible.

Then look at the ratios once, deliberately, when nothing is wrong. Not on a dashboard you glance at during incidents — as a piece of work, on a quiet afternoon, with the question "is that number what I would have guessed?" Half of them will be. The interesting half will not, and every one of those is a defect that was never going to announce itself.

And write down the assumption each default encodes. Every default is a claim about the population it serves — that most origins prefer X25519, that most responses fit in the buffer, that most retries are transient. A claim is checkable. A default that nobody ever wrote down as a claim becomes furniture, and furniture does not get measured.

The failures that page you are the ones you will fix. The ones that succeed are the ones you will pay for.