How We Measure Detection Accuracy (and Admit When We're Not Sure)

Alert fatigue is not a volume problem. It's a precision problem. You don't ignore your pager because it goes off too often — you ignore it because too many of those pages turn out to be nothing. After the fourth false alarm in a shift, the fifth gets swiped away too, and that's the night the real outage slips through.

Most monitoring tools are tuned to never miss. That sounds right. In practice it's how you end up with a channel nobody reads. We built Epok's detection the other way around — this post is why, and how we measure whether it's working.

Precision vs recall, without the jargon

Two numbers decide whether a detector is useful.

Recall is how much of the real trouble it catches. A detector with perfect recall never misses an incident. Precision is how often it's right when it fires. A detector with perfect precision never cries wolf — every alert is a real one.

The catch: you can't max both at once. Push a detector to catch everything and it flags every blip — precision falls, false positives pile up. Tighten it so every alert is real and it gets cautious — recall falls, a few real ones slip by. Every threshold you set is a vote for one side of that trade.

Here's the part the industry gets backwards. Most tools optimize for recall, because a missed incident is a visible, embarrassing failure ("why didn't we get paged?") while a false positive is an invisible cost — paid by the on-call engineer at 3am, one swipe at a time. So the defaults lean loud. The tool covers itself; you pay the bill in attention.

And that bill compounds. A noisy detector trains the team to distrust the whole system, and an alert nobody believes has zero recall in practice — no matter what the dashboard says.

Why "never miss" quietly fails

Beyond noise, the "never miss" stance pretends the tool is certain when it isn't.

Real telemetry is full of ambiguous signals. A latency percentile that ticks up for one window. An error count that doubles — from two to four. A metric that looks like a spike but is really the partial, still-filling final bucket of a query. A "10x increase" on a number so small the multiple is meaningless. A tool tuned to never miss treats all of these as incidents and lets you sort out the mess. It outsources its uncertainty to you.

That's the real root of alert fatigue. Not that tools alert too much — that they alert with false confidence. They don't distinguish "this is clearly broken" from "this might be something," and they don't tell you which one you're looking at.

Epok's stance: cap the confidence, stay quiet on thin evidence

Epok takes the opposite position. When the evidence is thin, it stays quiet — or it speaks up at a deliberately low severity and tells you it isn't sure. We call this the honest verdict.

Concretely, that shows up in a few rules baked into how detectors emit:

• —A new or unproven detector is born capped. It can't page you at the highest severity until it has run against real traffic long enough to earn it. A brand-new detection method doesn't get to wake you up on day one.
• —Relative changes need an absolute floor. A 10x jump on a sub-millisecond latency, or a "spike" measured in a handful of events, doesn't clear the bar — the multiple is real and the incident isn't.
• —Confidence is capped, not laundered. If a detector lowers its severity because the signal is weak, repetition can't quietly escalate it back up. A thin signal that keeps firing is still a thin signal, not a crisis that grew louder by waiting.
• —Absence is held to a higher bar. A service going quiet can mean it died — or that ingestion hiccuped on our side. We separate those before paging you, because "we stopped receiving your data" is our problem, not your outage.

The point isn't to suppress alerts. It's to make every alert mean something. When Epok pages you, it's saying *we're confident enough that this is worth your attention* — and when it isn't, it says that too, in plain language, at a severity that matches.

This is the same principle behind how Epok drafts root cause: every claim is cited to the exact log line, span, or metric it came from, so you can check the work instead of trusting a black box. A verdict you can audit is one that's allowed to be honest about its limits.

How we actually measure it

A stance is only worth anything if you can test it. Most of the accuracy work is measuring the false positives a "never miss" tool conveniently never counts.

We score detection on a labeled set of incidents and — just as important — on long stretches of boring, healthy telemetry where the correct number of alerts is zero. Precision and recall get measured separately, because a single blended "accuracy" number hides the trade-off we care most about. We track:

• —Precision — of the alerts that fired, how many were real. This is the alert-fatigue number.
• —Recall on the hard classes — not just the obvious error spikes, but the failures that never trip a static threshold: a service that goes silent, a slow cascade, a brand-new error, latency drift across many signals. These are the incidents most tools miss entirely, so they're the ones worth measuring.
• —False positives against quiet baselines — we run detectors across windows we know were healthy and count every alert as a defect. A detector that can't shut up during calm weather is broken, full stop.

Because Epok reads logs, metrics, traces, infrastructure, and RUM together, many would-be false positives die before they're emitted. A latency wobble on its own is ambiguous. The same wobble with no corresponding error rate, no trace-throughput change, and no deploy in the window is usually nothing — and the cross-signal context is what lets the detector hold its fire. Corroboration is itself a precision tool.

When we find a false positive, we don't just tune it away on one account. We catalog the *class* of mistake — the partial trailing bucket, the counter treated like a gauge, the inverted "free space" metric where less is bad — and add a guard so that whole shape of error stops across every detector. Precision isn't a one-time tuning pass. It's a standing discipline.

Won't a quieter tool miss things?

The obvious fear: if it stays silent more often, won't it miss real incidents? That's exactly why we score recall separately and lean hardest on the failure classes other tools never catch at all. Staying quiet on a thin signal isn't the same as being blind — and the failures that genuinely hurt, the silent and slow ones, are what detection-first monitoring exists to catch. You don't earn those by writing more threshold rules; you earn them by letting the tool learn what normal looks like and flagging departures from it.

Zero rules to write, a first real alert in under five minutes. The accuracy work is what makes that first alert worth reading.

Try it

The honest verdict is something you have to feel to trust. Point your telemetry at Epok's 14-day trial, run it across a normal week, and watch how often it pages you versus how often it's right. Pricing is flat — Team $199/month, Growth $599/month, Custom from $5,000/month, overage $0.20/GB, no free-forever tier. Verify current pricing at getepok.dev/pricing.

A detection tool earns its place by being right when it speaks and quiet when it can't be sure. That's the whole job.

FAQ

Is alert fatigue a precision problem or a volume problem?

It's a precision problem. The number of alerts only matters because of how many turn out to be false. A team will happily read a hundred alerts a day if they're all real, and will ignore three a week if two are noise. The fix isn't fewer alerts — it's a higher fraction of true ones, which is precision.

What's the difference between precision and recall in monitoring?

Recall is the share of real incidents a detector catches; precision is the share of its alerts that are real. Tuning a detector to catch more (higher recall) usually means firing on more ambiguous signals (lower precision), and vice versa. Most tools optimize for recall and pay for it in false positives. The two should be reported separately, because a single "accuracy" figure hides the trade-off.

How does Epok reduce false positives in monitoring?

Epok caps detector confidence and stays quiet when evidence is thin: new detectors can't page at top severity until they've proven themselves, relative changes need an absolute floor before they count, and weak signals can't escalate just by repeating. It also reads logs, metrics, traces, infrastructure, and RUM together, so an ambiguous change in one signal that isn't corroborated by the others usually never becomes an alert.

Why doesn't Epok just alert on everything to be safe?

Because alerting on everything is how a monitoring channel becomes one nobody reads. An alert nobody believes has zero real-world recall regardless of what the metrics say. Epok would rather emit a low-severity, clearly-hedged signal — or stay silent — than spend your trust on a guess.

How should I evaluate AI RCA accuracy before trusting it?

Ask whether each claim is cited to specific evidence you can open and check — an exact log line, span, or metric — rather than a summary you have to take on faith. Ask whether the tool reports precision and recall separately, and whether it measures false positives against known-healthy data, not just hits against known incidents. A root-cause engine that can show its work and admit uncertainty is one you can audit; one that can't, you can only believe.