Stop Building Monitoring by Hand

You should not be hand-building your monitoring. Static thresholds and hand-built dashboards both decay the moment you stop tending them, and they share the same fatal assumption: that you can predict, in advance, how your system will break. The alternative is detection that learns each service's normal behavior — per hour of day, per day of week, across logs, metrics, traces, infrastructure, and RUM — and tells you when something deviates, with zero rules to write.

Here's the meeting every team has eventually. "We need better alerting." Someone opens a spreadsheet. You list every service. You decide on numbers. Error rate above X. Latency above Y. CPU above Z. You burn a day encoding those in Prometheus, CloudWatch, or Datadog. Two weeks later, three rules are too noisy and get silenced. Five never fire because the thresholds were too cautious. And the next real incident is something nobody put in the spreadsheet, so not one of the rules covers it.

That cycle repeats every six months, sometimes every quarter. The rules pile up. Coverage never feels finished. This is the tax on building monitoring by hand, and most teams just pay it.

Why static thresholds rot

A static threshold assumes the system behaves the same way all the time. "Alert when error rate exceeds 5%" treats Monday at 9am exactly like Sunday at 3am. But those are not the same system. Your traffic is different, your error floor is different, your dependencies are under different load. The same 5% might be background noise during a peak and a five-alarm fire at 3am when almost nothing should be moving.

So teams add nuance. "Alert at 5% during business hours, 2% otherwise." Now it's two rules per metric and you still haven't handled holidays, deploy windows, or the slow drift as your product grows and last quarter's "normal" stops being normal. Every threshold is a guess dressed up as a number. Why 5% and not 3%? Why two seconds and not 1.5? You tune for hours and land somewhere between too noisy and too quiet, never both-right.

The deeper rot is conceptual. A static rule is a fossil of a failure you already survived. You write it after the incident and hope the next failure rhymes. Production doesn't oblige — it keeps finding new ways to break. The failures that actually take you down are the ones no rule was watching, because nobody had imagined them yet. You can't write a threshold for a class of incident you've never seen.

Why hand-built dashboards rot too

Dashboards are the other half of the do-it-yourself trap, and they fail the same way for a different reason. A good dashboard takes an hour to build and forever to maintain — across service renames, new deploys, changed metric labels, and the teammate who built it leaving. For a five-person team, that upkeep is a real, recurring cost. It's why so many Grafana instances quietly decompose into a graveyard of panels nobody trusts.

But the design flaw is worse than the maintenance bill. Nobody looks at a dashboard until something is already broken. It's an investigation tool, not a detection tool. By the time you're staring at one, you already know there's a problem — you're there to figure out the shape of it. Which means the dashboard didn't catch the incident. Something else did: a customer complaint, a Slack ping, a spike in support tickets. The dashboard was step two pretending to be step one.

The honest version of "we monitor with dashboards" is "we expect a human to keep watching them." That doesn't survive contact with reality. Asking people to stare at dashboards to catch problems is like asking them to watch security-camera feeds — it works for about an hour, then attention drifts and the thing you were watching for happens in the gap.

What learned baselines do instead

Detection should not start with a question you wrote down. It should start with the system learning what normal looks like and telling you when reality departs from it.

Instead of "is this above a line I drew," the question becomes "is this different from what's normal for *right now*." The tool builds a baseline from your actual data. Monday 9am gets its own expected range; Sunday 3am gets a different one. When the observed value falls outside the expected range *for that specific window*, that's an anomaly — no rule required.

This catches two things thresholds structurally cannot. Novel failures, because you no longer have to predict them — anything that deviates hard enough surfaces on its own. And context-dependent failures: fifty errors a minute at 3am is a disaster; fifty errors a minute at peak is Tuesday. A learned baseline knows the difference because it learned both.

The part that's changed since teams first tried this: it isn't only about log volume anymore. The same baseline thinking runs across signals. A service's log cadence going quiet, trace throughput dropping, a metric exporter that stopped reporting, an error fingerprint that has never appeared before, latency drifting up without ever crossing a fixed line — these are all deviations from normal, and they're exactly the failure classes a hand-written rule was never going to cover. We dig into why those slip past threshold alarms in detection-first observability, and how the cross-signal picture gets assembled in from alert to cited root cause.

Why teams kept doing it the hard way

Anomaly detection isn't a new idea. The reason most teams still hand-write rules is that doing it *well* is genuinely hard: you need baseline computation, seasonal pattern recognition, a sane statistical model, and enough operational scar tissue to set sensitivity so it's neither a fire hose nor silent.

Datadog and Grafana both ship anomaly-detection features. The catch is they're add-ons you configure per metric — you still pick which metrics, still set sensitivity, still do manual work per signal. Better than raw thresholds. Still building monitoring by hand, one knob at a time.

We built Epok so detection runs by default. Point your telemetry at it and it watches every service's volume, error patterns, and cadence across signals, learns the baseline, and alerts on deviation — no metrics to select, no sensitivity sliders, no rule library to maintain. First alert lands in under five minutes because there is nothing to configure before detection starts.

Where hard thresholds still earn their keep

To be fair to the spreadsheet: some thresholds are correct. Disk usage above 90% should always alert, regardless of what's "normal" — there's no graceful version of running out of disk. Payment success below 99.9% should always alert. These are business SLOs and physical limits, not anomaly problems, and a hard line is the right tool for them.

Epok supports threshold rules for exactly these cases. The point isn't to ban thresholds — it's to demote them. Use a hard line for hard constraints. Use learned detection for everything else. The failure mode of the last decade was making the manual thing the *primary* thing.

And dashboards keep their place too. Once you know something is wrong, a good dashboard helps you understand scope and impact, and they're fine for status screens and standups. Detection should be automatic; investigation can be manual. That's the split that actually holds up.

The best monitoring system is the one that catches what you didn't think to look for. You can't get there by typing more rules.

Point your stack at Epok and watch what it flags in the first day — no dashboards to build, no rules to write. Start the 14-day trial at getepok.dev (verify current pricing at getepok.dev/pricing).

FAQ

Should I stop writing alert rules entirely?

No — demote them, don't delete them. Hard thresholds are the right tool for fixed business constraints and physical limits, like disk above 90% or payment success below 99.9%. For everything else, learned anomaly detection catches more with no rules to author, because it flags deviation from each service's normal behavior instead of from a number you guessed. Use thresholds for the exceptions; let detection handle the default.

Why do static thresholds miss real incidents?

A static threshold assumes the system behaves identically at all times, so it treats Monday 9am like Sunday 3am even though traffic, error floors, and load are completely different. It also requires you to predict failure modes in advance — you write the rule after an incident and hope the next failure matches. The failures that cause the most damage are usually novel, so no existing rule covers them.

What is anomaly detection in monitoring?

Anomaly detection learns what "normal" looks like for each service and time window from your real data, then flags values that deviate significantly from that learned baseline. Unlike a static threshold, it adapts to context: it knows fifty errors a minute is fine at peak traffic but a disaster at 3am. It catches novel failures because it isn't matching against a predefined rule — it's watching for departure from normal.

Do I still need dashboards if detection is automatic?

Yes, for investigation. Dashboards are good for understanding the scope and impact of a problem you already know about, plus status screens and standups. They make a poor *detection* mechanism because nobody watches them until something is already broken — by then a customer or a Slack message has usually told you first. The useful split is automatic detection, manual investigation.

How is this different from Datadog or Grafana anomaly detection?

Datadog and Grafana offer anomaly detection as add-on features you configure per metric — you still choose which metrics to watch and what sensitivity to use, which is still manual work per signal. Epok runs detection automatically across every service's logs, metrics, traces, infrastructure, and RUM with no per-metric setup, learning baselines by hour of day and day of week. First alert typically lands within five minutes of connecting telemetry.