Alerting hygiene

Read this before you suppress anything

Suppression stops the notification, not the alert. Both mute schedules and maintenance windows are notify-time filters. Detectors keep running, the alert is still created, still deduplicated, still escalated, still auto-resolved, and still shows up in the alert list and the audit trail. Only the page, the Slack message and the email are withheld.

This is deliberate — you want the record of what happened during the deploy — but it is the single most common way people conclude that suppression "did not work". If you mute a service and then watch alerts appear in the UI, that is the system behaving correctly. The test is whether your phone stayed quiet.

Two consequences worth planning around. Alerts suppressed by a maintenance window are stamped as maintenance-suppressed and carry a window title, so you can tell "we chose not to page" apart from "we never noticed" when you read them back afterwards. The title recorded is that of the first window active at the time, so if you routinely run overlapping windows, do not treat that title as proof of which one did the suppressing.

Second: suppression is not replayed when it lifts. An alert that fired inside a window and is still firing after it closes does not generate a catch-up notification — it is by then an already-open alert, and open alerts only re-notify when they escalate. Check the alert list when a window ends rather than waiting to be told.

The lifecycle you are muting

An alert has two states: firing and resolved. Everything between them is automatic.

Those four defaults — 30 minute suppression, 5 minute grouping, 15 minute auto-resolve, 4 hour force-resolve — are the account defaults. Per-rule cooldowns are set on the rule itself; see custom rules.

Mute schedules vs maintenance windows

These are two different mechanisms and the names do not make that obvious. The distinction is recurring clock-time vs a one-off calendar interval, and they key off different things:

Practical rule: a planned deploy or migration is a maintenance window — it has a title, an owner, and an end. A standing pattern is a mute schedule — the 02:00 batch load that always spikes error volume. Using a mute schedule for a one-off means remembering to delete it, which nobody does.

Mute schedules

The trap: a schedule is unique per stream. Posting a second schedule for a stream you have already muted does not add a second window — it overwrites the first one in place and re-enables it. If you want a stream quiet on weeknight evenings and all weekend, that is one schedule whose days_of_week covers both, not two calls. The second call silently replaces the first.

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/mute-schedules \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "stream": "{service=\"batch-loader\"}",
    "days_of_week": [0, 1, 2, 3, 4],
    "start_hour": 22,
    "end_hour": 6,
    "timezone": "America/New_York"
  }'

The fields, and what the API will reject:

The overnight case is the one people get wrong. start_hour: 22 with end_hour: 6 is a single window running 22:00 to 06:00. Written the other way round — start_hour: 6, end_hour: 22 — you have muted the working day and left the night live.

The day check is evaluated against the local day in your chosen zone at the moment the alert would notify. For a window that wraps midnight, that means the day-of-week set is read at the instant of the alert, not at the instant the window opened.

# every schedule you have, enabled ones only
curl https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/mute-schedules \
  -H 'X-API-Key: YOUR_API_KEY'

# remove one by its id
curl -X DELETE https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/mute-schedules/17 \
  -H 'X-API-Key: YOUR_API_KEY'

The list returns only enabled schedules. Deleting one that is not yours returns 404 rather than pretending to succeed. In the app these live at Settings → Notifications → Mute Schedules.

Limit: the free tier is capped at 3 mute schedules; the fourth create returns 403. Paid plans are not capped.

Maintenance windows

The trap: omitting services mutes the whole account, not nothing. The field is optional and null means "all services". If you open a window for a database upgrade and forget the service list, you have also silenced every unrelated service for the duration — including the one that breaks because of your upgrade. Name the services unless you genuinely intend a full blackout.

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/maintenance-windows \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "title": "Postgres 16 upgrade",
    "start_at": "2026-07-24T01:00:00Z",
    "end_at":   "2026-07-24T04:00:00Z",
    "services": ["checkout", "orders"],
    "created_by": "deploy-bot"
  }'

title, start_at and end_at are required. Timestamps are ISO 8601 and a trailing Z is accepted. The API rejects, with 400: a missing or empty title, a title over 200 characters, an unparseable timestamp, an end_at at or before start_at, a duration over 7 days, a services value that is not a list of strings, and a services list longer than 100 entries.

created_by is a free-text string you supply — useful when a deploy pipeline opens the window, so the record says which pipeline rather than which human happened to hold the API key.

You can create a window that starts in the future; suppression begins when the clock reaches start_at. That is the intended pattern — open the window as part of the change request, not by scrambling at 01:00.

# most recent first, up to 50 — past, present and future windows
curl https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/maintenance-windows \
  -H 'X-API-Key: YOUR_API_KEY'

# cancel early (or clean up a mistake)
curl -X DELETE https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/maintenance-windows/8 \
  -H 'X-API-Key: YOUR_API_KEY'

The list is every window, past and future, newest start first, capped at 50. There is no update endpoint: to change a window, delete it and create the replacement. Deleting a window that is currently active ends suppression immediately, which is how you cut a maintenance short when the work finishes early.

Service matching is by name, against the service Epok resolves for the alert. A window naming checkout suppresses alerts Epok attributes to checkout; it does not suppress a different service that merely mentions checkout in a message.

A window does not guarantee total silence. It suppresses the notification for alerts raised while it is open. An alert that was already open before the window began, and that escalates during it, can still notify — escalation is the one path a maintenance window does not currently cover, though a mute schedule does. In practice this means opening the window before you start work, rather than after the first symptom has already raised an alert, is what actually keeps the rota quiet. If you need a hard guarantee over something already firing, use a mute schedule or resolve the open alert first.

Boundary, stated plainly: maintenance windows are API-only today. There is no screen for them in the app — create, list and delete them over HTTP, or from whatever runs your deploys.

The weekly digest

The digest is the non-paging channel: a weekly summary of volume, anomalies, new patterns, error-rate movement and deploys, sent to your configured notification channels. It goes out Monday at 04:00 UTC by default, and each account gets at most one per 23 hours regardless of how the checks land.

The useful part is that you never have to guess what it will say. Preview computes the real report over the last 7 days and returns it without sending anything:

curl https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/digest/preview \
  -H 'X-API-Key: YOUR_API_KEY'

{
  "period_start": "2026-07-13T09:00:00+00:00",
  "period_end":   "2026-07-20T09:00:00+00:00",
  "total_log_volume": 41822904,
  "daily_volumes":       [ { "day": "...", "count": 5983211 }, ... ],
  "per_service_volumes": [ { "service": "checkout", "count": 8120044 }, ... ],
  "anomaly_count": 37,
  "new_pattern_count": 12,
  "error_rate_this_week": 4102,
  "error_rate_last_week": 3388,
  "error_rate_change_pct": 21.1,
  "flatline_count": 1,
  "top_error_patterns": [ { "message": "...", "count": 611 }, ... ],
  "deploy_count": 19
}

error_rate_change_pct compares the last 7 days against the 7 before it, and is nullwhenever the prior week had zero errors — a new account's first digest, or a genuinely clean week. That is a divide-by-zero being reported honestly rather than an infinite percentage, not an error. top_error_patterns messages are truncated to 200 characters. Preview is computed live, so on a large account it is a real query, not a cached blob.

Once the preview looks right, send it now rather than waiting for Monday:

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/digest/send \
  -H 'X-API-Key: YOUR_API_KEY'
# -> {"status":"sent","channels_notified":2, ...the same report body as /preview}

channels_notified is the count that actually accepted the send. Watch for zero. With no notification channels configured the call still returns "status": "sent" with channels_notified: 0 — it generated a digest and had nowhere to put it. A per-channel failure is also counted out rather than raising, so a partial number means one destination rejected the message. Configure channels first at Settings → Notifications.

There is a PATCH .../digest with an enabled boolean, exposed in the app at Settings → Notifications → Weekly Digest. Be aware of its current limits before you build against it: the preference is held in the API process and is not durable across a restart, and the scheduled Monday send does not read it. Treat /digest/preview and /digest/send as the dependable pair; if you need the weekly send to stop for your account, ask rather than assuming the toggle held.

Limit: notification channels are capped at 2 on the free tier and unlimited on trial, Team, Growth and Custom plans. The digest fans out to all of them.

Audit trail

Every state change to an alert is recorded, whether a human or the engine caused it. This is what you read in a post-incident review when the question is "when did we know, and who acknowledged it?"

# most recent 50 entries, newest first
curl https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/audit-trail \
  -H 'X-API-Key: YOUR_API_KEY'

# only escalations, paged
curl 'https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/audit-trail?action=escalated&limit=200&offset=0' \
  -H 'X-API-Key: YOUR_API_KEY'

# everything that happened to alerts (as opposed to channels)
curl 'https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/audit-trail?target_type=alert' \
  -H 'X-API-Key: YOUR_API_KEY'

The response is { entries: [...], count: n }, newest first. Each entry carries the action, the target type and id, the actor (user id, email and name where a person did it — absent when the engine did), a JSON data blob with the specifics of that action, and a timestamp.

Actions recorded on alerts include created, acknowledged, unacknowledged, ack_expired, assigned, commented, escalated, escalation_triggered, repaged, snoozed, unsnoozed, snooze_expired, resolved, and the bulk variants bulk_resolved and bulk_snoozed. Beyond alerts you will also see auto_disabled against a channel — that is Epok switching off a notification destination that kept failing, which is worth knowing when someone reports that Slack went quiet.

Both filters are exact matches, not searches. target_type takes values like alert or channel; action takes one of the values above. limit accepts 1–500 and defaults to 50; offsetpages. Entries are scoped to your account — there is no way to read another tenant's trail.

Boundary:like maintenance windows, the audit trail is API-only — the per-alert timeline in the app shows an alert's own history, but the account-wide trail has no screen. Pull it over HTTP for reviews and compliance evidence.

Custom actions

A custom action is a saved outbound HTTP request that an operator can fire from a specific alert — open a ticket, page a different team, trigger a runbook, roll back a flag. It is not a notification channel: nothing fires automatically. Someone decides, on this alert, to run it.

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/custom-actions \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Open a Jira ticket",
    "description": "File the alert into the on-call board",
    "url": "https://automation.example.com/hooks/jira",
    "http_method": "POST",
    "headers": { "Authorization": "Bearer THEIR_TOKEN" },
    "body_template": "{\"summary\": \"{{title}}\", \"service\": \"{{service}}\", \"severity\": \"{{severity}}\", \"epok_alert\": \"{{alert_id}}\"}"
  }'

name and url are required. http_method defaults to POST. headers is a JSON object merged into the request — Content-Type: application/json is added for you unless you set it yourself. body_template is sent as the raw body.

Placeholders in the body are substituted from the alert at execution time. The available ones are exactly: {{alert_id}}, {{service}}, {{severity}}, {{title}}, {{description}}, {{stream}}, {{detector_type}} and {{fired_at}}. Substitution is plain text replacement, so a quote or newline in an alert title will break a hand-written JSON body. If the receiving system is strict, put the placeholder in a field where that cannot happen, or accept form-encoded input instead. An unknown placeholder is not an error — it is simply left in the body verbatim, which is the failure mode to look for when the far end complains about a literal {{ in its payload.

curl -X POST \
  https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/alerts/48376/execute-action/3 \
  -H 'X-API-Key: YOUR_API_KEY'
# -> {"status":"executed","status_code":201,"success":true}

Execution is synchronous with a 10 second timeoutand redirects are not followed. The response reports the far end's status code and a success boolean that is true below 400 — so a 4xx from your endpoint comes back as "success": false rather than an Epok error. A transport failure or timeout returns 502. Unknown action id or alert id returns 404.

URLs are SSRF-validated on every execution, not just on create. The target must be http or https, and hostnames resolving to loopback, private or link-local addresses, or to cloud metadata endpoints, are refused with 400. An action pointing at something inside your own network will not work from Epok — put a reachable automation endpoint in front of it.

Two things to know about the shape of this API. The list response includes an enabled flag, but there is no endpoint to toggle it and execution does not consult it — to retire an action, delete it. And there is no update endpoint: editing means delete and re-create, which changes the id.

Creating and deleting actions requires owner or admin, or an API key with write scope; members can see them but not change them. Actions appear on the alert detail view in the app, where the operator runs them.

What you do not need to set up

None of this is required to get paged correctly. Deduplication, grouping, escalation and auto-resolve are on by default with the values above — there is nothing to configure to stop duplicate pages, and no suppression policy to write before the first alert is useful.

You also do not need a mute schedule for a detector that is simply wrong about your system. Muting hides the symptom and keeps the detector wrong. Feedback on an alert tunes the detection instead — see detection tuning. Reach for suppression when the alert is correct and you already know.