Custom rules

Read this before you pick one

The names invite you to assume a composite rule is a threshold rule with extra powers. It is not. A composite rule is missing two things a threshold rule has, and both omissions bite in production:

• No flap guard. Composite rules have no for_duration_seconds. The expression is re-evaluated every 60 seconds and the rule fires the first cycle it comes out true — a sixty-second blip pages you. Your only damper is cooldown_seconds, which limits how often it re-fires but cannot stop the first one. If you need “true for five minutes, then tell me”, that is a threshold rule.
• No metrics. A threshold leaf inside a composite expression counts matching log events. It cannot evaluate a metric expression. Metric monitors exist only as threshold rules with rule_type: "metric".

What a composite rule gives you in exchange is the only thing a threshold rule cannot express: correlation between independent signals — AND / OR / NOT across log counts, currently-firing alerts, and detector health.

How rules interact with the built-in detectors

A custom rule is purely additive. It does not replace, tune, or silence any built-in detector. Writing a rule for “error rate above 2%” does not stop the error-rate detector from also alerting on the same incident — you get both, from two sources, and they de-duplicate only insofar as the alert manager groups them. If a detector is too noisy, the fix is suppression or a mute schedule, not a competing rule.

The productive pattern is the opposite: use a rule to express something the detectors cannot know — a business threshold, a contractual limit, a value only your domain gives meaning to. Refunds over a currency amount. A queue depth your ops runbook cares about. A specific log line that means a manual process is stuck.

Threshold rules

A threshold rule takes one measurement every 60 seconds and compares it to a number. For a log rule the measurement is the count of events matching your search over window_seconds. For a metric rule it is the value your metric expression evaluates to. If the comparison holds for for_duration_seconds and the rule is outside its cooldown, it fires an alert titled Threshold rule ‘name’ triggered (or Metric rule …).

Create one

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/rules \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Auth-failure burst",
    "rule_type": "log",
    "query": "level:warn AND _msg:\"auth failed\"",
    "condition_op": "gt",
    "condition_value": 50,
    "window_seconds": 60,
    "for_duration_seconds": 120,
    "cooldown_seconds": 900,
    "severity": "critical"
  }'

ACCOUNT_IDis your numeric account id. Your credential is bound to one account — a mismatched id in the path returns 403, it does not let you address someone else's data. The key must have write scope (or you must be signed in as owner/admin).

Every field

There is no enabled field on create. Rules are created enabled. Sending enabled in the create body is silently ignored — a real trap if you POST the same JSON you exported. To disable a rule without deleting it, PATCH it (below).

A rule that keeps failing turns itself off. If evaluation raises five times in a row — the query stops being valid, or it times out repeatedly — Epok sets enabled = false on the rule rather than retrying forever. It stops firing, and because the plan limit counts only enabled rules it also stops consuming a slot. Nothing notifies you, so if a rule has gone quiet, check its enabled flag with GET .../rules before assuming the condition never held.

A metric monitor

Same shape, different rule_type. The query is a Prometheus-compatible expression evaluated against the metrics store; the first finite series it returns is the measured value. Metric monitors require the metrics data plane — without it, create returns 400 and an existing rule is skipped rather than firing on absent data.

{
  "name": "Checkout p99 over budget",
  "rule_type": "metric",
  "query": "histogram_quantile(0.99, sum by (le) (rate(http_server_duration_milliseconds_bucket{service=\"checkout\"}[5m])))",
  "condition_op": "gt",
  "condition_value": 800,
  "window_seconds": 300,
  "for_duration_seconds": 300,
  "severity": "warning"
}

Worked examples

Manage them

Because rules are JSON over a plain REST surface, keeping them in your repo and applying them from CI works: GET to export, POST to apply. Alert config then ships in the same commit as the service change that needs it.

There is no dry-run endpoint. Backtesting a threshold rule is a UI feature: in the app, open Alerting → Threshold rules and use the dry-run control in the rule form. It replays the last hour of history and tells you how many times the rule would have fired. Metric monitors are not backtestable — they evaluate live values, so save it and watch the next cycle.

Composite rules

A composite rule is a boolean expression tree. Every 60 seconds Epok walks the tree, evaluates each leaf, combines them with the branch operators, and fires one alert — Composite rule triggered: name— if the root comes out true. The evidence attached to the alert includes each leaf's result, so you can see which condition was the one that flipped.

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/composite-rules \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Checkout 5xx while the DB detector is already firing",
    "expression": {
      "op": "and",
      "conditions": [
        {
          "op": "threshold",
          "query": "service:checkout AND status_code:>=500",
          "comparator": "gt",
          "value": 10,
          "window_seconds": 300
        },
        {
          "op": "alert_firing",
          "detector_type": "database_intelligence"
        }
      ]
    },
    "severity": "critical",
    "cooldown_seconds": 900,
    "enabled": true
  }'

Top-level fields

Leaf operators — the three things you can measure

There are exactly three. Anything else is rejected with 400 Invalid operator.

Branch operators — the three ways to combine

• and — true when every child is true. Takes a non-empty conditions array.
• or — true when any child is true. Takes a non-empty conditions array.
• not — inverts its single child. Takes exactly one entry in conditions; two or zero returns 400.

Two structural limits, both enforced at create/update/test time: nesting is capped at 5 levels and the serialised expression at 10 KB. If you are approaching either, you are writing a program, not an alert.

A worked not — page on worker errors only when the queue is not already alerting, so the queue incident does not generate two pages:

{
  "op": "and",
  "conditions": [
    {
      "op": "threshold",
      "query": "service:worker AND level:error",
      "comparator": "gt",
      "value": 25,
      "window_seconds": 300
    },
    {
      "op": "not",
      "conditions": [
        { "op": "alert_firing", "service": "queue" }
      ]
    }
  ]
}

Test an expression before you save it

Composite rules — unlike threshold rules — have a real dry-run endpoint. It evaluates the expression right now, returns whether it would fire and which leaves matched, and persists nothing. No alert is emitted and no cooldown is consumed.

curl -X POST https://app.getepok.dev/api/v1/tenants/ACCOUNT_ID/composite-rules/test \
  -H 'X-API-Key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "expression": {
      "op": "and",
      "conditions": [
        { "op": "threshold", "query": "level:error", "value": 50, "window_seconds": 300 },
        { "op": "alert_firing", "detector_type": "error_rate" }
      ]
    }
  }'

# → {"met": false,
#    "op": "and",
#    "details": [
#      {"met": true,  "op": "threshold", "query": "level:error",
#       "actual": 91, "threshold": 50, "comparator": "gt"},
#      {"met": false, "op": "alert_firing",
#       "detector_type": "error_rate", "service": null, "firing_count": 0}
#    ]}

Manage them

The same rules are editable in the app under Alerting → Composite rules, which wraps the test endpoint in a builder if you would rather not hand-write the tree.

Limits

These are the limits configured per plan. What the create endpoints enforce today is narrower on some plans — the two bullets under the table are the behaviour you will actually hit, and they win.

Three details that surprise people:

• The threshold-rule limit counts only enabled rules. Disabling a rule frees a slot; you can keep an archive of disabled rules at no cost. The composite-rule limit counts every stored rule, enabled or not — there you have to delete.
• Creating a threshold rule over the API currently admits Team accounts only. The gate is an exact match on the Team plan, so Free — but also Trial, Growth and Enterprise — get 403 This feature requires a paid plan (Team) on the first rule, whatever the table above says. If you are on a plan that should qualify and see that 403, it is us, not your request; contact support and we will lift it. Separately, being genuinely over the limit returns 403 for threshold rules and 429 for composite rules.
• An uncapped plan is not yet uncapped for composite rules. Plans configured with no composite cap are compared against the raw limit without special-casing it, so the check fails on the very first rule and returns 429 Composite rule limit reached. Until that is fixed, treat Free / Trial / Team's numbers as the real ones and talk to support if you need more.

“My rule fires too much”

Work through these in order, because each one is cheaper than the next:

1. Raise for_duration_seconds on a threshold rule. Most “noisy rule” reports are a rule with the default 0, firing on single-cycle blips.
2. Raise cooldown_seconds. This does not stop the first page, but it stops the next nine. Composite rules already default to 300; threshold rules default to 0.
3. Narrow the query. For a log rule, run the query in Explore over the last 24 hours first — the hit count Explore shows you over your window is the same measurement the rule compares against.
4. Suppress at the alert layer. Mute schedules, maintenance windows and suppression rules live in alerting and apply to detector output and custom rules alike. Use these for known-noisy periods rather than deleting a rule you will want back.

One anti-pattern worth naming: do not write rules against high-volume level:info patterns. That is what the detectors are for, and they learn a baseline instead of comparing to a number you guessed.