# Epok — Agent Operations Guide

This guide is written for AI/coding agents setting up or operating Epok on
behalf of a user. It is self-contained: authentication, ingest for every
signal, the query API, search syntax, rate limits, and verification steps.
Schemas: https://getepok.dev/openapi.json — if that URL is unreachable
from your environment (some networks see a bot challenge on JSON paths),
proceed with this guide alone; every endpoint you need is documented here
with its parameters.

Epok is an incident-intelligence engine that works an incident in all three
tenses: it forecasts forming failures where they're genuinely predictable
(saturation, exhaustion, capacity), recommends the reversible action that
restores service (you execute), and drafts the cited root-cause verdict. It
grades its own confidence against real outcomes and abstains, with next steps,
when the evidence is thin rather than guessing. You send telemetry (logs,
metrics, traces, infrastructure, RUM); detectors run automatically — anomaly
detection, new-error detection, incident grouping, and root-cause analysis.
There is NOTHING to
configure to get detection: no alert rules required, no thresholds, no
dashboards-before-value. Custom threshold rules and notification channels
are optional additions, not prerequisites.

## 1. Authentication

- The user signs up at https://app.getepok.dev (Google sign-in, trial-first).
- API keys are created in the app: Settings → API Keys. Keys are
  tenant-scoped and carry scopes: `ingest` (send data), `read` (query),
  `write` (mutations). Ask the user for a key with the scopes you need —
  for setup you typically want `ingest` + `read`.
- Ask the user to create the key as an AGENT key (the "agent key" option,
  or `is_agent: true` via the key API). Agent keys get their own query
  budget, so your activity can never starve the team's UI — and your
  usage is attributable, which keeps everyone comfortable giving you
  access. Don't request `write` scope unless the task genuinely needs it.
- Every request: `Authorization: Bearer <key>` header.
- The numeric `tenant_id` appears in the app and is required in query API
  paths (`/api/v1/tenants/{tenant_id}/...`). Ask the user for it alongside
  the key.
- Never commit keys to the repository. Put them in environment variables or
  the user's secret manager. If you printed or pasted a key anywhere
  public, tell the user to rotate it in Settings → API Keys.

## 2. Ingest — base URL https://ingest.getepok.dev

All ingest endpoints authenticate with the same Bearer key. Gzip request
bodies are accepted (`Content-Encoding: gzip`).

### OpenTelemetry (recommended — one pipeline, all signals)

| Signal  | Endpoint      | Encodings                  |
|---------|---------------|----------------------------|
| Logs    | POST /v1/logs    | OTLP JSON or protobuf   |
| Traces  | POST /v1/traces  | OTLP JSON or protobuf   |
| Metrics | POST /v1/metrics | OTLP JSON (recommended) or protobuf |

There is no gRPC endpoint. Use `http/protobuf` or `http/json` — never
`:4317`, and never the `otlp` gRPC exporter.

On metrics encoding: `encoding: json` on the collector's `otlphttp`
exporter is RECOMMENDED, not required. JSON is accepted by every build. A
stock collector defaults to protobuf, which newer builds also accept — but
if you cannot confirm which build the user is on, set `encoding: json` and
the request is accepted either way. A 400 whose body mentions OTLP JSON
means you hit a build that did not decode the protobuf; setting
`encoding: json` (or `OTEL_EXPORTER_OTLP_PROTOCOL=http/json`) resolves it.

Metrics also accept two NON-OTel protocols (same Bearer key, same
cardinality handling — pick whichever the user already runs):

- **Prometheus remote_write**: `POST /api/v1/write` — point an existing
  Prometheus / vmagent / Grafana Alloy at it; no collector needed.
- **InfluxDB line protocol**: `POST /write` or `/api/v2/write` —
  Telegraf and Influx clients work as-is.

If the user already runs Prometheus, remote_write is the fastest path
to metrics in Epok — don't make them deploy an OTel Collector for
metrics alone.

### Browser RUM + session replay

Real-user monitoring rides the SAME OTLP endpoints: browser web-vitals
are OTLP metrics and browser errors/spans are OTLP traces — point the
OTel Web SDK (or `initRum`) at `/v1/metrics` and `/v1/traces`. There is
no separate "RUM" ingest protocol. Session replay (rrweb) IS separate:
the SDK posts gzipped event chunks to `POST /v1/rum/replay` (ingest
key); query sessions back at `GET /api/v1/tenants/{tenant_id}/rum/sessions`
and a single session at `.../rum/sessions/{session_id}`. Replay requires
a plan with session replay enabled (otherwise the endpoint returns 503).

GOTCHA WORTH THE ONE LINE: an OTel Collector's `otlphttp` exporter defaults
to protobuf. Newer builds of the metrics endpoint decode protobuf, older
ones answer 400 while logs and traces keep working — a confusing partial
failure. Setting `encoding: json` is accepted by every build, so set it and
stop thinking about which build you are talking to.

Known-good OTel Collector exporter (works for all three signals):

```yaml
exporters:
  otlphttp/epok:
    endpoint: https://ingest.getepok.dev
    encoding: json          # accepted by every build; fine for all signals
    compression: gzip
    headers:
      Authorization: "Bearer ${EPOK_API_KEY}"

service:
  pipelines:
    logs:
      exporters: [otlphttp/epok]
    traces:
      exporters: [otlphttp/epok]
    metrics:
      exporters: [otlphttp/epok]
```

(Add `otlphttp/epok` to the exporter LIST of each existing pipeline — in
collector config, lists replace wholesale, so restate the full list.)

⚠️ Spanmetrics exception (APM RED): if the collector runs the `spanmetrics`
connector, export THAT pipeline via `prometheusremotewrite` to
`https://ingest.getepok.dev/api/v1/write` — NOT `otlphttp`. The OTLP metrics
importer does not apply Prometheus name suffixes (`_total`,
`_milliseconds_bucket`), so spanmetrics sent over OTLP never matches the
series the service page reads (silent failure). Note raw spans alone already
populate the service page (RED is derived server-side from stored spans);
spanmetrics is the optional pre-sampling accuracy upgrade for tail-sampled
fleets.

### Other log shippers

- `POST /insert/elasticsearch/_bulk` — Elasticsearch bulk NDJSON. Fields:
  `_msg` (message), `_time` (ISO8601), plus any structured fields
  (`level`, `service`, ...). Fastest way to test from curl.
- `POST /loki/api/v1/push` — Loki push protocol (FluentBit, Vector,
  Promtail, Alloy). Basic auth: user = API key, password = `x`.
- `POST /api/v1/syslog` — syslog lines over HTTP.
- `POST /services/collector/event` (+ `/raw`) — Splunk HEC wire protocol,
  natively. Auth with Bearer / `x-api-key` (the `Authorization: Splunk
  <token>` header form is not parsed).
- `POST /gelf` — GELF over HTTP (app-level emitters; `?api_key=` accepted).
  Docker's built-in gelf driver is UDP/TCP-only and cannot reach this —
  use FluentBit/Vector for container logs.

Timestamp rule: `_time` (and equivalents) are honored AS-IS. An event dated
in the past is stored at that historical point — ingest returns accepted,
but the event won't appear in Live Tail or recent-window queries. Omit the
timestamp to have Epok stamp arrival time; if "sent but not visible",
check the timestamp before suspecting the pipe.

Quick ingest test:

```bash
curl -X POST https://ingest.getepok.dev/insert/elasticsearch/_bulk \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '
{"create":{}}
{"_msg":"agent setup test","level":"info","service":"agent-setup"}
'
```

Tag telemetry with a `service` field/resource attribute — most Epok
intelligence is per-service.

## 3. Query API — base URL https://app.getepok.dev

All read endpoints need a `read`-scope key and the numeric tenant id in
the path. Full schemas in /openapi.json.

- `GET /api/v1/tenants/{id}/search/unified?q=<query>&start=-1h&end=now&limit=5`
  — ONE query across every signal: logs, spans, RUM events, metric events,
  metric names, and replay sessions. `limit` is per-section (1–20). Start
  here when investigating; it tells you which signal has the evidence.
- `POST /api/v1/tenants/{id}/search` — log search with the full search
  syntax (body schema in the OpenAPI spec). Result caps apply.
- `GET /api/v1/tenants/{id}/hits?query=...&start=...&end=...&step=1m` —
  time-bucketed counts (the histogram primitive).
- `GET /api/v1/tenants/{id}/facets?query=...` — top values per field.
- `GET /api/v1/tenants/{id}/streams` — known log streams/services.
- `GET /api/v1/tenants/{id}/patterns` — clustered log patterns.
- `GET /api/v1/tenants/{id}/alerts` — alerts (active + recent).
  `GET .../alerts/{alert_id}` for detail with full evidence;
  `POST .../alerts/{alert_id}/resolve` to resolve.
- WebSockets: `/ws/livetail/{tenant_id}` (live log stream),
  `/ws/alerts/{tenant_id}` (alert push).

Time parameters accept relative (`-5m`, `-1h`, `-7d`) and `now`, or
ISO8601 timestamps.

## 4. Search syntax (essentials)

- Bare words full-text match: `timeout`
- Phrases: `"connection refused"`
- Fields: `level:error`, `service:checkout`, `status_code:500`
- Boolean: `level:error AND service:api`, `OR`, negation with `-`:
  `level:error -service:noisy-cron`
- Regex: `~"pool exhausted .*retry"` (tilde-quoted — NOT `re(...)`)
- Time filter inside a query: `_time:5m`
- Aggregations are piped: `level:error | stats count()`
- Top values: `| top 10 (service)` — note the parentheses and NO `by`
  keyword: `top 10 (field)`, not `top 10 by field`.

## 5. Rate limits and caps

- Query rate limits are per-tenant and per-plan. On breach you get HTTP
  429 with a `Retry-After` header — honor it.
- Export-style reads are capped (10,000 rows per request). Page with time
  windows rather than retrying larger limits.
- Query time ranges are clamped to the plan's retention window.
- Daily ingest volume is metered per plan; over-limit ingest returns 429.
- Be a polite agent: prefer `search/unified` + narrow follow-ups over
  broad export sweeps; use `hits` (counts) before pulling raw rows.

## 6. Source maps (symbolicated frontend stack traces)

If the user's frontend ships minified JS, upload source maps at build
time so Epok resolves error stacks to original files/lines. One call per
bundle per release (ingest-scope key; gzip body accepted):

```bash
curl -X POST https://app.getepok.dev/api/v1/tenants/TENANT_ID/sourcemaps \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d "{
    \"release\": \"$(git rev-parse --short HEAD)\",
    \"url\": \"https://cdn.example.com/assets/app.min.js\",
    \"map\": $(cat dist/app.min.js.map)
  }"
```

- `url` must be the EXACT URL the browser loads the bundle from.
- Set the SAME release in the RUM SDK init (`release` option) so frames
  match: `initRum({ ..., release: "<git sha>" })`.
- Resolve programmatically: `POST .../sourcemaps/resolve` with
  `{"release": str, "frames": [{"url", "line" (1-based), "column" (0-based)}]}`
  → frames enriched with `original_source`/`original_line`/`original_name`.
- Frames without an uploaded map pass through untouched (vendor bundles
  routinely ship no maps). List uploads: `GET .../sourcemaps`.
- Wire this into the user's CI right after their bundler step — agents
  should add it to the build script, keyed off the same release id.

## 7. Configuration endpoints

These exist and are usable from the API. An earlier revision of this file
said they were app-UI-only; that was wrong, and agents acting on it sent
users to click through the app for things they could have scripted.

| Thing | Endpoints | Guide |
|-------|-----------|-------|
| Threshold rules | `GET/POST /api/v1/tenants/{id}/rules`, `PATCH`/`DELETE .../rules/{rule_id}` | /docs/custom-rules |
| Composite rules | `GET/POST /api/v1/tenants/{id}/composite-rules`, `GET`/`DELETE .../{rule_id}` | /docs/custom-rules |
| Notification channels | `GET/POST /api/v1/tenants/{id}/channels`, `DELETE`/`POST .../{channel_id}/test` | /docs/alerting |
| Mute schedules | `GET/POST /api/v1/tenants/{id}/mute-schedules`, `DELETE .../{schedule_id}` | /docs/alerting |
| Maintenance windows | `GET/POST /api/v1/tenants/{id}/maintenance-windows`, `DELETE .../{window_id}` | /docs/alerting |
| Level rules (severity mapping) | `GET/POST /api/v1/tenants/{id}/level-rules`, `POST .../level-rules/test`, `DELETE .../{rule_id}` | /docs/detection-tuning |
| Dashboards | `GET/POST /api/v1/tenants/{id}/dashboards`, `DELETE .../{dashboard_id}` | — |

Caveats worth knowing before you script against these:

- Mutating endpoints need a key with `write` scope; most also require the
  settings-write permission, so a `read`-only key gets 403, not 401.
- Threshold-rule creation is gated to paid plans and currently returns 403
  on plans that the published limits table suggests should allow it. If you
  get a 403 creating a rule, that is a plan gate, not a bad key — check
  /docs/limits before retrying.
- Not everything has an API: the audit trail and maintenance windows have no
  app UI, and conversely some app surfaces have no endpoint. Team management
  remains app-only.
- These endpoints are absent from the published openapi.json. This file and
  the guides linked above are the reference for them.

## 8. Recommended agent setup workflow

1. Ask the user for: API key (ingest+read scopes) and tenant id. Remind
   them keys live in env vars, never in committed code.
2. Verify reachability: `GET https://app.getepok.dev/health` → `{"status":"ok"}`.
3. Wire ingest (Section 2). For an OTel Collector, remember
   `encoding: json`.
4. Verify data arrival (allow ~30–60s):
   `GET /api/v1/tenants/{id}/search/unified?q=service:<your-service>&start=-10m`
   — confirm non-zero counts in the right signal sections.
5. Confirm detection is live: `GET /api/v1/tenants/{id}/alerts` (an empty
   list is fine — it means no anomalies; detectors are on by default).
6. Tell the user what you shipped: which signals are flowing, from which
   services, and where to look (https://app.getepok.dev).

## 9. Human-facing docs — route map

This file is self-contained; you should not need to fetch these. Use them
when the USER asks where something is documented, or when you need a detail
this guide compresses away. Every path below is a real page.

Start here:
- /docs — hub: supported protocols, shipper picker, links to everything
- /docs/quickstart — one curl, verify, first search; the timestamp trap
- /docs/authentication — per-plane credential matrix, 401 vs 403 triage
- /docs/limits — body caps, rate/burst, daily quota, status codes

Install (one page per platform, each ending in a verification step):
- /docs/install/traces, /docs/install/metrics, /docs/install/prometheus
- /docs/install/kubernetes, /docs/install/aws, /docs/install/gcp
- /docs/install/vercel, /docs/install/railway, /docs/install/drains
- /docs/install/rum, /docs/install/sourcemaps

Reference and configuration:
- /docs/search-syntax — filters, booleans, wildcards, regex, pipes
- /docs/detectors — catalog; per-detector pages at /docs/detectors/{id}
- /docs/custom-rules — threshold and composite rules
- /docs/detection-tuning — level rules, field mappings, resets
- /docs/alerting — mute, maintenance, digest, audit trail, custom actions
- /docs/api — interactive OpenAPI reference
- /docs/migrate, /docs/changelog, /docs/what-we-dont-do

Machine-readable (stable URLs — safe to hard-code):
- /llms.txt, /llms-full.txt (this file), /openapi.json, /docs/agents

## 10. Support

- Docs: https://getepok.dev/docs
- Email: support@getepok.dev