Tee your Prometheus to Epok

1. Get an ingest key

In Epok, go to Settings → API Keysand create a restricted, ingest-only key. Keep it handy — you paste it literally into the config below. Prometheus does not expand environment variables in prometheus.yml: writing ${EPOK_API_KEY} ships that exact string as the bearer token and every write fails with 403— a well-formed credential that matches no key.

2. Add one block to prometheus.yml

This is additive— your existing scrape configs and any current remote_write targets stay untouched.

remote_write:
  - url: https://ingest.getepok.dev/api/v1/write
    authorization:
      type: Bearer
      credentials: YOUR_API_KEY     # paste your key literally — Prometheus does not expand env vars
    # optional — keep the send lean; trim high-noise internals
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*|process_.*"
        action: drop
    queue_config:
      max_samples_per_send: 2000
      capacity: 10000

3. Reload Prometheus

kill -HUP $(pgrep prometheus)
# or, if --web.enable-lifecycle is set:
curl -X POST http://localhost:9090/-/reload

That's it. Every series Prometheus already collects now reaches Epok. Nothing about your existing pipeline changed.

Verify the write landed

A successful write returns 204 with no body, so success is silent. Failure is not: Prometheus logs a Failed to send batchline carrying the status code. Read that code before you touch anything else — it separates problems that look identical from your side.

• 403— a key arrived and was rejected. Nine times out of ten this is the env-var trap above: you wrote ${EPOK_API_KEY}, Prometheus shipped that literal string, and it is a perfectly well-formed credential that matches no key. The other causes are a rotated key and a key without the ingest scope.
• 401— no credential was found at all. This is a shape problem, not a wrong-key problem: the authorizationblock is missing, empty, or nested at the wrong level, so nothing was sent to reject. Note which of the two you get — 401 means fix the config, 403 means fix the key.
• 413 — the batch is over the 8 MB compressed cap (or 500k samples). Lower max_samples_per_send. Prometheus defaults sit well under both.
• 429— you are over your daily ingest volume or the per-tenant rate limit, not misconfigured. See limits.
• 503— metrics ingest is not switched on for your account. This is a deliberate gate, not a fault, and no config change on your side clears it — ask us.

Good to know

• Fully additive. remote_write fans out — keep writing to your existing long-term store and Epok. Nothing to rip out.
• Flat billing.You're metered on ingested volume, flat — no per-series or per-host fee. Use write_relabel_configs to drop high-cardinality internals if you want.
• 8 MB compressedper request (snappy), 500k samples per request; Prometheus defaults sit comfortably under both. Full numbers — daily volume, rate limits, retention — are on limits.
• Kubernetes — the same block goes under prometheus.prometheusSpec.remoteWrite:

# kube-prometheus-stack values.yaml — same block, nested
prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://ingest.getepok.dev/api/v1/write
        authorization:
          type: Bearer
          credentials: YOUR_API_KEY   # paste literally — Prometheus does not expand env vars

Your first look

Give it one to two minutes, then open Infrastructure → Metrics. That view opens filtered to infrastructure-origin series, which is where a fleet tee lands. Use the filter box at the top of the metric list and type node_— if node_exporter is in your scrape config, its series are there. kube_ and container_ behave the same way.

Then do the one thing a dashboard makes you build by hand: pick a series, and use Break down by to split it across instance or service. Every label Prometheus already carries is a chip — the outlier host separates from the pack without you writing a query.

One expectation to set honestly. A bare tee gets you exploration and the anomaly detectors, which read any shape of series. The utilization-shaped detectors — saturation, capacity forecast, the ones that say “root fs on web-01, full in ~6h”— need a signal they can read as a 0–1 fraction. Raw node_exporter emits bytes and seconds, so a series like node_memory_MemAvailable_bytes is not eligible for them. Two recording rules fix that, and they are the first thing worth adding after this page.

Metric detectors also hold fire for the first ~3 hoursafter a tenant's first metric arrives. That is a deliberate learning gate: a baseline built from three minutes of startup noise produces false pages, so nothing alerts until the signal has existed long enough to have a shape.

Next

• The node_exporter recording-rule bridge — the two rules that turn raw bytes into the utilization series saturation and capacity forecasting read. Do this one next.
• Instrument your app for tracing — metrics tell you a host is sick; spans tell you which request path. Correlation across both is where the root-cause verdict comes from.
• Browser RUM & session replay — close the loop from a saturated node to the user who felt it.
• What watches these series — every detector, what it fires on, and what it needs to be eligible.
• Limits — payload caps, daily volume, rate limits, retention.