Send Railway logs to Epok

Path 1: Direct HTTP push from app code

Simplest path. Set EPOK_API_KEYas a Railway environment variable, then have your app POST log batches to Epok's Elasticsearch Bulk endpoint. The examples below enrich each entry with Railway's built-in environment variables so logs are pre-tagged with service / deploy / region.

Required Railway environment variables

Set EPOK_API_KEY in the Railway service's Variables tab. Railway already injects the identity vars below into every service at runtime — use them to tag your logs:

EPOK_API_KEY            = epk_REPLACE_ME    # you set this
RAILWAY_SERVICE_NAME    = (auto)            # e.g. "api-gateway"
RAILWAY_ENVIRONMENT_NAME= (auto)            # e.g. "production"
RAILWAY_DEPLOYMENT_ID   = (auto)            # short id of current deploy
RAILWAY_REPLICA_ID      = (auto)            # short id of this replica

Node.js example (no dependencies — fetch is built-in)

// epok.js — drop-in logger that batches + ships to Epok.
const ENDPOINT = 'https://ingest.getepok.dev/insert/elasticsearch/_bulk';
const API_KEY = process.env.EPOK_API_KEY;

const queue = [];
let timer = null;

function flush() {
  if (queue.length === 0) return;
  const batch = queue.splice(0, queue.length);
  const body = batch
    .flatMap((e) => [
      JSON.stringify({ create: {} }),
      JSON.stringify({
        _msg: e.msg,
        _time: e.time,
        level: e.level || 'info',
        service: process.env.RAILWAY_SERVICE_NAME || 'unknown',
        env: process.env.RAILWAY_ENVIRONMENT_NAME || 'production',
        deploy: process.env.RAILWAY_DEPLOYMENT_ID,
        replica: process.env.RAILWAY_REPLICA_ID,
        ...e.fields,
      }),
    ])
    .join('\n') + '\n';

  fetch(ENDPOINT, {
    method: 'POST',
    headers: { Authorization: `Bearer ${API_KEY}`, 'Content-Type': 'application/json' },
    body,
  }).catch(() => {}); // never let logging break the request path
}

// Flush every 2s OR once 100 entries are queued — whichever is first.
function schedule() {
  if (timer) return;
  timer = setTimeout(() => { timer = null; flush(); }, 2000);
}

export function log(level, msg, fields = {}) {
  queue.push({ time: new Date().toISOString(), level, msg, fields });
  if (queue.length >= 100) flush();
  else schedule();
}

// Flush on process exit so the last batch isn't lost.
process.on('SIGTERM', flush);
process.on('beforeExit', flush);

// Usage:
// import { log } from './epok.js';
// log('info', 'User signup completed', { user_id: 4821 });
// log('error', 'Payment gateway timeout', { gateway: 'stripe', latency_ms: 5021 });

Python example (stdlib logging handler)

# epok_logger.py — attach to Python's root logger; works with stdlib,
# loguru, or any structured-logging library that forwards to logging.
import json, logging, os, queue, threading, time, urllib.request

ENDPOINT = "https://ingest.getepok.dev/insert/elasticsearch/_bulk"
API_KEY = os.environ["EPOK_API_KEY"]
SERVICE = os.environ.get("RAILWAY_SERVICE_NAME", "unknown")
ENV = os.environ.get("RAILWAY_ENVIRONMENT_NAME", "production")
DEPLOY = os.environ.get("RAILWAY_DEPLOYMENT_ID", "")
REPLICA = os.environ.get("RAILWAY_REPLICA_ID", "")


class EpokHandler(logging.Handler):
    def __init__(self):
        super().__init__()
        self.q: queue.Queue = queue.Queue(maxsize=10_000)
        threading.Thread(target=self._worker, daemon=True).start()

    def emit(self, record: logging.LogRecord) -> None:
        try:
            self.q.put_nowait({
                "_msg": record.getMessage(),
                "_time": record.created * 1000,  # ms epoch
                "level": record.levelname.lower(),
                "service": SERVICE, "env": ENV,
                "deploy": DEPLOY, "replica": REPLICA,
                "logger": record.name,
            })
        except queue.Full:
            pass  # drop newest if backed up — never block the app

    def _worker(self) -> None:
        while True:
            batch = []
            try:
                batch.append(self.q.get(timeout=2))
                while len(batch) < 100:
                    batch.append(self.q.get_nowait())
            except queue.Empty:
                pass
            if not batch:
                continue
            body_lines = []
            for entry in batch:
                body_lines.append('{"create":{}}')
                body_lines.append(json.dumps(entry))
            body = ("\n".join(body_lines) + "\n").encode("utf-8")
            try:
                req = urllib.request.Request(
                    ENDPOINT,
                    data=body,
                    headers={
                        "Authorization": f"Bearer {API_KEY}",
                        "Content-Type": "application/json",
                    },
                )
                urllib.request.urlopen(req, timeout=10).read()
            except Exception:
                pass  # never raise from a log handler


# Wire into root logger
logging.getLogger().addHandler(EpokHandler())
logging.getLogger().setLevel(logging.INFO)

Path 2: OpenTelemetry SDK (vendor-portable)

Use OpenTelemetry's OTLP HTTP exporter pointed at Epok's OTLP endpoint. Best if you might switch observability vendors later — instrument once, retarget the OTLP endpoint env var, no code changes.

One caveat: OpenTelemetry SDKs don't flush on shutdown by default. Call shutdown() on the logger provider from a SIGTERM handler in your app, otherwise the last batch drops on container restart.

Service environment variables

# Railway service → Variables tab
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT = https://ingest.getepok.dev/v1/logs
OTEL_EXPORTER_OTLP_LOGS_HEADERS  = Authorization=Bearer epk_REPLACE_ME
OTEL_SERVICE_NAME                = ${{RAILWAY_SERVICE_NAME}}
OTEL_RESOURCE_ATTRIBUTES         = deployment.environment=${{RAILWAY_ENVIRONMENT_NAME}},service.version=${{RAILWAY_DEPLOYMENT_ID}}

The Railway-injected variables (RAILWAY_SERVICE_NAMEet al.) get interpolated into the OTEL vars via Railway's reference syntax, so every log carries the correct service/env/deploy tags without app code knowing about Railway specifically.

Path 3 (advanced): Vector as a separate Railway service

If you can't modify your app at all, deploy Vector as its own Railway service that pulls logs via Railway's GraphQL API and ships them to Epok. This is more setup than Path 1 or 2 and only worth it if app-side instrumentation is off the table.

See the AWS install guide for the Vector config shape (sources + transforms + Epok elasticsearch sink). Replace the file/journal sources with a http_clientsource polling Railway's GraphQL API. The cleaner long-term play is usually Path 2 (OpenTelemetry).

Verify

1. Trigger a request to your Railway service.
2. Open app.getepok.dev → Live Tail. Within ~10 s you should see the log line with service populated from RAILWAY_SERVICE_NAME.
3. Open Services— every Railway service that you've instrumented appears with its own hit + error rate.

Common gotchas

• No sidecars.Railway doesn't run a second process inside your service container. If your stack traditionally uses a Fluent Bit / Vector sidecar (Kubernetes pattern), you'll need to fold that into the app process itself or deploy it as a separate Railway service (Path 3).
• Flush on shutdown. Both Node and Python examples handle SIGTERM by flushing the queue. If you adapt the code to your own logger, keep that signal handler — container restarts otherwise lose ~2 s of unflushed entries.
• Stdout still works.Logging to Epok via app code doesn't replace Railway's built-in log viewer. Your console.log / printoutput still shows up in the Railway dashboard. You're adding persistence and intelligence on top, not replacing the existing path.
• Per-replica replicas. If your Railway service runs multiple replicas, every replica has a unique RAILWAY_REPLICA_ID. Useful for tracking per-replica error rates in Epok's Services view.

When the key is rejected (401 vs 403)

The status tells you which half of the problem you have. 401 with the body {"error":"unauthorized","detail":"..."} means no credential was read at all — a missing header or an unexpanded variable. 403 with {"error":"forbidden","detail":"Invalid API key. ..."} means a credential was read and then refused: unknown or rotated key, expired key, or a key without the ingest scope. The ordinary "wrong key" is a 403, not a 401. Railway is the worst platform on which to get either wrong, because both app-side paths are built to swallow it — deliberately. A logger must never break the request path, so the examples above discard delivery failures. That is correct in production and actively hostile during setup.

• Path 1, Node. The .catch(() => {}) is not even what hides it. fetch resolves on a 401 — only a network-level failure rejects — so the catch block never runs and the code never inspects res.ok. A wrong key produces zero output on stdout, zero output in the Railway log viewer, and zero logs in Epok. Nothing anywhere says why.
• Path 1, Python. urlopen does raise on any non-2xx, but the worker thread wraps it in except Exception: pass— the comment on that line says "never raise from a log handler" and it means it. Same outcome: total silence.
• Path 2, OpenTelemetry. The SDK reports export failures through its own internal diagnostic logger, which is quiet by default. Set OTEL_LOG_LEVEL=debugas a Railway variable to surface the exporter's response before assuming the endpoint is wrong.

So make it loud while you set up, then take it back out:

// In flush(), replace the bare .catch(() => {}) while you are setting up.
// fetch RESOLVES on 401 and 403 — only a network error rejects — so without
// this check a rejected key produces no output of any kind.
fetch(ENDPOINT, {
  method: 'POST',
  headers: { Authorization: `Bearer ${API_KEY}`, 'Content-Type': 'application/json' },
  body,
})
  .then(async (res) => {
    if (!res.ok) {
      console.error('epok ingest rejected:', res.status, await res.text());
    }
  })
  .catch((err) => console.error('epok ingest unreachable:', err));

The ingest plane accepts exactly three credential headers: X-API-Key: <key>, Authorization: Bearer <key> (what both examples above use), and Authorization: Basic <base64> with the key as the username and any password. A ?api_key= query parameter also works, but only on drain-style sources that cannot set headers — which is not the case here, so use a header. Notably Authorization: Token <key> is not accepted on ingest even though the app plane does accept it — on ingest a Token header parses as no credential at all, so a correct key sent that way still returns 401. Full per-plane matrix: authentication.

One Railway-specific cause worth ruling out first: variables are per-service and per-environment. A key set on production is not present in a PR environment, and process.env.EPOK_API_KEY being undefined sends the literal string Bearer undefined — which does not crash and does not even 401: undefined is a perfectly parseable credential, so it is read, looked up, and refused as a 403. (The Python example throws KeyError at import instead, because it reads os.environ["EPOK_API_KEY"] directly — the one place on this page where an unset variable is loud.)

A 403 is good news during setup. It proves your header shape is right and the plane read your credential — the key itself is then the thing to fix: wrong or rotated key, expired key, or a key missing the ingest scope (Settings → API Keys). On the search call below, 403 also covers a key without the read scope and a tenant ID in the path that the key does not belong to. Plan and feature gates are a separate matter, covered in limits.

Your first query

Seeing bytes move is not the same as being able to ask a question. Run this against the logs you just wired up — the Path 1 examples tag every entry with replica from RAILWAY_REPLICA_ID, so asking for lines where that field exists returns exactly what your Railway service sent and nothing else.

curl -X POST https://app.getepok.dev/api/v1/tenants/YOUR_TENANT_ID/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "replica:*",
    "start": "-1h",
    "end":   "now",
    "limit": 100
  }'

Your tenant ID is not printed on the API Keys screen. The quickest way to read it: while signed in, open app.getepok.dev/auth/me — the JSON it returns includes tenant_id. The body accepts five fields and no others: query, start, end, limit (max 10000), and before — an ISO timestamp cursor for paging to older logs.

A good result is a 200 whose count is non-zero and whose logs entries carry a _time inside the last hour plus the service, env, deploy and replica tags. That last one is the real proof: if replicais populated, your enrichment is reading Railway's injected variables correctly and per-replica error rates will work. On Path 2 those fields are not set — query service:"your-service-name" instead, matching OTEL_SERVICE_NAME.

A 200 with "count": 0 is not an auth failure. It means the request was authenticated and routed correctly and nothing matched. Re-run with "query": "*" before you assume delivery is broken — if * returns rows, your logs are landing and it is the field names that differ from what you asked for.

The same query in the product: open app.getepok.dev → Explore, paste replica:*, set the range to the last hour. Live Tail shows you the stream; Explore is where you interrogate it.