Send logs & metrics from AWS to Epok

Path 1: CloudWatch Logs → Lambda → Epok

Best when your logs already live in CloudWatch (Lambda functions, API Gateway, RDS, etc). One Lambda function subscribes to one or more log groups and forwards each batch to Epok.

1. Create the forwarder Lambda. Runtime: Python 3.12. Copy the code below into the function's lambda_function.py.
2. Set EPOK_API_KEYin the Lambda's environment variables. The key starts with epk_.
3. On each CloudWatch log group you want to forward, add a subscription filter. Filter pattern: leave it empty— that's CloudWatch's canonical "match every event" pattern. Destination: the Lambda you just created.
4. Verify in Live Tail — see the "Verify" section below.

Lambda function (Python 3.12)

# lambda_function.py — CloudWatch Logs → Epok forwarder
import base64
import gzip
import json
import os
import urllib.request

EPOK_ENDPOINT = "https://ingest.getepok.dev/insert/elasticsearch/_bulk"
EPOK_API_KEY = os.environ["EPOK_API_KEY"]


def lambda_handler(event, _context):
    # CloudWatch wraps log events in base64+gzip; unwrap.
    raw = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(raw))

    log_group = payload.get("logGroup", "unknown")
    log_stream = payload.get("logStream", "unknown")
    service = log_group.split("/")[-1] or log_group

    # Bulk API format: alternating {create:{}} + payload lines.
    lines = []
    for entry in payload.get("logEvents", []):
        lines.append(json.dumps({"create": {}}))
        lines.append(json.dumps({
            "_msg": entry["message"],
            "_time": int(entry["timestamp"]),  # ms epoch — accepted as-is
            "service": service,
            "log_group": log_group,
            "log_stream": log_stream,
        }))
    body = ("\n".join(lines) + "\n").encode("utf-8")

    req = urllib.request.Request(
        EPOK_ENDPOINT,
        data=body,
        headers={
            "Authorization": f"Bearer {EPOK_API_KEY}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return {"status": resp.status, "events": len(payload.get("logEvents", []))}

IAM permissions

The Lambda's execution role needs the AWS-managed AWSLambdaBasicExecutionRole policy (CloudWatch write for its own logs). Subscription filters use a separate role — AWS auto-prompts to create it when you attach the filter.

Path 2: Vector on EC2 / ECS host

Best when you control the host filesystem and want to tail /var/log/* directly (systemd journals, nginx access logs, application files). Vector is a single binary with no JVM, no Python — ideal for resource-constrained nodes.

1. Install Vector on the host: curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | bash
2. Drop the config below at /etc/vector/vector.yaml.
3. Set EPOK_API_KEY via /etc/default/vector or systemd environment file.
4. systemctl enable --now vector.

# /etc/vector/vector.yaml — EC2 host logs → Epok

sources:
  syslog:
    type: file
    include: ["/var/log/syslog", "/var/log/messages"]

  nginx:
    type: file
    include: ["/var/log/nginx/access.log", "/var/log/nginx/error.log"]

  app:
    type: file
    include: ["/var/log/myapp/*.log"]

transforms:
  add_host:
    type: remap
    inputs: [syslog, nginx, app]
    source: |
      .host = get_hostname!()
      .region = "us-east-1"

sinks:
  epok:
    type: elasticsearch
    inputs: [add_host]
    endpoints: ["https://ingest.getepok.dev"]
    bulk:
      index: logs
    # Required. Epok is not a real Elasticsearch: the default api_version
    # "auto" sniffs the version against our root (404) and the sink then
    # never starts, so nothing ships and nothing says why.
    api_version: v8
    healthcheck:
      enabled: false
    auth:
      strategy: basic
      user: ${EPOK_API_KEY}
      password: x

Path 3: ECS / EKS Fluent Bit sidecar

For containerized workloads, the AWS-maintained aws-for-fluent-bit image is the standard. Add it as a sidecar to your ECS task or as a DaemonSet in EKS.

# fluent-bit.conf

[SERVICE]
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            cri
    Tag               kube.*
    Refresh_Interval  5

[FILTER]
    Name              modify
    Match             *
    Add               env production

[OUTPUT]
    Name              loki
    Match             *
    Host              ingest.getepok.dev
    Port              443
    TLS               On
    HTTP_User         ${EPOK_API_KEY}
    HTTP_Passwd       x
    Labels            cluster=${ECS_CLUSTER}, task=${ECS_TASK_FAMILY}
    drop_single_key   on

For EKS, see the dedicated Kubernetes install guide—it covers DaemonSet manifest, RBAC, and node selectors.

Path 4: CloudWatch metrics → Epok

For infra metrics — EC2 CPU, RDS connections, ELB latency, Lambda errors, SQS depth — run the OpenTelemetry Collector's awscloudwatch receiver somewhere that can read CloudWatch, and push to Epok over OTLP. The AWS read credentials stay in your environment; Epok never holds them. Once the series land, the same metric detectors (saturation, anomaly, slow-drift, reporting-gap) run on them automatically — no rules to write.

# otel-cloudwatch.yaml — pull CloudWatch metrics → Epok
receivers:
  awscloudwatch:
    region: us-east-1
    poll_interval: 5m
    metrics:
      named:
        - { namespace: "AWS/EC2", metric_name: "CPUUtilization",      period: 5m }
        - { namespace: "AWS/RDS", metric_name: "DatabaseConnections", period: 5m }
        - { namespace: "AWS/ApplicationELB", metric_name: "TargetResponseTime", period: 5m }

processors:
  resource:
    attributes:
      - key: service.name        # the service these metrics belong to
        value: aws-infra
        action: upsert

exporters:
  otlphttp/epok:
    metrics_endpoint: https://ingest.getepok.dev/v1/metrics
    encoding: json               # set this — correct against every build
    headers: { Authorization: "Bearer ${EPOK_API_KEY}" }

service:
  pipelines:
    metrics:
      receivers: [awscloudwatch]
      processors: [resource]
      exporters: [otlphttp/epok]

The collector's IAM role needs read-only cloudwatch:GetMetricData + cloudwatch:ListMetrics. CloudWatch bills per metric pulled, so name the metrics you care about rather than autodiscovering everything. Confirm in Epok → Metrics within one poll interval.

Verify

1. Open app.getepok.dev → Live Tail. Within 60 seconds you should see log lines streaming in.
2. Open Services. The services you're sending should appear as cards with hit-rate + error-rate metrics.
3. Open New Errors. The first time any error-level log arrives, it shows up here grouped by pattern.

Common gotchas

• Subscription-filter throttling. CloudWatch throttles a single filter to ~5 MB/s. For high-volume log groups, split the subscription across multiple Lambdas with disjoint filter patterns.
• Lambda timeout. Default Lambda timeout is 3 seconds. Bump to 30 s to absorb occasional Epok-side latency spikes without dropping events.
• VPC-attached Lambdas need a NAT. If your forwarder Lambda lives in a private subnet, it needs internet egress to reach ingest.getepok.dev. Either move it out of the VPC, or add a NAT Gateway.
• Vector permissions. Vector runs as the vector user by default and needs read access to the log files. chgrp vector /var/log/myapp && chmod g+r /var/log/myapp/*.log covers the common case.

When the key is rejected (401 vs 403)

The status tells you which half of the problem you have. 401 with the body {"error":"unauthorized","detail":"..."} means no credential was read at all — a missing header, an unexpanded variable, or a scheme the log-ingest endpoints do not parse. 403 with {"error":"forbidden","detail":"Invalid API key. ..."} means a credential was read and then refused: unknown or rotated key, expired key, or a key without the ingest scope. So the ordinary "wrong key" is a 403, not a 401. The problem is that none of the four shippers above puts either one in front of you the same way — and two of them never show you the status code at all.

• Path 1 (Lambda). urllib.request.urlopen raises on any non-2xx, so a wrong key fails the whole invocation with HTTPError: HTTP Error 403: Forbidden (or 401 if the header never got set at all) — in the forwarder's ownCloudWatch log group, not the log group you subscribed. CloudWatch then retries the batch, so a bad key presents as a Lambda error-rate spike rather than as a delivery problem. Check the forwarder's log group first.
• Path 2 (Vector). Because healthcheck.enabled: false is required for this sink (see Path 2 above), Vector starts cleanly with a dead key — nothing fails at boot and nothing is printed at startup. The rejection appears only as per-request sink errors: journalctl -u vector -f.
• Path 3 (Fluent Bit). The key rides as HTTP Basic (HTTP_User), and the output plugin reports a generic delivery failure and retries with backoff — the status code is not reliably in the line. Set Log_Level debug in [SERVICE] to see the response before you go hunting for a network problem.
• Path 4 (collector). The collector logs the failing export with the response status, so this is the one path where the status code appears verbatim.

The log-ingest endpoints (Paths 1–3) accept exactly three credential headers: X-API-Key: <key>, Authorization: Bearer <key>, and Authorization: Basic <base64> with the key as the username (any password — the Vector and Fluent Bit configs above use x). A ?api_key= query parameter also works, but only on drain-style sources that cannot set headers. Notably Authorization: Token <key> is not accepted there — it parses as no credential at all, so a correct key sent that way still returns 401. Path 4 is the exception: /v1/metrics is served by the app plane, which does accept Token. The plane is decided by the path, not the hostname. Full per-plane matrix: authentication.

A 403 is good news during setup. It proves your header shape is right and the plane read your credential — the key itself is then the thing to fix: wrong or rotated key, expired key, or a key missing the ingest scope (Settings → API Keys). On the search call below, 403 also covers a key without the read scope and a tenant ID in the path that the key does not belong to. Plan and feature gates are a separate matter, covered in limits.

Your first query

Seeing bytes move is not the same as being able to ask a question. Run this against the logs you just wired up — the Path 1 forwarder tags every entry with log_group, so asking for lines where that field exists returns exactly what came through CloudWatch.

curl -X POST https://app.getepok.dev/api/v1/tenants/YOUR_TENANT_ID/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "log_group:*",
    "start": "-1h",
    "end":   "now",
    "limit": 100
  }'

Your tenant ID is not printed on the API Keys screen. The quickest way to read it: while signed in, open app.getepok.dev/auth/me — the JSON it returns includes tenant_id. The body accepts five fields and no others: query, start, end, limit (max 10000), and before — an ISO timestamp cursor for paging to older logs.

A good result is a 200 whose count is non-zero and whose logs entries carry a _time inside the last hour plus the log_group, log_stream and service fields the forwarder set. On the Vector path substitute host:*; on the Fluent Bit path, cluster:*.

A 200 with "count": 0 is not an auth failure. It means the request was authenticated and routed correctly and nothing matched. Re-run with "query": "*" before you assume delivery is broken — if * returns rows, your logs are landing and it is the field names that differ from what you asked for.

The same query in the product: open app.getepok.dev → Explore, paste log_group:*, set the range to the last hour. Live Tail shows you the stream; Explore is where you interrogate it.