Send logs from AWS to Epok

Three working paths, ordered by setup time. Pick the one that matches where your logs already are. All three end with logs landing in Epok Live Tail within 60 seconds and Epok's detectors running on them automatically.

Time to first log: 5–10 min · Trial: 14 days, no card · API key: app.getepok.dev → Settings → API Keys

Path 1: CloudWatch Logs → Lambda → Epok

Best when your logs already live in CloudWatch (Lambda functions, API Gateway, RDS, etc). One Lambda function subscribes to one or more log groups and forwards each batch to Epok.

Create the forwarder Lambda. Runtime: Python 3.12. Copy the code below into the function's lambda_function.py.
Set EPOK_API_KEYin the Lambda's environment variables. The key starts with epk_.
On each CloudWatch log group you want to forward, add a subscription filter. Filter pattern: leave it empty— that's CloudWatch's canonical "match every event" pattern. Destination: the Lambda you just created.
Verify in Live Tail — see the "Verify" section below.

Lambda function (Python 3.12)

python

# lambda_function.py — CloudWatch Logs → Epok forwarder
import base64
import gzip
import json
import os
import urllib.request

EPOK_ENDPOINT = "https://ingest.getepok.dev/insert/elasticsearch/_bulk"
EPOK_API_KEY = os.environ["EPOK_API_KEY"]


def lambda_handler(event, _context):
    # CloudWatch wraps log events in base64+gzip; unwrap.
    raw = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(raw))

    log_group = payload.get("logGroup", "unknown")
    log_stream = payload.get("logStream", "unknown")
    service = log_group.split("/")[-1] or log_group

    # Bulk API format: alternating {create:{}} + payload lines.
    lines = []
    for entry in payload.get("logEvents", []):
        lines.append(json.dumps({"create": {}}))
        lines.append(json.dumps({
            "_msg": entry["message"],
            "_time": int(entry["timestamp"]),  # ms epoch — accepted as-is
            "service": service,
            "log_group": log_group,
            "log_stream": log_stream,
        }))
    body = ("\n".join(lines) + "\n").encode("utf-8")

    req = urllib.request.Request(
        EPOK_ENDPOINT,
        data=body,
        headers={
            "Authorization": f"Bearer {EPOK_API_KEY}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return {"status": resp.status, "events": len(payload.get("logEvents", []))}

IAM permissions

The Lambda's execution role needs the AWS-managed AWSLambdaBasicExecutionRole policy (CloudWatch write for its own logs). Subscription filters use a separate role — AWS auto-prompts to create it when you attach the filter.

Path 2: Vector on EC2 / ECS host

Best when you control the host filesystem and want to tail /var/log/* directly (systemd journals, nginx access logs, application files). Vector is a single binary with no JVM, no Python — ideal for resource-constrained nodes.

Install Vector on the host: curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | bash
Drop the config below at /etc/vector/vector.yaml.
Set EPOK_API_KEY via /etc/default/vector or systemd environment file.
systemctl enable --now vector.

yaml

# /etc/vector/vector.yaml — EC2 host logs → Epok

sources:
  syslog:
    type: file
    include: ["/var/log/syslog", "/var/log/messages"]

  nginx:
    type: file
    include: ["/var/log/nginx/access.log", "/var/log/nginx/error.log"]

  app:
    type: file
    include: ["/var/log/myapp/*.log"]

transforms:
  add_host:
    type: remap
    inputs: [syslog, nginx, app]
    source: |
      .host = get_hostname!()
      .region = "us-east-1"

sinks:
  epok:
    type: elasticsearch
    inputs: [add_host]
    endpoint: https://ingest.getepok.dev
    bulk:
      index: logs
    auth:
      strategy: basic
      user: ${EPOK_API_KEY}
      password: x

Path 3: ECS / EKS Fluent Bit sidecar

For containerized workloads, the AWS-maintained aws-for-fluent-bit image is the standard. Add it as a sidecar to your ECS task or as a DaemonSet in EKS.

ini

# fluent-bit.conf

[SERVICE]
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            cri
    Tag               kube.*
    Refresh_Interval  5

[FILTER]
    Name              modify
    Match             *
    Add               env production

[OUTPUT]
    Name              loki
    Match             *
    Host              ingest.getepok.dev
    Port              443
    TLS               On
    HTTP_User         ${EPOK_API_KEY}
    HTTP_Passwd       x
    Labels            cluster=${ECS_CLUSTER}, task=${ECS_TASK_FAMILY}
    drop_single_key   on

For EKS, see the dedicated Kubernetes install guide—it covers DaemonSet manifest, RBAC, and node selectors.

Verify

Open app.getepok.dev → Live Tail. Within 60 seconds you should see log lines streaming in.
Open Services. The services you're sending should appear as cards with hit-rate + error-rate metrics.
Open New Errors. The first time any error-level log arrives, it shows up here grouped by pattern.

Common gotchas

Subscription-filter throttling. CloudWatch throttles a single filter to ~5 MB/s. For high-volume log groups, split the subscription across multiple Lambdas with disjoint filter patterns.
Lambda timeout. Default Lambda timeout is 3 seconds. Bump to 30 s to absorb occasional Epok-side latency spikes without dropping events.
VPC-attached Lambdas need a NAT. If your forwarder Lambda lives in a private subnet, it needs internet egress to reach ingest.getepok.dev. Either move it out of the VPC, or add a NAT Gateway.
Vector permissions. Vector runs as the vector user by default and needs read access to the log files. chgrp vector /var/log/myapp && chmod g+r /var/log/myapp/*.log covers the common case.

Logs flowing? Configure notification channels in Settings → Notifications so Epok can page you when a new error appears or a service goes silent.