epok
Live demoStart trial

Send Kubernetes logs to Epok

Updated Jul 28, 2026 · 1d ago

Vector as a DaemonSet on every node. One pod per worker reads /var/log/containers/*.logvia the host mount, parses the CRI log format, and ships to Epok. Once logs arrive, Epok's 70+ Kubernetes detection rules fire automatically: CrashLoopBackOff, OOMKilled, ImagePullBackOff, FailedScheduling, probe failures, eviction patterns.

Time to first log: 5 min · Works on: EKS, GKE, AKS, k3s, kind, self-hosted · API key: app.getepok.dev → Settings → API Keys

1. Store the API key as a secret

bash
kubectl create namespace epok
kubectl -n epok create secret generic epok-credentials \
  --from-literal=api-key=epk_REPLACE_ME

2. Vector config (ConfigMap)

yaml
# epok-vector-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: epok-vector-config
  namespace: epok
data:
  vector.yaml: |
    sources:
      kube_logs:
        type: kubernetes_logs
        # Vector auto-discovers pods, attaches namespace/pod/container
        # labels, and parses the CRI/JSON log format. No extra config
        # needed for the common case.
        glob_minimum_cooldown_ms: 200

    transforms:
      enrich:
        type: remap
        inputs: [kube_logs]
        source: |
          # Map the kubernetes_logs fields to Epok's expected shape.
          .service = .kubernetes.container_name
          .namespace = .kubernetes.pod_namespace
          .pod = .kubernetes.pod_name
          .node = .kubernetes.pod_node_name
          ._msg = .message
          del(.message)

    sinks:
      epok:
        type: elasticsearch
        inputs: [enrich]
        endpoints: ["https://ingest.getepok.dev"]
        bulk:
          index: logs
        # Required. Epok is not a real Elasticsearch: the default
        # api_version "auto" sniffs the version against our root (404) and the
        # sink then never starts, so nothing ships and nothing says why.
        api_version: v8
        healthcheck:
          enabled: false
        auth:
          strategy: basic
          user: ${EPOK_API_KEY}
          password: x
        # Sensible defaults for k8s log volume.
        batch:
          max_events: 1000
          timeout_secs: 5
        buffer:
          type: memory
          max_events: 5000
          when_full: drop_newest

3. RBAC for Vector

Vector needs read access to pods + namespaces to enrich log entries with metadata.

yaml
# epok-vector-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: epok-vector
  namespace: epok
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: epok-vector
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces", "nodes"]
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: epok-vector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: epok-vector
subjects:
- kind: ServiceAccount
  name: epok-vector
  namespace: epok

4. DaemonSet

yaml
# epok-vector-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: epok-vector
  namespace: epok
spec:
  selector:
    matchLabels: { app: epok-vector }
  template:
    metadata:
      labels: { app: epok-vector }
    spec:
      serviceAccountName: epok-vector
      tolerations:
      - operator: Exists                  # Run on every node, incl. tainted ones
      containers:
      - name: vector
        # Pin to a specific version, not :latest, so a Vector release
        # never breaks your log pipeline. Bump deliberately after testing.
        image: timberio/vector:0.43.0-alpine
        env:
        - name: EPOK_API_KEY
          valueFrom:
            secretKeyRef:
              name: epok-credentials
              key: api-key
        resources:
          requests: { cpu: 50m,  memory: 128Mi }
          limits:   { cpu: 500m, memory: 512Mi }
        volumeMounts:
        - { name: config,   mountPath: /etc/vector }
        - { name: var-log,  mountPath: /var/log,           readOnly: true }
        - { name: var-lib,  mountPath: /var/lib/docker,    readOnly: true }
        - { name: data,     mountPath: /vector-data }
      volumes:
      - { name: config,   configMap: { name: epok-vector-config } }
      - { name: var-log,  hostPath:  { path: /var/log } }
      - { name: var-lib,  hostPath:  { path: /var/lib/docker } }
      - { name: data,     emptyDir: {} }

5. Apply everything

bash
kubectl apply -f epok-vector-config.yaml
kubectl apply -f epok-vector-rbac.yaml
kubectl apply -f epok-vector-daemonset.yaml

# Watch the pods come up
kubectl -n epok rollout status daemonset/epok-vector

# Tail one of the agents to confirm
kubectl -n epok logs -l app=epok-vector --tail=20 -f

Verify

  1. Open app.getepok.dev → Live Tail. Container logs should stream in within 30–60 s, tagged with namespace, pod, container.
  2. Trigger a failure: kubectl delete pod <some-pod> and force a CrashLoopBackOff. Within ~2 minutes the K8s Intelligence detector fires an alert in New Errors.
  3. Open Services. Each unique container_name appears as a service card with hit counts.

When the key is wrong, nothing says so

This is the failure mode to internalize before you go looking anywhere else. The DaemonSet carries the key in the sink config, and we told you to set healthcheck.enabled: false above (you have to — the sink otherwise never starts). The consequence is that Vector comes up healthy with a bad key. The pods are Running, the rollout succeeded, kubectl get podsis clean, and Epok shows nothing — because the rejection is happening one layer down, on every batch, and the only place it is written down is the agent's own log.

terminal
bash
# The sink errors live in the agent's OWN log, not in Epok.
kubectl -n epok logs -l app=epok-vector --tail=200 | grep -iE 'error|401|403|unauthorized|forbidden'

# Check the secret for a trailing newline — the #1 cause of a 403 here.
# --from-literal is clean; --from-file keeps the file's final newline.
kubectl -n epok get secret epok-credentials \
  -o jsonpath='{.data.api-key}' | base64 -d | xxd | tail -2

The newline trap is specific to this setup. The Vector config authenticates with strategy: basic, which sends the key as the HTTP Basic username. Epok trims whitespace around a key sent via X-API-Key or Bearer, but the username parsed out of a Basic credential is used as-is — so any stray whitespace that survives into that username hashes to something that is not your key, and you get a 403 rather than a 401. --from-literal (step 1) stores the key clean; --from-file keeps whatever trailing newline the file had, which is why the xxd check above is worth ten minutes of staring at YAML. Use printf, not echo, if you must write that file.

To separate a key problem from a config problem, take the cluster out of the loop and hit the same endpoint the sink hits:

terminal
bash
curl -i -X POST https://ingest.getepok.dev/_bulk \
  -H 'Content-Type: application/x-ndjson' \
  -H 'X-API-Key: YOUR_API_KEY' \
  --data-binary $'{"create":{}}\n{"_msg":"auth probe","service":"probe"}\n'

# 200 {"status":"ok","accepted":1}            key + header are both fine
# 401 {"error":"unauthorized","detail":...}   no key found in ANY accepted header
# 403 {"error":"forbidden","detail":...}      a key WAS found, and rejected
# 429 {"error":"rate_limited",...}            key is FINE — quota or rate limit

401 and 403 mean different things. 401 is “no credential found in any accepted header” — the key never made it out of the secret, or the sink is sending a header name Epok does not read. 403 is “a key was read, and rejected” — rotated, invalid (the newline case), missing the ingest scope, or the tenant disabled. The detail field names which one. A quota or plan gate is not a 403: an expired trial or a blown daily volume cap answers 429 with a retry_after, which means your key is good and your billing is not. Every accepted credential form is in Authentication; the plan gates are in Limits & error codes.

Common gotchas

  • Containerd vs Docker. Modern clusters use containerd — logs live under /var/log/containers/ (already mounted above). The /var/lib/docker mount is a no-op on pure-containerd nodes and harmless to leave in.
  • PodSecurity / Pod Security Admission. DaemonSets that mount host paths need a privileged or baseline policy. If your cluster enforces restricted, create the epok namespace with the pod-security.kubernetes.io/enforce=privileged label.
  • EKS + Fargate.Fargate nodes can't run DaemonSets. Use Fluent Bit's built-in EKS Fargate logging instead — config snippet in the AWS guide.
  • Resource limits. The 500 m CPU / 512 Mi memory cap is right for ~5 k logs/sec/node. Bump if you see Vector OOMing under heavy log bursts.

Kubernetes metrics (CPU, memory, pods)

The Vector DaemonSet above ships logs. To get node and pod CPU/memory, restart counts, and pod phase — the signals the metric detectors and the per-service infrastructure panel read — run the OpenTelemetry Collector as a DaemonSet with the kubeletstats and k8s_cluster receivers. Add this to the collector ConfigMap:

otel-collector-metrics.yaml (DaemonSet ConfigMap)
yaml
receivers:
  kubeletstats:                       # per-node + per-pod resource usage
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: https://${env:K8S_NODE_NAME}:10250
    insecure_skip_verify: true
    metric_groups: [node, pod, container, volume]
  k8s_cluster:                        # cluster-level state (pod phase, restarts)
    collection_interval: 30s
processors:
  # The Vector log DaemonSet above keys logs by container name as `service`.
  # Stamp the same onto metrics so pod CPU/memory JOIN the right service —
  # kubeletstats leaves service unset, so without this the metrics land with no
  # service and never correlate with the logs on the service page.
  transform/service:
    metric_statements:
      - context: resource
        statements:
          - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["k8s.container.name"] != nil
exporters:
  otlphttp/epok:
    metrics_endpoint: https://ingest.getepok.dev/v1/metrics
    encoding: json                    # set this — correct against every build
    headers: { Authorization: "Bearer YOUR_API_KEY" }
service:
  pipelines:
    metrics:
      receivers: [kubeletstats, k8s_cluster]
      processors: [transform/service]
      exporters: [otlphttp/epok]

The collector's ServiceAccount needs read access to the kubelet (/stats/summary). Once metrics arrive, the metric detectors (saturation, anomaly, slow drift, reporting gap) switch on automatically for the cluster, and pod CPU/memory appear on each service page. For cluster-wide series like Pending pods or restart storms, add kube-state-metrics.

Your first query

Shipping is not the finish line — reading it back is. The enrich transform in step 2 puts a namespace on every line the DaemonSet sends, so namespace:* (the field exists) is a clean way to ask “show me everything that arrived from the cluster.” Your tenant id is in Settings → API Keys, and in the app URL.

terminal
bash
curl -X POST https://app.getepok.dev/api/v1/tenants/YOUR_TENANT_ID/search \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "namespace:*",
  "start": "-1h",
  "end": "now",
  "limit": 100
}'

A good result is a logs array of up to 100 entries, each carrying _msg, _time, and the service / namespace / pod / node fields the remap set — plus a count and an elapsed_ms. If count is 0 but the agent logs are clean, widen start to -24h before concluding anything: an idle cluster genuinely may not have logged in the last hour. The request body takes exactly five fields — query, start, end, limit, and a before timestamp cursor for paging backward.

The more useful second query is a breakdown — which pods are actually talking, and how much:

request body
json
{
  "query": "namespace:* | stats by (pod) count()",
  "start": "-1h",
  "end": "now",
  "limit": 100
}

In the product this is Explore — same syntax, same results, with the field list down the side. Note that container logs arrive without a level unless your application emits one as a JSON field, so level:error across a whole cluster is usually a thinner filter than people expect; the full grammar, and how to work around that, is in Search syntax.

Want pod-level views out of the box? Once logs are flowing, Epok auto-builds Service Health views per container; add the metrics DaemonSet above and each service page also shows pod CPU, memory, and restart counts. No dashboards to configure.

Next

  • Metrics & infrastructure — kube-state-metrics and node series, so a restart storm has CPU and memory next to it.
  • Traces (APM) — the next signal. Logs tell you a pod is unhappy; spans tell you which request path made it that way.
  • Detectors — what fires on this data once it is flowing, and what each one needs before it will fire at all.
  • AWS — EKS on Fargate, plus the CloudWatch and ALB log paths around the cluster.
  • Authentication — the full credential matrix behind the 401 / 403 triage above.