Send logs from GCP to Epok

Two paths. Pick by where your logs already live: Cloud Logging (most GCP workloads) or directly on a GCE/GKE host.

Time to first log: 5–10 min · Trial: 14 days, no card · API key: app.getepok.dev → Settings → API Keys

Path 1: Cloud Logging → Pub/Sub → Cloud Function

Standard pattern. A Logs Router sink ships every matching log to a Pub/Sub topic; a Cloud Function consumes the topic and POSTs batches to Epok. Works for Cloud Run, GKE, Compute Engine, App Engine, and any service that writes to Cloud Logging.

1. Create the Pub/Sub topic

bash

gcloud pubsub topics create epok-logs

2. Create a Logs Router sink that writes to the topic

bash

# Send every log from project PROJECT_ID to the topic. Narrow with
# --log-filter='resource.type="cloud_run_revision"' if you only want
# a subset.

gcloud logging sinks create epok-sink \
  pubsub.googleapis.com/projects/PROJECT_ID/topics/epok-logs \
  --log-filter='severity >= DEFAULT' \
  --description='Forward Cloud Logging entries to Epok'

# Grant the sink's writer identity permission to publish.
WRITER=$(gcloud logging sinks describe epok-sink --format='value(writerIdentity)')
gcloud pubsub topics add-iam-policy-binding epok-logs \
  --member="$WRITER" --role=roles/pubsub.publisher

3. Deploy the consumer Cloud Function (Python 3.12)

python

# main.py — Pub/Sub-triggered consumer for epok-logs
import base64
import json
import os
import urllib.request

EPOK_ENDPOINT = "https://ingest.getepok.dev/insert/elasticsearch/_bulk"
EPOK_API_KEY = os.environ["EPOK_API_KEY"]


def forward(event, _context):
    """Triggered by a single Pub/Sub message."""
    raw = base64.b64decode(event["data"]).decode("utf-8")
    entry = json.loads(raw)

    body = (
        json.dumps({"create": {}}) + "\n" +
        json.dumps({
            "_msg": entry.get("textPayload") or json.dumps(entry.get("jsonPayload", {})),
            "_time": entry.get("timestamp"),
            "severity": entry.get("severity", "DEFAULT").lower(),
            "service": (entry.get("resource", {}).get("labels", {}).get("service_name")
                        or entry.get("logName", "").split("/")[-1]),
            "resource_type": entry.get("resource", {}).get("type"),
        }) + "\n"
    ).encode("utf-8")

    req = urllib.request.Request(
        EPOK_ENDPOINT,
        data=body,
        headers={
            "Authorization": f"Bearer {EPOK_API_KEY}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return {"status": resp.status}

bash

gcloud functions deploy epok-forwarder \
  --gen2 \
  --runtime=python312 \
  --region=us-central1 \
  --trigger-topic=epok-logs \
  --entry-point=forward \
  --set-env-vars=EPOK_API_KEY=epk_REPLACE_ME \
  --max-instances=10

Note: GCP's severity field uses the labels DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY. Epok recognizes these natively — but if you want a different mapping (e.g. NOTICE → info, ALERT → critical), set it up under Settings → Log Processing → Level Mapping Rules.

Path 2: Vector on GCE / GKE node

Skip Cloud Logging entirely if you control the host. Vector tails files or systemd journal and ships directly. Cheaper than the Pub/Sub path at high volume (no Cloud Logging egress fees).

yaml

# /etc/vector/vector.yaml — GCE/GKE host → Epok

sources:
  journal:
    type: journald
    include_units: [my-app.service, nginx.service]
    current_boot_only: false

  app:
    type: file
    include: ["/var/log/myapp/*.log"]

transforms:
  enrich:
    type: remap
    inputs: [journal, app]
    source: |
      .host = get_hostname!()
      .region = "us-central1"
      .cloud = "gcp"

sinks:
  epok:
    type: elasticsearch
    inputs: [enrich]
    endpoint: https://ingest.getepok.dev
    bulk:
      index: logs
    auth:
      strategy: basic
      user: ${EPOK_API_KEY}
      password: x

For GKE specifically (DaemonSet manifests, RBAC), use the Kubernetes install guide instead — same Vector binary, simpler delivery.

Verify

Open app.getepok.dev → Live Tail. Within 60–120 seconds (Pub/Sub adds a few seconds of buffering) you should see log lines.
Open Services — entries should appear keyed on the resource_type (e.g. cloud_run_revision, k8s_container).
If your severitylabels aren't mapping cleanly, check Settings → Log Processing → Level Mapping Rules and add field-value rules for any non-standard labels.

Common gotchas

Logs Router sink scope. Sinks are project-scoped by default. For a multi-project org, create the sink at the folder or organization level (gcloud logging sinks create --organization=ORG_ID).
Cloud Function cold starts. First trigger after an idle period can add ~1–3 s of latency. Set --min-instances=1 for sub-second forwarding if it matters.
Pub/Sub message size.10 MB per message. Cloud Logging entries are usually well under this; if you're shipping binary payloads, configure the sink's --output-version-format=V2.
JSON payloads. Cloud Run / GKE structured logs arrive with jsonPayload instead of textPayload. The forwarder above serializes the whole object — if you'd rather pick fields, customize the_msg derivation in the function.

Logs flowing? Wire up notification channels under Settings → Notifications so Epok pages you when something breaks.