Skip to content

Loki: storage, retention, and the Alloy pipeline

The deployment is documented in kubernetes/system/loki.md. This page covers what's specific to our config:

  1. Storage backend is R2. Chunks + index live in the shared tomoda-observability bucket alongside Tempo blocks. PVC is disabled.
  2. Retention is pinned to 7 days Loki-side, 14 days bucket-side (lifecycle).
  3. The Alloy pipeline parses the tomoda backend's Zap JSON logs and surfaces trace_id for trace-to-log navigation from Grafana / Tempo.

Loki runs the loki chart (single-binary, Loki 3.x) with a single tsdb/v13 schema period, so structured metadata is enabled; log shipping runs on Grafana Alloy (a DaemonSet).

Storage: shared R2 bucket

Loki writes chunks and the tsdb index to tomoda-observability (infrastructure/cloudflare/r2.tf), the same bucket as Tempo, separated by prefix:

Writer Prefix What lands here
Loki bucket root (chunks), index_ (tsdb index) Chunks, index, compactor working files
Tempo tempo/ (set explicitly in tempo values) Trace blocks

Credentials come from the monitoring/r2-observability-credentials Secret (ExternalSecret over tomoda-r2-cluster-credentials), injected as R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY and expanded into the S3 config by -config.expand-env=true.

Retention math

Knob Value Why
loki.limits_config.retention_period 168h (7d) Query-API retention. After this Loki refuses to query the chunks even if they still exist in the bucket.
loki.compactor.retention_enabled true Without this, retention is best-effort only.
loki.compactor.retention_delete_delay 2h Soft window before old chunks are actually deleted by the compactor.
loki.compactor.delete_request_store s3 Required by the v13/tsdb path.
R2 lifecycle on the bucket 14 days → Delete Bucket-side safety net. Loki normally drops chunks at 7d via the compactor; this 14d backstop catches any chunks the compactor misses.

Estimated write rate (current cluster):

  • ~10 backend / system pods continuously logging at ~5 lines/sec each
  • ≈ 4.3 M lines/day raw
  • ≈ 1.5 GB/day uncompressed
  • ≈ 150 MB/day after Loki's gzip chunk compression
  • 7 × 150 MB ≈ 1.05 GB of active chunk data in R2 at steady state

That sits inside R2's free 10 GB. The win is operational (no PVC to grow, snapshot, or recover) and architectural (Tempo and Loki share one storage primitive).

To change retention, bump both:

  1. loki.limits_config.retention_period in values.yaml (Loki query API)
  2. The observability lifecycle in infrastructure/cloudflare/r2.tf (deletion floor)

together: Loki querying chunks the bucket already deleted returns confusing 5xx.

If you see write-rate spikes:

  • Compactor is actually running. kubectl logs -n monitoring -l app.kubernetes.io/name=loki | grep compactor. If it's silent for hours, retention isn't being enforced.
  • One pod isn't spamming. Run topk(10, sum by (namespace, pod) (rate({__name__=~".+"}[5m]))) in Grafana to find the loudest pod and either fix it or drop its logs at the Alloy level.

Alloy pipeline

The pipeline lives in k8s/envs/platform/loki/values-alloy.yaml as an Alloy River config (loki.process "pipeline"). Two stage.match blocks run in order:

Traefik

Selector: {container="traefik"}. Parses the JSON access log and promotes entryPointName, request_Host, status, method, level to labels. The Traefik Grafana dashboard's access-log tables depend on these labels.

Tomoda backend

Selector: {app=~"tomoda-(api|async)"}. The backend uses Zap with a JSON encoder, so each log line is shaped like:

{
  "level": "info",
  "ts": 1716480000,
  "caller": "main.go:45",
  "msg": "request handled",
  "trace_id": "8a4f5e0e9b1b9c1f1e1d1c1b1a191817",
  "span_id": "0123456789abcdef"
}

The pipeline:

  1. Parses the JSON.
  2. Promotes level to a label. Cardinality is bounded (debug, info, warn, error, fatal, panic), so this is safe and gives free filtering: {app="tomoda-api", level="error"}.
  3. trace_id and span_id stay in the raw log body: not promoted to a label, not lifted into structured metadata. They're queryable at query time via LogQL's json stage.
stage.match {
  selector = "{app=~\"tomoda-(api|async)\"}"
  stage.json {
    expressions = { level = "level" }
  }
  stage.labels {
    values = { level = "" }
  }
}

trace_id stays out of the index

Promoting trace_id to a Loki label would multiply the index by per-request cardinality, the standard way to wreck a Loki cluster. It stays in the log body, queried at query time:

{app="tomoda-api"} | json | trace_id="8a4f5e0e..."

The same syntax backs Grafana's tracesToLogsV2 link. Now that we're on Loki 3.x / schema v13, structured_metadata is available as a future home for trace_id / span_id / caller (cardinality-safe, indexed): add a stage.structured_metadata block in values-alloy.yaml. Query-time | json remains correct and is the current pattern.

Label strategy: what's safe and what's not

Field Promote to label? Why
namespace yes (Alloy relabel does it automatically) bounded by namespace count
app yes (auto) bounded by app count
level yes bounded enum (debug/info/warn/error/fatal/panic)
status (Traefik) yes bounded HTTP status codes
method yes bounded HTTP method set
trace_id No: query-time JSON parse per-request, unbounded
span_id No: query-time JSON parse per-request, unbounded
caller No: query-time JSON parse many call sites; not useful as a label
user_id (if ever added) No: query-time JSON parse per-user, unbounded
request_id No: query-time JSON parse per-request

Rule of thumb: if a field has fewer than ~100 distinct values cluster-wide, label is OK. Anything per-request, per-user, or per-trace either gets queried via | json | <field>="..." or lifted into structured_metadata (available now on Loki 3.x / v13). It never becomes a label.

Trace-to-log jump

This is what makes the Tempo integration useful. From a Tempo span:

{namespace="tomoda-prod"} | json | trace_id="8a4f5e0e9b1b9c1f1e1d1c1b1a191817"

is run automatically when you click "View logs" on a span. The tracesToLogsV2 config in monitoring/values.yaml wires this; see Tempo. The | json stage is required because trace_id isn't a label (see retention sizing note above); it's pulled from the JSON body at query time.

Going the other direction, the Loki data source has a derivedFields rule:

- name: TraceID
  matcherRegex: '"trace_id":"(\w+)"'
  url: '${__value.raw}'
  datasourceUid: tempo

Any log line containing "trace_id":"..." gets a clickable link in Grafana that opens the matching trace in Tempo.

Debugging the pipeline

# Is Alloy running on every node and shipping?
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "loki.write\|error" | head -20

# Tail a recent tomoda log line and check the extracted labels
kubectl port-forward -n monitoring svc/loki 3100:3100
curl -s -G 'http://localhost:3100/loki/api/v1/labels' | jq

# Sample query for a known trace
curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={app="tomoda-api"} | trace_id="abc..."' \
  --data-urlencode 'start='$(date -d '1 hour ago' +%s)000000000 | jq '.data.result[0]'

If a tomoda log line shows up in Grafana but level is missing as a label, the JSON parse failed, usually because the line isn't JSON (e.g. a panic stack trace, or a third-party library logging with a different format). The pipeline keeps those; they're queryable by {app="tomoda-api"} without structured filtering.