Loki — storage, retention, and the Alloy pipeline

The deployment is documented in kubernetes/system/loki.md. This page covers what's specific to our config:

1. Storage backend is GCS. Chunks + index live in the shared tomoda-observability-${project_id} bucket alongside Tempo blocks. PVC is disabled.
2. Retention is pinned to 7 days Loki-side, 14 days GCS-side (lifecycle).
3. The Alloy pipeline parses the tomoda backend's Zap JSON logs and surfaces trace_id for trace-to-log navigation from Grafana / Tempo.

Loki runs the loki chart (single-binary, Loki 3.x) with the tsdb/v13 schema; log shipping runs on Grafana Alloy (a DaemonSet), which replaced the EOL Promtail. The migration cutover runbook below covers the schema boundary.

Storage — shared GCS bucket

Loki writes chunks and the tsdb index to tomoda-observability-${project_id} (provisioned by infrastructure/gcp/tempo.tf) — same bucket as Tempo. Two writers, one bucket, separated by chart-default prefixes:

Writer	Prefix	What lands here
Loki	loki/ (chart default, no explicit prefix needed)	Chunks + tsdb index + compactor working files
Tempo	tempo/ (set explicitly in tempo values)	Trace blocks

Authentication via Workload Identity — KSA monitoring/loki impersonates the GCP SA observability@${project_id}.iam.gserviceaccount.com. No service account key is mounted. The link is one annotation:

serviceAccount:
  create: true
  name: loki
  annotations:
    iam.gke.io/gcp-service-account: observability@development-485000.iam.gserviceaccount.com

The SA holds roles/storage.objectAdmin on the bucket (full read/write/delete on objects). See tempo.md for the matching binding.

Retention math

Knob	Value	Why
loki.limits_config.retention_period	168h (7d)	Query-API retention. After this Loki refuses to query the chunks even if they still exist in GCS.
loki.compactor.retention_enabled	true	Without this, retention is best-effort only.
loki.compactor.retention_delete_delay	2h	Soft window before old chunks are actually deleted by the compactor.
loki.compactor.delete_request_store	gcs	Required by the v13/tsdb path (was implicit under v11).
GCS lifecycle on the bucket	age = 14 → Delete	Bucket-side safety net. Loki normally drops chunks at 7d via the compactor; this 14d backstop catches any chunks the compactor misses.

Estimated write rate (current cluster):

• ~10 backend / system pods continuously logging at ~5 lines/sec each
• ≈ 4.3 M lines/day raw
• ≈ 1.5 GB/day uncompressed
• ≈ 150 MB/day after Loki's gzip chunk compression
• 7 × 150 MB ≈ 1.05 GB of active chunk data in GCS at steady state

At GCS Standard pricing (~$0.020/GB/month in us-central1), 1 GB / month is ~$0.02 — well under the noise floor of the rest of the observability stack. The Loki migration from PVC to GCS is a net-zero cost change at this volume; the win is operational (no PVC to grow, snapshot, or recover) and architectural (Tempo and Loki share one storage primitive).

If retention needs ever change, bump:

1. loki.limits_config.retention_period in values.yaml (Loki query API)
2. The GCS lifecycle rule in infrastructure/gcp/tempo.tf (deletion floor)

…together. They have to move in lockstep — Loki querying chunks that GCS deleted yields confusing 5xx.

If you see write-rate spikes:

• Compactor is actually running. kubectl logs -n monitoring -l app.kubernetes.io/name=loki | grep compactor. If it's silent for hours, retention isn't being enforced.
• One pod isn't spamming. Run topk(10, sum by (namespace, pod) (rate({__name__=~".+"}[5m]))) in Grafana to find the loudest pod and either fix it or drop its logs at the Alloy level.

Alloy pipeline

The pipeline lives in k8s/envs/platform/loki/values-alloy.yaml as an Alloy River config (loki.process "pipeline"). Two stage.match blocks run in order — the same behaviour the Promtail match stages had:

Traefik (unchanged behavior)

Selector: {container="traefik"}. Parses the JSON access log and promotes entryPointName, request_Host, status, method, level to labels. This is what powers the Traefik logs Grafana dashboard (grafana.com 13702).

Tomoda backend (new)

Selector: {app=~"tomoda-(api|async)"}. The backend uses Zap with a JSON encoder, so each log line is shaped like:

{
  "level": "info",
  "ts": 1716480000,
  "caller": "main.go:45",
  "msg": "request handled",
  "trace_id": "8a4f5e0e9b1b9c1f1e1d1c1b1a191817",
  "span_id": "0123456789abcdef"
}

The pipeline:

1. Parses the JSON.
2. Promotes level to a label. Cardinality is bounded (debug, info, warn, error, fatal, panic), so this is safe and gives free filtering: {app="tomoda-api", level="error"}.
3. trace_id and span_id stay in the raw log body — not promoted to a label, not lifted into structured metadata. They're queryable at query time via LogQL's json stage.

stage.match {
  selector = "{app=~\"tomoda-(api|async)\"}"
  stage.json {
    expressions = { level = "level" }
  }
  stage.labels {
    values = { level = "" }
  }
}

trace_id stays out of the index

Promoting trace_id to a Loki label would multiply the index by per-request cardinality — the canonical way to wreck a Loki cluster. It stays in the log body, queried at query time:

{app="tomoda-api"} | json | trace_id="8a4f5e0e..."

The same syntax backs Grafana's tracesToLogsV2 link. Now that we're on Loki 3.x / schema v13, structured_metadata is available as a future home for trace_id / span_id / caller (cardinality-safe, indexed): add a stage.structured_metadata block in values-alloy.yaml. Query-time | json remains correct and is the current pattern.

Label strategy — what's safe and what's not

Field	Promote to label?	Why
namespace	yes (Alloy relabel does it automatically)	bounded by namespace count
app	yes (auto)	bounded by app count
level	yes	bounded enum (debug/info/warn/error/fatal/panic)
status (Traefik)	yes	bounded HTTP status codes
method	yes	bounded HTTP method set
trace_id	NO — query-time JSON parse	per-request, unbounded
span_id	NO — query-time JSON parse	per-request, unbounded
caller	NO — query-time JSON parse	many call sites; not useful as a label
user_id (if ever added)	NO — query-time JSON parse	per-user, unbounded
request_id	NO — query-time JSON parse	per-request

Rule of thumb: if a field has fewer than ~100 distinct values cluster-wide, label is OK. Anything per-request, per-user, or per-trace either gets queried via | json | <field>="..." or lifted into structured_metadata (available now on Loki 3.x / v13). It never becomes a label.

Trace-to-log jump

This is what makes the Tempo integration useful. From a Tempo span:

{namespace="tomoda"} | json | trace_id="8a4f5e0e9b1b9c1f1e1d1c1b1a191817"

is run automatically when you click "View logs" on a span. The tracesToLogsV2 config in monitoring/values.yaml wires this — see tempo.md. The | json stage is required because trace_id isn't a label (see retention sizing note above); it's pulled from the JSON body at query time.

Going the other direction, the Loki data source has a derivedFields rule:

- name: TraceID
  matcherRegex: '"trace_id":"(\w+)"'
  url: '${__value.raw}'
  datasourceUid: tempo

Any log line containing "trace_id":"..." gets a clickable link in Grafana that opens the matching trace in Tempo.

Debugging the pipeline

# Is Alloy running on every node and shipping?
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "loki.write\|error" | head -20

# Tail a recent tomoda log line and check the extracted labels
kubectl port-forward -n monitoring svc/loki 3100:3100
curl -s -G 'http://localhost:3100/loki/api/v1/labels' | jq

# Sample query for a known trace
curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={app="tomoda-api"} | trace_id="abc..."' \
  --data-urlencode 'start='$(date -d '1 hour ago' +%s)000000000 | jq '.data.result[0]'

If a tomoda log line shows up in Grafana but level is missing as a label, it means the JSON parse failed — usually because the line isn't JSON (e.g. a panic stack trace, or a third-party library logging with a different format). The pipeline doesn't drop those — they're just queryable by {app="tomoda-api"} without structured filtering.

Migrating loki-stack → Loki 3.x

The stack moved off the deprecated loki-stack meta-chart (Loki 2.9 + Promtail) to the loki chart (single-binary, Loki 3.x) + alloy. The delivery is GitOps — merging the PR is what applies it — but the schema boundary and the chart swap have gotchas worth knowing before and during the cutover.

What changes

	Before	After
Chart	loki-stack 2.10.x	loki 7.3.0 + alloy 1.11.x
Loki version	2.9.x	3.6.x
Index schema	v11 / boltdb-shipper	v13 / tsdb (v11 kept for old data)
Log shipper	Promtail (DaemonSet)	Alloy (DaemonSet)
Argo apps	loki	loki + alloy

Schema boundary is additive, not a rewrite. values.yaml keeps the v11/boltdb-shipper entry and adds a v13/tsdb entry with a near-future from: date. Old chunks stay readable under v11; everything after the cutover date is v13. Nothing rewrites existing GCS objects. With 7-day retention, the v11 tail de-references itself within a week, then only v13 remains.

The from: date must be in the future at merge time

A v13 schema entry whose from: is already in the past when Loki starts makes Loki expect tsdb index files for a period that only has boltdb-shipper data → query errors on that window. If the merge slips past the pinned date, bump the from: in values.yaml to the next day before merging.

allow_structured_metadata: false while a v11 period is active

Loki 3.x defaults allow_structured_metadata: true, but structured metadata / native OTLP require every active schema period to be v13/tsdb. While the additive v11/boltdb-shipper period is still live (before the cutover date), Loki refuses to start with a MULTIPLE CONFIG ERRORS validation failure unless limits_config.allow_structured_metadata: false is set. Flip it back to true only after the v11 tail has aged out (7 days past cutover) and the schema is v13-only — that's the same point at which trace_id/span_id can move into structured_metadata.

Cutover steps

1. Confirm the from: date in k8s/envs/platform/loki/values.yaml is still in the future. Bump it if the maintenance window slipped.
2. Merge the PR. Argo creates the alloy app and updates loki. The chart swap replaces the loki-stack StatefulSet — because the app name (loki) and Service (loki:3100) are unchanged, the Grafana datasource needs no edit.
3. Watch the roll. The old Promtail DaemonSet is pruned (Argo prune: true); the new Alloy DaemonSet schedules one pod per node. The Loki StatefulSet recreates on the new image.

   kubectl -n monitoring get pods -l app.kubernetes.io/name=loki -w
   kubectl -n monitoring get ds -l app.kubernetes.io/name=alloy
   

   If the loki StatefulSet is stuck on a port/volume rename the SSA can't do in place, the fix is the same as Tempo's — argocd app delete-resource loki --kind StatefulSet (GCS-backed, no PVC to lose) then --hard-refresh.
4. Verify GCS auth. Loki must impersonate the observability SA. A 403/PermissionDenied in the loki log means the WI annotation didn't land — check the ServiceAccount has iam.gke.io/gcp-service-account.
5. Verify ingestion end-to-end. In Grafana Explore, run {namespace="tomoda"} for a recent window — new lines confirm Alloy → Loki → v13 works. Then run a query spanning the cutover date to confirm old v11 chunks still read.

Rollback. Revert the PR. loki-stack comes back with Promtail; the v11 chunks written before cutover are still there. Any v13 chunks written during the window become unreadable by Loki 2.9 (schema it doesn't know) but aren't deleted — they re-read fine if you roll forward again. Keep the window short to minimise that gap.