Skip to content

Loki

Cluster-wide log aggregation. Grafana Alloy tails every pod's logs and ships them to Loki; Grafana queries Loki using its built-in Loki datasource.

Installed by k8s/envs/platform/loki/application.yaml (two Argo CD Applications: loki and alloy), configured by k8s/envs/platform/loki/values.yaml and values-alloy.yaml.

Charts and sources

Component Chart Version App version
Loki store loki 7.3.0 Loki 3.6.18 (pinned in values.yaml)
Log shipper alloy 1.13.1 Alloy v1.20.x

Both from https://grafana.github.io/helm-charts, both sync into the monitoring namespace (shared with kube-prometheus-stack).

Loki runs in single-binary mode (deploymentMode: SingleBinary, one StatefulSet, target=all): no read/write/backend split and no memcached tier. The SimpleScalable components and caches are explicitly disabled in values.yaml.

Object storage (R2)

Chunks and index live in the R2 bucket tomoda-observability, shared with Tempo (Loki's chunks at the bucket root and its index under index_, Tempo under tempo/). Credentials come from the monitoring/r2-observability-credentials Secret, expanded into the S3 config at start (-config.expand-env=true). A 10Gi local-path volume at /var/loki holds the WAL and compactor state, so a restart replays unflushed logs.

Schema

values.yaml carries a single v13/tsdb schema entry, so structured metadata is allowed. The tsdb path requires compactor.delete_request_store: s3 (set in values.yaml).

Alloy (log shipper)

Alloy runs as a DaemonSet (one pod per node), discovering pods on its node and tailing their logs (CRI decode is built in). It does not require workloads to opt in; every pod's stdout/stderr lands in Loki by default. The pipeline is authored in Alloy's config language in values-alloy.yaml; edit stages there:

  • Traefik access logs ({container="traefik"}): Traefik keeps query strings (see Traefik). A stage.replace first redacts the value of these params, case-insensitively: token, access_token, id_token, refresh_token, code, state, password, otp, key, api_key, apikey, client_secret, secret, sig, signature, identifier, nonce and every X-Amz-*. RequestPath lands as /ws/chats/abc?token=REDACTED. The pattern also matches \u0026, the JSON escape Traefik writes for &. Add a param there when the app starts sending a new secret in a URL. Then stage.json extracts Traefik fields and stage.labels promotes a bounded set (host, status, method, level, entrypoint). The Traefik Grafana dashboard's access-log tables query that label set.
  • Tomoda backend Zap JSON ({app=~"tomoda-(api|async)"}): promotes level only (bounded to {debug,info,warn,error,fatal,panic}). trace_id / span_id stay in the body and are queryable at query time via LogQL | json, so the Loki index doesn't explode on per-request cardinality. This backs trace→log navigation from Tempo.

Non-JSON logs still get ingested, just without the structured labels; query them by {namespace="…"} / {app="…"}.

Querying

Loki exposes http://loki:3100 in-cluster (the Grafana datasource in k8s/envs/platform/monitoring/values.yaml) and a ServiceMonitor (label release: monitoring) for Prometheus to scrape its own metrics.

Open Grafana at https://grafana.tomoda.life, Explore tab, pick the Loki datasource, and use LogQL:

{namespace="tomoda-prod"} |= "error"
{namespace="traefik-system", status=~"5.."}

Operational notes

  • R2-backed. Log volume is capped by the bucket lifecycle (14 days, infrastructure/cloudflare/r2.tf), not disk size. Loki's own retention_period stays at 7d so the query API behaves predictably and chunks are de-referenced before the bucket deletes them.
  • Single Loki across all environments. tomoda-dev and tomoda-prod ship to the same Loki; filter by namespace. There is no env label on logs.
  • Label strategy and LogQL recipes: Observability → Loki.