Loki¶
Cluster-wide log aggregation. Grafana Alloy tails every pod's logs and ships them to Loki; Grafana queries Loki using its built-in Loki datasource.
Installed by k8s/envs/platform/loki/application.yaml (two Argo CD Applications: loki and alloy), configured by k8s/envs/platform/loki/values.yaml and values-alloy.yaml.
Charts and sources¶
| Component | Chart | Version | App version |
|---|---|---|---|
| Loki store | loki |
7.3.0 |
Loki 3.6.18 (pinned in values.yaml) |
| Log shipper | alloy |
1.13.1 |
Alloy v1.20.x |
Both from https://grafana.github.io/helm-charts, both sync into the monitoring namespace (shared with kube-prometheus-stack).
Loki runs in single-binary mode (deploymentMode: SingleBinary, one StatefulSet, target=all): no read/write/backend split and no memcached tier. The SimpleScalable components and caches are explicitly disabled in values.yaml.
Object storage (R2)¶
Chunks and index live in the R2 bucket tomoda-observability, shared with Tempo (Loki's chunks at the bucket root and its index under index_, Tempo under tempo/). Credentials come from the monitoring/r2-observability-credentials Secret, expanded into the S3 config at start (-config.expand-env=true). A 10Gi local-path volume at /var/loki holds the WAL and compactor state, so a restart replays unflushed logs.
Schema¶
values.yaml carries a single v13/tsdb schema entry, so structured metadata is allowed. The tsdb path requires compactor.delete_request_store: s3 (set in values.yaml).
Alloy (log shipper)¶
Alloy runs as a DaemonSet (one pod per node), discovering pods on its node and tailing their logs (CRI decode is built in). It does not require workloads to opt in; every pod's stdout/stderr lands in Loki by default. The pipeline is authored in Alloy's config language in values-alloy.yaml; edit stages there:
- Traefik access logs (
{container="traefik"}): Traefik keeps query strings (see Traefik). Astage.replacefirst redacts the value of these params, case-insensitively:token,access_token,id_token,refresh_token,code,state,password,otp,key,api_key,apikey,client_secret,secret,sig,signature,identifier,nonceand everyX-Amz-*.RequestPathlands as/ws/chats/abc?token=REDACTED. The pattern also matches\u0026, the JSON escape Traefik writes for&. Add a param there when the app starts sending a new secret in a URL. Thenstage.jsonextracts Traefik fields andstage.labelspromotes a bounded set (host, status, method, level, entrypoint). The Traefik Grafana dashboard's access-log tables query that label set. - Tomoda backend Zap JSON (
{app=~"tomoda-(api|async)"}): promoteslevelonly (bounded to{debug,info,warn,error,fatal,panic}).trace_id/span_idstay in the body and are queryable at query time via LogQL| json, so the Loki index doesn't explode on per-request cardinality. This backs trace→log navigation from Tempo.
Non-JSON logs still get ingested, just without the structured labels; query them by {namespace="…"} / {app="…"}.
Querying¶
Loki exposes http://loki:3100 in-cluster (the Grafana datasource in k8s/envs/platform/monitoring/values.yaml) and a ServiceMonitor (label release: monitoring) for Prometheus to scrape its own metrics.
Open Grafana at https://grafana.tomoda.life, Explore tab, pick the Loki datasource, and use LogQL:
{namespace="tomoda-prod"} |= "error"
{namespace="traefik-system", status=~"5.."}
Operational notes¶
- R2-backed. Log volume is capped by the bucket lifecycle (14 days,
infrastructure/cloudflare/r2.tf), not disk size. Loki's ownretention_periodstays at 7d so the query API behaves predictably and chunks are de-referenced before the bucket deletes them. - Single Loki across all environments.
tomoda-devandtomoda-prodship to the same Loki; filter bynamespace. There is noenvlabel on logs. - Label strategy and LogQL recipes: Observability → Loki.