Skip to content

Loki

Cluster-wide log aggregation. Grafana Alloy tails every pod's logs and ships them to Loki; Grafana queries Loki using its built-in Loki datasource. There is no Elasticsearch, no Cloud Logging sink, no Fluent Bit — just Grafana's loki chart plus alloy.

Installed by k8s/envs/platform/loki/application.yaml (two Argo CD Applications: loki and alloy), configured by k8s/envs/platform/loki/values.yaml and values-alloy.yaml.

Charts and sources

Component Chart Version App version
Loki store loki 7.3.0 Loki 3.6.x
Log shipper alloy 1.11.1 Alloy v1.18.x

Both from https://grafana.github.io/helm-charts, both sync into the monitoring namespace (shared with kube-prometheus-stack).

Loki runs in single-binary mode (deploymentMode: SingleBinary, one StatefulSet, target=all) — no read/write/backend split and no memcached tier, matching the previous monolithic footprint. The SimpleScalable components and caches are explicitly disabled in values.yaml.

Object storage (GCS)

Chunks and index live in the GCS bucket tomoda-observability-development-485000, shared with Tempo (Loki under the loki/ object prefix, Tempo under tempo/). Loki authenticates via Workload Identity: the KSA monitoring/loki impersonates the observability@development-485000 GCP service account (provisioned in infrastructure/gcp/tempo.tf) — no static key. There is no PVC of record; the StatefulSet mounts only a small emptyDir for the WAL and the active tsdb index cache.

Schema

values.yaml carries two schema entries: the legacy v11/boltdb-shipper block (so chunks written by the old Loki 2.9 stay readable) and a v13/tsdb block that takes over at the cutover date. Retention is 7 days, so the v11 tail de-references itself within a week of cutover and the old boltdb index ages out on its own. The v13/tsdb path requires compactor.delete_request_store: gcs (set in values.yaml), which was implicit under v11.

Alloy (log shipper)

Alloy runs as a DaemonSet — one pod per node — discovering pods on its node and tailing their logs (CRI decode is built in). It does not require workloads to opt in; every pod's stdout/stderr lands in Loki by default. The pipeline is authored in Alloy's config language in values-alloy.yaml and reproduces the previous Promtail stages:

  • Traefik access logs ({container="traefik"}): a stage.replace redacts WebSocket-handshake JWTs (?token=… / &token=…) before any downstream stage sees the line, then stage.json extracts Traefik fields and stage.labels promotes a bounded set (host, status, method, level, entrypoint). That label set is what makes the Traefik logs Grafana dashboard (gnetId: 13702) work out of the box. RequestPath lands as /ws/chats/abc?token=REDACTED. The unredacted line still lives in the pod's stdout until Alloy picks it up (sub-second window).
  • Tomoda backend Zap JSON ({app=~"tomoda-(api|async)"}): promotes level only (bounded to {debug,info,warn,error,fatal,panic}). trace_id / span_id stay in the body and are queryable at query time via LogQL | json, so the Loki index doesn't explode on per-request cardinality — this is what backs trace→log navigation from Tempo.

Non-JSON logs still get ingested, just without the structured labels; query them by {namespace="…"} / {app="…"}.

Querying

Loki exposes http://loki:3100 in-cluster (the port is unchanged from the loki-stack days, so the Grafana datasource in k8s/envs/platform/monitoring/values.yaml needs no edit) and a ServiceMonitor (label release: monitoring) for Prometheus to scrape its own metrics.

Open Grafana at https://grafana.tomoda.life, Explore tab, pick the Loki datasource, and use LogQL:

{namespace="tomoda"} |= "error"
{namespace="traefik-system", status=~"5.."}

Operational notes

  • GCS-backed, no PVC. Log volume is capped by bucket lifecycle (bucket-side, set in Terraform), not disk size. Loki's own retention_period stays at 7d so the query API behaves predictably and chunks are de-referenced before GCS deletes them.
  • Single Loki across all environments. Both the tomoda (dev) and tomoda-prod namespaces ship to the same Loki. Filter by namespace for prod-only traffic.
  • Alloy replaced Promtail. Promtail (bundled with the old loki-stack meta-chart) is EOL. The River pipeline in values-alloy.yaml is the equivalent; edit stages there.
  • Migrating from loki-stack. See operations/observability/loki.md for the schema-boundary cutover runbook and the label strategy.