Loki: storage, retention, and the Alloy pipeline¶
The deployment is documented in kubernetes/system/loki.md. This page covers what's specific to our config:
- Storage backend is R2. Chunks + index live in the shared
tomoda-observabilitybucket alongside Tempo blocks. PVC is disabled. - Retention is pinned to 7 days Loki-side, 14 days bucket-side (lifecycle).
- The Alloy pipeline parses the tomoda backend's Zap JSON logs and surfaces
trace_idfor trace-to-log navigation from Grafana / Tempo.
Loki runs the loki chart (single-binary, Loki 3.x) with a single tsdb/v13 schema period, so structured metadata is enabled; log shipping runs on Grafana Alloy (a DaemonSet).
Storage: shared R2 bucket¶
Loki writes chunks and the tsdb index to tomoda-observability (infrastructure/cloudflare/r2.tf), the same bucket as Tempo, separated by prefix:
| Writer | Prefix | What lands here |
|---|---|---|
| Loki | bucket root (chunks), index_ (tsdb index) |
Chunks, index, compactor working files |
| Tempo | tempo/ (set explicitly in tempo values) |
Trace blocks |
Credentials come from the monitoring/r2-observability-credentials Secret (ExternalSecret over tomoda-r2-cluster-credentials), injected as R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY and expanded into the S3 config by -config.expand-env=true.
Retention math¶
| Knob | Value | Why |
|---|---|---|
loki.limits_config.retention_period |
168h (7d) |
Query-API retention. After this Loki refuses to query the chunks even if they still exist in the bucket. |
loki.compactor.retention_enabled |
true |
Without this, retention is best-effort only. |
loki.compactor.retention_delete_delay |
2h |
Soft window before old chunks are actually deleted by the compactor. |
loki.compactor.delete_request_store |
s3 |
Required by the v13/tsdb path. |
| R2 lifecycle on the bucket | 14 days → Delete | Bucket-side safety net. Loki normally drops chunks at 7d via the compactor; this 14d backstop catches any chunks the compactor misses. |
Estimated write rate (current cluster):
- ~10 backend / system pods continuously logging at ~5 lines/sec each
- ≈ 4.3 M lines/day raw
- ≈ 1.5 GB/day uncompressed
- ≈ 150 MB/day after Loki's gzip chunk compression
- 7 × 150 MB ≈ 1.05 GB of active chunk data in R2 at steady state
That sits inside R2's free 10 GB. The win is operational (no PVC to grow, snapshot, or recover) and architectural (Tempo and Loki share one storage primitive).
To change retention, bump both:
loki.limits_config.retention_periodinvalues.yaml(Loki query API)- The
observabilitylifecycle ininfrastructure/cloudflare/r2.tf(deletion floor)
together: Loki querying chunks the bucket already deleted returns confusing 5xx.
If you see write-rate spikes:
- Compactor is actually running.
kubectl logs -n monitoring -l app.kubernetes.io/name=loki | grep compactor. If it's silent for hours, retention isn't being enforced. - One pod isn't spamming. Run
topk(10, sum by (namespace, pod) (rate({__name__=~".+"}[5m])))in Grafana to find the loudest pod and either fix it or drop its logs at the Alloy level.
Alloy pipeline¶
The pipeline lives in k8s/envs/platform/loki/values-alloy.yaml as an Alloy River config (loki.process "pipeline"). Two stage.match blocks run in order:
Traefik¶
Selector: {container="traefik"}. Parses the JSON access log and promotes entryPointName, request_Host, status, method, level to labels. The Traefik Grafana dashboard's access-log tables depend on these labels.
Tomoda backend¶
Selector: {app=~"tomoda-(api|async)"}. The backend uses Zap with a JSON encoder, so each log line is shaped like:
{
"level": "info",
"ts": 1716480000,
"caller": "main.go:45",
"msg": "request handled",
"trace_id": "8a4f5e0e9b1b9c1f1e1d1c1b1a191817",
"span_id": "0123456789abcdef"
}
The pipeline:
- Parses the JSON.
- Promotes
levelto a label. Cardinality is bounded (debug, info, warn, error, fatal, panic), so this is safe and gives free filtering:{app="tomoda-api", level="error"}. trace_idandspan_idstay in the raw log body: not promoted to a label, not lifted into structured metadata. They're queryable at query time via LogQL'sjsonstage.
stage.match {
selector = "{app=~\"tomoda-(api|async)\"}"
stage.json {
expressions = { level = "level" }
}
stage.labels {
values = { level = "" }
}
}
trace_id stays out of the index
Promoting trace_id to a Loki label would multiply the index by per-request cardinality, the standard way to wreck a Loki cluster. It stays in the log body, queried at query time:
{app="tomoda-api"} | json | trace_id="8a4f5e0e..."
The same syntax backs Grafana's tracesToLogsV2 link. Now that we're on Loki 3.x / schema v13, structured_metadata is available as a future home for trace_id / span_id / caller (cardinality-safe, indexed): add a stage.structured_metadata block in values-alloy.yaml. Query-time | json remains correct and is the current pattern.
Label strategy: what's safe and what's not¶
| Field | Promote to label? | Why |
|---|---|---|
namespace |
yes (Alloy relabel does it automatically) | bounded by namespace count |
app |
yes (auto) | bounded by app count |
level |
yes | bounded enum (debug/info/warn/error/fatal/panic) |
status (Traefik) |
yes | bounded HTTP status codes |
method |
yes | bounded HTTP method set |
trace_id |
No: query-time JSON parse | per-request, unbounded |
span_id |
No: query-time JSON parse | per-request, unbounded |
caller |
No: query-time JSON parse | many call sites; not useful as a label |
user_id (if ever added) |
No: query-time JSON parse | per-user, unbounded |
request_id |
No: query-time JSON parse | per-request |
Rule of thumb: if a field has fewer than ~100 distinct values cluster-wide, label is OK. Anything per-request, per-user, or per-trace either gets queried via | json | <field>="..." or lifted into structured_metadata (available now on Loki 3.x / v13). It never becomes a label.
Trace-to-log jump¶
This is what makes the Tempo integration useful. From a Tempo span:
{namespace="tomoda-prod"} | json | trace_id="8a4f5e0e9b1b9c1f1e1d1c1b1a191817"
is run automatically when you click "View logs" on a span. The tracesToLogsV2 config in monitoring/values.yaml wires this; see Tempo. The | json stage is required because trace_id isn't a label (see retention sizing note above); it's pulled from the JSON body at query time.
Going the other direction, the Loki data source has a derivedFields rule:
- name: TraceID
matcherRegex: '"trace_id":"(\w+)"'
url: '${__value.raw}'
datasourceUid: tempo
Any log line containing "trace_id":"..." gets a clickable link in Grafana that opens the matching trace in Tempo.
Debugging the pipeline¶
# Is Alloy running on every node and shipping?
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "loki.write\|error" | head -20
# Tail a recent tomoda log line and check the extracted labels
kubectl port-forward -n monitoring svc/loki 3100:3100
curl -s -G 'http://localhost:3100/loki/api/v1/labels' | jq
# Sample query for a known trace
curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
--data-urlencode 'query={app="tomoda-api"} | trace_id="abc..."' \
--data-urlencode 'start='$(date -d '1 hour ago' +%s)000000000 | jq '.data.result[0]'
If a tomoda log line shows up in Grafana but level is missing as a label, the JSON parse failed, usually because the line isn't JSON (e.g. a panic stack trace, or a third-party library logging with a different format). The pipeline keeps those; they're queryable by {app="tomoda-api"} without structured filtering.