Skip to content

Tempo

Distributed-tracing backend. The tomoda backend emits OpenTelemetry spans (Go OTel SDK + OTLP gRPC exporter), Tempo stores them in R2, and Grafana queries Tempo as a data source. Spans link back to Loki logs via trace_id.

Installed by k8s/envs/platform/tempo/application.yaml, configured by k8s/envs/platform/tempo/values.yaml. The bucket comes from infrastructure/cloudflare/r2.tf; credentials from the monitoring/r2-observability-credentials ExternalSecret.

Chart and source

Field Value
Helm chart tempo (monolithic single-binary mode)
Repository https://grafana.github.io/helm-charts
Version 1.24.4 (Tempo 2.9.5, pinned in values.yaml)
Destination namespace monitoring (shared with kube-prometheus-stack and Loki)
Argo CD Application tempo

We picked the monolithic chart over tempo-distributed deliberately: trace volume is low (10% sampling on a small backend), and a single replica with object storage is plenty. If write throughput grows past a few thousand spans/sec, switch to tempo-distributed (distributors + ingesters + queriers split).

Storage: shared R2 bucket

Tempo stores trace blocks in tomoda-observability, shared with Loki: Tempo writes under prefix tempo/, Loki at the bucket root (index under index_). The bucket's 14-day lifecycle rule deletes old objects across both prefixes; Tempo's compactor block_retention is 336h so the index never references deleted blocks.

tempo:
  extraArgs:
    config.expand-env: true
  extraEnvFrom:
    - secretRef:
        name: r2-observability-credentials
  storage:
    trace:
      backend: s3
      s3:
        bucket: tomoda-observability
        prefix: tempo
        endpoint: <account>.r2.cloudflarestorage.com
        region: auto
        access_key: ${R2_ACCESS_KEY_ID}
        secret_key: ${R2_SECRET_ACCESS_KEY}
        forcepathstyle: true

See loki.md for the Loki side of the same bucket and credentials.

Ingestion: OTLP gRPC

Tempo's receivers config exposes OTLP on both gRPC (:4317) and HTTP (:4318). The tomoda backend uses gRPC:

OBSERVABILITY_OTLP_ENDPOINT=tempo.monitoring.svc.cluster.local:4317
OBSERVABILITY_OTLP_INSECURE=true
OBSERVABILITY_SAMPLE_RATE=0.1

Apps in any namespace can reach Tempo via that FQDN. Plain gRPC without mTLS is fine inside the cluster: NetworkPolicies and the trust boundary already restrict who can connect.

The sampling rate is set on the application side (head sampling: the Go SDK decides per trace). Tempo itself doesn't sub-sample.

Grafana integration

The Tempo data source is added in k8s/envs/platform/monitoring/values.yaml alongside Loki:

- name: Tempo
  type: tempo
  uid: tempo
  url: http://tempo:3200
  jsonData:
    tracesToLogsV2:
      datasourceUid: loki
      filterByTraceID: true
      query: '{namespace="$${__span.tags["k8s.namespace.name"]}"} | trace_id="$${__span.traceId}"'

Two cross-data-source jumps are wired up:

  • Trace → logs: click a span in the Tempo UI, jump to a Loki query filtered to the same trace_id in a ±5min window. Uses LogQL's | json stage to extract trace_id from the JSON log body at query time (see loki.md).
  • Logs → trace: the Loki data source has a derivedFields rule that matches "trace_id":"..." in JSON log lines and renders it as a clickable Tempo link.

Operational notes

  • Single replica. Monolithic mode means one Tempo pod handles ingest + query + compaction. Restarts cause brief ingest gaps (OTel exporters retry, but the SDK's queue is bounded, so bursts during a restart can drop spans). Acceptable at current volume; if you see drops in app metrics (otelcol_exporter_send_failed_spans_total), bump replicas or switch chart.
  • R2 is the source of truth. The 5Gi PVC holds only the WAL; trace data lives in R2. PVC loss is recoverable; bucket loss is not.
  • No multi-tenant mode. multitenancy_enabled: false: all spans share one tenant ID (single-tenant).
  • 14-day retention. Driven by the R2 lifecycle rule. Bump the lifecycle in r2.tf + Tempo's block_retention together if longer trace history is needed.

Debugging

# Verify Tempo is healthy
kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo

# Check it's accepting traces (look for "received OTLP traces" lines)
kubectl logs -n monitoring deploy/tempo --tail=200 | grep -i otlp

# Credentials synced?
kubectl get externalsecret r2-observability-credentials -n monitoring

# List recent trace blocks in R2 (rclone remote "r2")
rclone lsl r2:tomoda-observability/tempo/single-tenant/ | tail -20

# Tempo's /ready endpoint
kubectl port-forward -n monitoring svc/tempo 3200:3200
curl localhost:3200/ready

If Tempo logs AccessDenied / InvalidAccessKeyId on flush, the R2 token in tomoda-r2-cluster-credentials is wrong or lacks write on tomoda-observability.