Tempo¶
Distributed-tracing backend. The tomoda backend emits OpenTelemetry spans (Go OTel SDK + OTLP gRPC exporter), Tempo stores them in R2, and Grafana queries Tempo as a data source. Spans link back to Loki logs via trace_id.
Installed by k8s/envs/platform/tempo/application.yaml, configured by k8s/envs/platform/tempo/values.yaml. The bucket comes from infrastructure/cloudflare/r2.tf; credentials from the monitoring/r2-observability-credentials ExternalSecret.
Chart and source¶
| Field | Value |
|---|---|
| Helm chart | tempo (monolithic single-binary mode) |
| Repository | https://grafana.github.io/helm-charts |
| Version | 1.24.4 (Tempo 2.9.5, pinned in values.yaml) |
| Destination namespace | monitoring (shared with kube-prometheus-stack and Loki) |
| Argo CD Application | tempo |
We picked the monolithic chart over tempo-distributed deliberately: trace volume is low (10% sampling on a small backend), and a single replica with object storage is plenty. If write throughput grows past a few thousand spans/sec, switch to tempo-distributed (distributors + ingesters + queriers split).
Storage: shared R2 bucket¶
Tempo stores trace blocks in tomoda-observability, shared with Loki: Tempo writes under prefix tempo/, Loki at the bucket root (index under index_). The bucket's 14-day lifecycle rule deletes old objects across both prefixes; Tempo's compactor block_retention is 336h so the index never references deleted blocks.
tempo:
extraArgs:
config.expand-env: true
extraEnvFrom:
- secretRef:
name: r2-observability-credentials
storage:
trace:
backend: s3
s3:
bucket: tomoda-observability
prefix: tempo
endpoint: <account>.r2.cloudflarestorage.com
region: auto
access_key: ${R2_ACCESS_KEY_ID}
secret_key: ${R2_SECRET_ACCESS_KEY}
forcepathstyle: true
See loki.md for the Loki side of the same bucket and credentials.
Ingestion: OTLP gRPC¶
Tempo's receivers config exposes OTLP on both gRPC (:4317) and HTTP (:4318). The tomoda backend uses gRPC:
OBSERVABILITY_OTLP_ENDPOINT=tempo.monitoring.svc.cluster.local:4317
OBSERVABILITY_OTLP_INSECURE=true
OBSERVABILITY_SAMPLE_RATE=0.1
Apps in any namespace can reach Tempo via that FQDN. Plain gRPC without mTLS is fine inside the cluster: NetworkPolicies and the trust boundary already restrict who can connect.
The sampling rate is set on the application side (head sampling: the Go SDK decides per trace). Tempo itself doesn't sub-sample.
Grafana integration¶
The Tempo data source is added in k8s/envs/platform/monitoring/values.yaml alongside Loki:
- name: Tempo
type: tempo
uid: tempo
url: http://tempo:3200
jsonData:
tracesToLogsV2:
datasourceUid: loki
filterByTraceID: true
query: '{namespace="$${__span.tags["k8s.namespace.name"]}"} | trace_id="$${__span.traceId}"'
Two cross-data-source jumps are wired up:
- Trace → logs: click a span in the Tempo UI, jump to a Loki query filtered to the same
trace_idin a ±5min window. Uses LogQL's| jsonstage to extracttrace_idfrom the JSON log body at query time (see loki.md). - Logs → trace: the Loki data source has a
derivedFieldsrule that matches"trace_id":"..."in JSON log lines and renders it as a clickable Tempo link.
Operational notes¶
- Single replica. Monolithic mode means one Tempo pod handles ingest + query + compaction. Restarts cause brief ingest gaps (OTel exporters retry, but the SDK's queue is bounded, so bursts during a restart can drop spans). Acceptable at current volume; if you see drops in app metrics (
otelcol_exporter_send_failed_spans_total), bump replicas or switch chart. - R2 is the source of truth. The 5Gi PVC holds only the WAL; trace data lives in R2. PVC loss is recoverable; bucket loss is not.
- No multi-tenant mode.
multitenancy_enabled: false: all spans share one tenant ID (single-tenant). - 14-day retention. Driven by the R2 lifecycle rule. Bump the lifecycle in
r2.tf+ Tempo'sblock_retentiontogether if longer trace history is needed.
Debugging¶
# Verify Tempo is healthy
kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
# Check it's accepting traces (look for "received OTLP traces" lines)
kubectl logs -n monitoring deploy/tempo --tail=200 | grep -i otlp
# Credentials synced?
kubectl get externalsecret r2-observability-credentials -n monitoring
# List recent trace blocks in R2 (rclone remote "r2")
rclone lsl r2:tomoda-observability/tempo/single-tenant/ | tail -20
# Tempo's /ready endpoint
kubectl port-forward -n monitoring svc/tempo 3200:3200
curl localhost:3200/ready
If Tempo logs AccessDenied / InvalidAccessKeyId on flush, the R2 token in tomoda-r2-cluster-credentials is wrong or lacks write on tomoda-observability.