Skip to content

Sentry in Grafana

Frontend metrics from Sentry surfaced inside Grafana via the grafana-sentry-datasource plugin. Gives operators a single place to look at backend health (Prometheus, Loki, Tempo) and frontend health (Sentry-sourced issue counts, crash-free %, transaction performance) — same dashboards, same alerting surface.

Why this pairing

The frontend Sentry SDK in tomoda-labs/tomoda emits:

  • Errors — issue counts, new issues per release, crash-free users / sessions
  • Performance traces — transaction durations (page loads, screen renders, fetch calls), sampled at 10%
  • Sessions — release-health rollups

Sentry's UI is good for triaging individual issues. But for "what's the overall health of the frontend right now" — and especially for correlating a frontend issue spike with a backend rollout — having the same time-range slider drive both Prometheus panels and Sentry panels is the right ergonomics. That's what the data source plugin enables.

Architecture

Sentry HTTP API                                  GCP Secret Manager
(api.sentry.io)                                   ┌─────────────────────────────────────┐
       ▲                                          │ tomoda-sentry-grafana-token         │
       │ HTTPS, query for                         │ (scopes: event:read, org:read,      │
       │ issues / metrics / events                │  project:read, member:read)         │
       │ Authorization: Bearer <token>            └─────────────────────────────────────┘
       │                                                              │
┌──────┴──────────────────────────────┐                                │ ExternalSecret
│ Grafana                              │◄──── extraEnvs / secretKeyRef │ refresh: 1h
│ (kube-prometheus-stack)              │                                │
│                                      │      ┌─────────────────────────▼──────────────┐
│  Plugin: grafana-sentry-datasource   │      │ K8s Secret in monitoring/              │
│  Datasource: type=grafana-sentry-... │      │ sentry-grafana-credentials             │
│  secureJsonData.authToken=$ENV       │──────│   key: auth-token                      │
└──────────────────────────────────────┘      └────────────────────────────────────────┘

Configuration

Helm values

In k8s/envs/dev/sys/monitoring/values.yaml, three additions:

  1. grafana.plugins: [grafana-sentry-datasource] — Grafana auto-installs the plugin on chart upgrade.
  2. grafana.extraEnvs — injects SENTRY_AUTH_TOKEN into the Grafana pod from the ESO-projected K8s Secret.
  3. grafana.additionalDataSources — registers the Sentry data source with orgSlug: tomoda-platforms-inc and authToken: $SENTRY_AUTH_TOKEN (env interpolation happens at Grafana provisioner runtime).

Secret projection

k8s/envs/dev/sys/manifests/external-secrets-config.yaml gains a new ExternalSecret named sentry-grafana-secret in the monitoring namespace. It projects GCP SM tomoda-sentry-grafana-token into a K8s Secret named sentry-grafana-credentials with key auth-token. ESO refreshes every 1 hour.

Verification

# 1. The K8s Secret exists in monitoring/
kubectl get secret sentry-grafana-credentials -n monitoring
#   NAME                          TYPE     DATA   AGE
#   sentry-grafana-credentials    Opaque   1      ...

# 2. Grafana pod has the env var
kubectl set env -n monitoring deploy/monitoring-grafana --list | grep SENTRY
#   SENTRY_AUTH_TOKEN=<from K8s Secret>

# 3. Plugin is installed
kubectl exec -n monitoring deploy/monitoring-grafana -c grafana -- ls /var/lib/grafana/plugins/
#   grafana-sentry-datasource

# 4. Data source returns 200 from Sentry
#    Grafana UI -> Connections -> Data sources -> Sentry -> "Save & Test"
#    Should show: Data source is working

If "Save & Test" fails:

  • 401 Unauthorized — token doesn't have read scopes or expired. Check Sentry → User Settings → Auth Tokens, regenerate with the required scopes, push the new value into GCP SM with gcloud secrets versions add tomoda-sentry-grafana-token --data-file=-, then kubectl rollout restart deploy/monitoring-grafana -n monitoring (or wait 1h for ESO + the pod to pick it up on its own).
  • 404 Not Found — orgSlug is wrong. Check the slug in Sentry → Settings → Organization (top-left of the URL when you're on the org page).
  • Plugin not loaded — chart didn't pick up the plugins block. kubectl logs deploy/monitoring-grafana -n monitoring -c grafana | grep -i plugin to see install logs.

Querying

Inside Grafana, create a new panel → select Sentry data source → choose a query type:

Query type What it returns Useful for
Issues Issue counts grouped by various dimensions (release, environment, level) "How many new issues this hour?" panels
Events Raw event counts High-cardinality scatter plots
Statistics Org/project rollups "Crash-free sessions over time" gauge

Combine with backend Prometheus / Loki panels on the same dashboard to see, e.g.:

  • Backend deploy spike (timestamp) ↔ frontend issue rate (Sentry) ↔ API 5xx rate (Prometheus)
  • Native crash-free % (Sentry) ↔ backend p95 latency (Prometheus) — answers "did the backend latency cause client-side timeouts that surfaced as crashes?"

Rotation

The read token rotates independently of the source-map upload token (tomoda-sentry-auth-token):

# 1. In Sentry UI, create a new token with the same scopes
#    (event:read, org:read, project:read, member:read)
# 2. Push to GCP SM as a new version
echo -n "<new-token>" | gcloud secrets versions add tomoda-sentry-grafana-token \
  --project=development-485000 --data-file=-
# 3. ESO syncs within 1h; force-restart to verify immediately
kubectl rollout restart deploy/monitoring-grafana -n monitoring
# 4. Revoke the old Sentry token via the UI