Skip to content

Sentry in Grafana

Frontend metrics from Sentry surfaced inside Grafana via the grafana-sentry-datasource plugin. Gives operators a single place to look at backend health (Prometheus, Loki, Tempo) and frontend health (Sentry issue counts, crash-free %, transaction performance) on the same dashboards.

Why this pairing

The frontend Sentry SDK in tomoda-labs/tomoda emits:

  • Errors: issue counts, new issues per release, crash-free users / sessions
  • Performance traces: transaction durations (page loads, screen renders, fetch calls), sampled at 10%
  • Sessions: release-health rollups

Sentry's UI triages individual issues. The data source answers "how healthy is the frontend right now" and correlates a frontend issue spike with a backend rollout, with one time range driving both Prometheus and Sentry panels.

Architecture

Sentry HTTP API                                  GCP Secret Manager
(api.sentry.io)                                   ┌─────────────────────────────────────┐
       ▲                                          │ tomoda-sentry-grafana-token         │
       │ HTTPS, query for                         │ (scopes: event:read, org:read,      │
       │ issues / metrics / events                │  project:read, member:read)         │
       │ Authorization: Bearer <token>            └─────────────────────────────────────┘
       │                                                              │
┌──────┴──────────────────────────────┐                                │ ExternalSecret
│ Grafana                              │◄──── extraEnvs / secretKeyRef │ refresh: 1h
│ (kube-prometheus-stack)              │                                │
│                                      │      ┌─────────────────────────▼──────────────┐
│  Plugin: grafana-sentry-datasource   │      │ K8s Secret in monitoring/              │
│  Datasource: type=grafana-sentry-... │      │ sentry-grafana-credentials             │
│  secureJsonData.authToken=$ENV       │──────│   key: auth-token                      │
└──────────────────────────────────────┘      └────────────────────────────────────────┘

Configuration

Helm values

In k8s/envs/platform/monitoring/values.yaml, three additions:

  1. grafana.plugins: [grafana-sentry-datasource]: Grafana auto-installs the plugin on chart upgrade.
  2. grafana.extraEnvs: injects SENTRY_AUTH_TOKEN into the Grafana pod from the ESO-projected K8s Secret.
  3. grafana.additionalDataSources: registers the Sentry data source with orgSlug: tomoda-platforms-inc and authToken: $SENTRY_AUTH_TOKEN (env interpolation happens at Grafana provisioner runtime).

Secret projection

k8s/envs/platform/manifests/external-secrets-config.yaml gains a new ExternalSecret named sentry-grafana-secret in the monitoring namespace. It projects GCP SM tomoda-sentry-grafana-token into a K8s Secret named sentry-grafana-credentials with key auth-token. ESO refreshes every 1 hour.

Verification

# 1. The K8s Secret exists in monitoring/
kubectl get secret sentry-grafana-credentials -n monitoring
#   NAME                          TYPE     DATA   AGE
#   sentry-grafana-credentials    Opaque   1      ...

# 2. Grafana pod has the env var
kubectl set env -n monitoring deploy/monitoring-grafana --list | grep SENTRY
#   SENTRY_AUTH_TOKEN=<from K8s Secret>

# 3. Plugin is installed
kubectl exec -n monitoring deploy/monitoring-grafana -c grafana -- ls /var/lib/grafana/plugins/
#   grafana-sentry-datasource

# 4. Data source returns 200 from Sentry
#    Grafana UI -> Connections -> Data sources -> Sentry -> "Save & Test"
#    Should show: Data source is working

If "Save & Test" fails:

  • 401 Unauthorized: token doesn't have read scopes or expired. Check Sentry → User Settings → Auth Tokens, regenerate with the required scopes, push the new value into GCP SM with gcloud secrets versions add tomoda-sentry-grafana-token --data-file=-, then kubectl rollout restart deploy/monitoring-grafana -n monitoring (or wait 1h for ESO + the pod to pick it up on its own).
  • 404 Not Found: orgSlug is wrong. Check the slug in Sentry → Settings → Organization (top-left of the URL when you're on the org page).
  • Plugin not loaded: chart didn't pick up the plugins block. kubectl logs deploy/monitoring-grafana -n monitoring -c grafana | grep -i plugin to see install logs.

Querying

Inside Grafana, create a new panel → select Sentry data source → choose a query type:

Query type What it returns Useful for
Issues Issue counts grouped by various dimensions (release, environment, level) "How many new issues this hour?" panels
Events Raw event counts High-cardinality scatter plots
Statistics Org/project rollups "Crash-free sessions over time" gauge

Combine with backend Prometheus / Loki panels on the same dashboard to see, e.g.:

  • Backend deploy spike (timestamp) ↔ frontend issue rate (Sentry) ↔ API 5xx rate (Prometheus)
  • Native crash-free % (Sentry) ↔ backend p95 latency (Prometheus): answers "did the backend latency cause client-side timeouts that surfaced as crashes?"

Rotation

The read token rotates independently of the source-map upload token (tomoda-sentry-auth-token):

# 1. In Sentry UI, create a new token with the same scopes
#    (event:read, org:read, project:read, member:read)
# 2. Copy the token, then push it to GCP SM as a new version
pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-sentry-grafana-token \
  --project=development-485000 --data-file=-
pbcopy </dev/null
# 3. ESO syncs within 1h; force-restart to verify immediately
kubectl rollout restart deploy/monitoring-grafana -n monitoring
# 4. Revoke the old Sentry token via the UI