Skip to content

Observability Manual Setup

The accounts and values the observability stack needs that live outside Terraform: Sentry, the Cloudflare API token, the Discord webhook and the synthetic probe password. Everything else (tunnel, Access app and service token, Worker, Alertmanager routing) is in code. The secret containers exist already (infrastructure/gcp/secrets.tf); this page fills their values.

Sentry

Frontend crash reporting (@sentry/react-native).

  1. Sign in at sentry.io with a @tomoda.life account so membership stays with the company.
  2. Organization slug tomoda-platforms-inc, project tomoda-frontend (platform React Native). Both are defaults in frontend/app.config.js (overridable with SENTRY_ORG / SENTRY_PROJECT) and the org slug is in the Grafana data source jsonData.orgSlug. They're configuration, not secrets.
  3. Copy the DSN (Project Settings → Client Keys).
  4. Create two auth tokens (Settings → Account → Auth Tokens), each shown once. Separate tokens keep the write scope off the cluster and let them rotate independently:

    Name Scopes Used by
    sourcemap-upload project:releases, org:read frontend/scripts/release/build-with-secrets.sh at native and web release builds
    grafana-readonly event:read, org:read, project:read, member:read Grafana's Sentry data source in monitoring
  5. Install the Discord integration (Project Settings → Integrations → Discord) for the Tomoda guild, then add an issue alert "When a new issue is created" → notify #alerts-frontend. It's OAuth, so there's no webhook secret. Sentry groups events into issues, so one bug posts one message; if the channel gets noisy, use the per-issue notification rate limit, not an alert-rule limit (that drops new issues).

Value GCP SM secret Consumer
DSN tomoda-sentry-dsn Baked into the JS bundle at release build time
sourcemap-upload token tomoda-sentry-auth-token Sentry CLI during release builds; never in the bundle
grafana-readonly token tomoda-sentry-grafana-token ESO → monitoring/sentry-grafana-credentials (Sentry data source)

Cloudflare API token

The token Terraform uses for the cloudflare stack, read by CI from tomoda-cloudflare-api-token. Create a custom token scoped to the tomoda.life zone and the Tomoda account, with the permissions listed in Cloudflare. It's separate from the external-dns token.

The cloudflared tunnel token is generated by Terraform. After the first apply, copy it into Secret Manager for the in-cluster connector:

cd infrastructure/cloudflare
terraform output -raw tunnel_token | gcloud secrets versions add \
  tomoda-cloudflare-tunnel-token --project=development-485000 --data-file=-

Discord webhook

One webhook on #monitoring carries Alertmanager, the synthetic Worker and Cloudflare billing notifications (#alerts-frontend is Sentry's OAuth integration).

  1. #monitoring → Edit Channel → Integrations → Webhooks → New Webhook (name Tomodachi).
  2. Copy the URL (https://discord.com/api/webhooks/<id>/<token>) into tomoda-alert-webhook-primary.

The secret name is provider-neutral, so switching alert providers changes only the value.

Synthetic probe user

The synthetic Worker and the post-deploy smoke test log in as synthetic+probe@tomoda.life. The backend creates or updates that user at startup from SYNTHETIC_PROBE_PASSWORD (ExternalSecret over tomoda-synthetic-probe-password); it's flagged account_type = 'synthetic' and excluded from MFA. Never point the probes at a real account: each probe writes login history and would skew user metrics.

Setting the values

scripts/setup-gcp-secrets.sh sets any value passed as an env var and reports which secrets are still empty; it generates the probe password when unset:

TOMODA_SENTRY_DSN="..." \
TOMODA_SENTRY_AUTH_TOKEN="..." \
TOMODA_SENTRY_GRAFANA_TOKEN="..." \
TOMODA_CLOUDFLARE_API_TOKEN="..." \
TOMODA_ALERT_WEBHOOK_PRIMARY="..." \
./scripts/setup-gcp-secrets.sh

Or one at a time from stdin, so nothing lands in shell history:

gcloud secrets versions add tomoda-sentry-dsn --project=development-485000 --data-file=-

Check with task secrets:empty.

Rotation

ESO refreshes hourly; kubectl rollout restart the consumer to pick a value up immediately.

Secret Rotate Then
tomoda-cloudflare-api-token Annually or if leaked Next Terraform run uses it
tomoda-alert-webhook-primary If leaked or changing provider Restart Alertmanager; re-run the cloudflare apply (gh workflow run terraform-apply.yml) so the Worker and notification destination pick it up
tomoda-cloudflare-tunnel-token If leaked Restart the cloudflared Deployment in monitoring
tomoda-synthetic-probe-password Quarterly and on team turnover Restart tomoda-api in tomoda-prod (re-seeds the hash) and re-run the cloudflare apply
tomoda-worker-auth-secret If leaked Re-run the cloudflare apply, then restart tomoda-api in both envs; Worker calls signed with the old key are treated as ordinary clients in between
tomoda-sentry-dsn, tomoda-sentry-auth-token Annually or if leaked The next release build pulls the new value
tomoda-sentry-grafana-token Annually or if leaked kubectl rollout restart deploy/monitoring-grafana -n monitoring