Observability Manual Setup¶
The accounts and values the observability stack needs that live outside Terraform: Sentry, the Cloudflare API token, the Discord webhook and the synthetic probe password. Everything else (tunnel, Access app and service token, Worker, Alertmanager routing) is in code. The secret containers exist already (infrastructure/gcp/secrets.tf); this page fills their values.
Sentry¶
Frontend crash reporting (@sentry/react-native).
- Sign in at sentry.io with a
@tomoda.lifeaccount so membership stays with the company. - Organization slug
tomoda-platforms-inc, projecttomoda-frontend(platform React Native). Both are defaults infrontend/app.config.js(overridable withSENTRY_ORG/SENTRY_PROJECT) and the org slug is in the Grafana data sourcejsonData.orgSlug. They're configuration, not secrets. - Copy the DSN (Project Settings → Client Keys).
-
Create two auth tokens (Settings → Account → Auth Tokens), each shown once. Separate tokens keep the write scope off the cluster and let them rotate independently:
Name Scopes Used by sourcemap-uploadproject:releases,org:readfrontend/scripts/release/build-with-secrets.shat native and web release buildsgrafana-readonlyevent:read,org:read,project:read,member:readGrafana's Sentry data source in monitoring -
Install the Discord integration (Project Settings → Integrations → Discord) for the Tomoda guild, then add an issue alert "When a new issue is created" → notify
#alerts-frontend. It's OAuth, so there's no webhook secret. Sentry groups events into issues, so one bug posts one message; if the channel gets noisy, use the per-issue notification rate limit, not an alert-rule limit (that drops new issues).
| Value | GCP SM secret | Consumer |
|---|---|---|
| DSN | tomoda-sentry-dsn |
Baked into the JS bundle at release build time |
sourcemap-upload token |
tomoda-sentry-auth-token |
Sentry CLI during release builds; never in the bundle |
grafana-readonly token |
tomoda-sentry-grafana-token |
ESO → monitoring/sentry-grafana-credentials (Sentry data source) |
Cloudflare API token¶
The token Terraform uses for the cloudflare stack, read by CI from tomoda-cloudflare-api-token. Create a custom token scoped to the tomoda.life zone and the Tomoda account, with the permissions listed in Cloudflare. It's separate from the external-dns token.
The cloudflared tunnel token is generated by Terraform. After the first apply, copy it into Secret Manager for the in-cluster connector:
cd infrastructure/cloudflare
terraform output -raw tunnel_token | gcloud secrets versions add \
tomoda-cloudflare-tunnel-token --project=development-485000 --data-file=-
Discord webhook¶
One webhook on #monitoring carries Alertmanager, the synthetic Worker and Cloudflare billing notifications (#alerts-frontend is Sentry's OAuth integration).
#monitoring→ Edit Channel → Integrations → Webhooks → New Webhook (nameTomodachi).- Copy the URL (
https://discord.com/api/webhooks/<id>/<token>) intotomoda-alert-webhook-primary.
The secret name is provider-neutral, so switching alert providers changes only the value.
Synthetic probe user¶
The synthetic Worker and the post-deploy smoke test log in as synthetic+probe@tomoda.life. The backend creates or updates that user at startup from SYNTHETIC_PROBE_PASSWORD (ExternalSecret over tomoda-synthetic-probe-password); it's flagged account_type = 'synthetic' and excluded from MFA. Never point the probes at a real account: each probe writes login history and would skew user metrics.
Setting the values¶
scripts/setup-gcp-secrets.sh sets any value passed as an env var and reports which secrets are still empty; it generates the probe password when unset:
TOMODA_SENTRY_DSN="..." \
TOMODA_SENTRY_AUTH_TOKEN="..." \
TOMODA_SENTRY_GRAFANA_TOKEN="..." \
TOMODA_CLOUDFLARE_API_TOKEN="..." \
TOMODA_ALERT_WEBHOOK_PRIMARY="..." \
./scripts/setup-gcp-secrets.sh
Or one at a time from stdin, so nothing lands in shell history:
gcloud secrets versions add tomoda-sentry-dsn --project=development-485000 --data-file=-
Check with task secrets:empty.
Rotation¶
ESO refreshes hourly; kubectl rollout restart the consumer to pick a value up immediately.
| Secret | Rotate | Then |
|---|---|---|
tomoda-cloudflare-api-token |
Annually or if leaked | Next Terraform run uses it |
tomoda-alert-webhook-primary |
If leaked or changing provider | Restart Alertmanager; re-run the cloudflare apply (gh workflow run terraform-apply.yml) so the Worker and notification destination pick it up |
tomoda-cloudflare-tunnel-token |
If leaked | Restart the cloudflared Deployment in monitoring |
tomoda-synthetic-probe-password |
Quarterly and on team turnover | Restart tomoda-api in tomoda-prod (re-seeds the hash) and re-run the cloudflare apply |
tomoda-worker-auth-secret |
If leaked | Re-run the cloudflare apply, then restart tomoda-api in both envs; Worker calls signed with the old key are treated as ordinary clients in between |
tomoda-sentry-dsn, tomoda-sentry-auth-token |
Annually or if leaked | The next release build pulls the new value |
tomoda-sentry-grafana-token |
Annually or if leaked | kubectl rollout restart deploy/monitoring-grafana -n monitoring |
Related¶
- Secrets management: secret-store architecture and the full inventory
- Alerting: Alertmanager routing, inhibition and throttling
- Synthetics: the Worker, tunnel and Access setup