Secrets Management¶
How application and platform secrets are stored, synced, and rotated.
Overview¶
One external secret store, GCP Secret Manager, bridged into Kubernetes by the External Secrets Operator (ESO) through the gsm-tomoda ClusterSecretStore:
GCP Secret Manager --> ClusterSecretStore (gsm-tomoda) --> backend-secrets-{dev|prod}, object-storage-credentials,
r2-*-credentials, platform secrets
ESO authenticates with a key for the eso-reader service account, stored as the cluster's only hand-created Secret (external-secrets/gcpsm-credentials). No secret value is stored in Git. ESO refreshes every hour.
Inventory¶
Every secret in project development-485000. The containers are Terraform (infrastructure/gcp/secrets.tf); add a new one there first (Adding a secret). "Auto" values come from scripts/setup-gcp-secrets.sh; everything else is set by hand with gcloud secrets versions add.
| GCP SM key | Value | Consumer |
|---|---|---|
| Backend | ||
tomoda-db-password-{dev,prod} |
auto | Backend + CNPG owner password |
tomoda-redis-password-{dev,prod} |
auto | Backend + Redis |
tomoda-jwt-secret-{dev,prod} |
auto | Backend token signing |
tomoda-asset-signing-key-{dev,prod} |
auto | Backend media URL signing + assets Workers (Terraform reads it) |
tomoda-r2-app-{dev,prod}-credentials |
R2 token JSON | Backend media uploads |
tomoda-nats-password-{dev,prod} |
openssl rand -hex 32 |
NATS server (nats-bus-auth) |
tomoda-nats-url-{dev,prod} |
nats://tomoda-<env>:<password>@nats-bus.platform.svc.cluster.local:4222 |
Backend NATS_URL (backend-nats) |
tomoda-google-client-id, -client-secret, -ios-client-id, -android-client-id |
Google Cloud Console | Backend sign-in; the web client is also the SSO client for Argo CD Dex, oauth2-proxy and Grafana |
tomoda-google-places-api-key |
Google Cloud Console | Backend places lookups |
tomoda-apple-{client-id,service-id,team-id,key-id,auth-key-p8} |
Apple Developer | Backend sign-in + push |
tomoda-stripe-secret-key, -webhook-secret |
Stripe | Backend payments |
tomoda-email-api-key |
email provider | Backend email |
tomoda-klipy-api-key, tomoda-deepseek-api-key, tomoda-serper-api-key |
vendor dashboards | Backend |
| Cluster and platform | ||
tomoda-r2-cluster-credentials |
R2 token JSON | CNPG backups, k3s etcd snapshots, Loki, Tempo, OTA (bundles + DB dumps), seed loader |
tomoda-ghcr-read-credentials |
<github-user>:<classic PAT> (username prefix required) |
Node image pulls (bootstrap-node.sh), Image Updater, Trivy (ghcr-pull) |
tomoda-github-app-{id,installation-id,private-key} |
tomoda-ci-automation GitHub App |
Argo CD repo access, ARC runners, trivy-issue-reporter |
tomoda-github-terraform-app-private-key |
tomoda-terraform GitHub App |
Terraform CI (infrastructure/github) |
tomoda-oauth2-proxy-cookie-secret |
openssl rand -base64 32 |
oauth2-proxy |
tomoda-pgadmin-password-{dev,prod} |
auto | pgAdmin |
tomoda-ota-admin-password, tomoda-ota-signing-key |
manual | Xavia OTA (both envs) |
tomoda-ota-{dev,prod}-db-password, tomoda-ota-{dev,prod}-upload-key |
manual | Xavia OTA per env; the prod upload key is also read by tomoda ota.yml on main |
tomoda-release-app-private-key |
manual | tomoda-release GitHub App: tomoda release.yml and ota.yml, devops notify-prod-deploy.yml, all on main |
| Observability and edge | ||
tomoda-cloudflare-api-token |
Cloudflare | Terraform CI (DNS, Workers, R2, Tunnel, Access), external-dns, cert-manager DNS-01 |
tomoda-cloudflare-tunnel-token |
Cloudflare Zero Trust | cloudflared (Loki push only) |
tomoda-tailscale-oauth-client-{id,secret} |
Tailscale OAuth client | Terraform tailscale stack (CI plan, apply on merge) |
tomoda-alert-webhook-primary |
Discord webhook | Alertmanager, Sentry, synthetic Worker, Cloudflare billing notifications |
tomoda-synthetic-probe-password |
auto | Synthetic Worker, prod smoke test, prod backend (seeds the probe user) |
tomoda-review-account-password |
auto | Dev and prod backends (seed the app-store review account); also handed to store reviewers in the Play Console and App Store Connect sign-in details |
tomoda-worker-auth-secret |
openssl rand -hex 32, set by hand |
Synthetic and OG preview Workers sign API calls; dev and prod backends verify them (backend-worker-auth) |
tomoda-sentry-dsn, tomoda-sentry-auth-token |
Sentry | App release builds (tomoda repo) |
tomoda-sentry-grafana-token |
Sentry | Grafana Sentry data source, Backstage Crashes (CRASHES_API_TOKEN) |
tomoda-grafana-admin-password |
auto | Grafana local admin login |
tomoda-backstage-master-password-{dev,prod} |
auto, by setup-gcp-secrets.sh |
Read by an operator's task backstage:bootstrap (tomoda repo) with their own gcloud credentials to create or rotate the first Backstage admin. Never mounted in a pod |
| Mobile release builds (tomoda repo, not the cluster) | ||
tomoda-android-upload-keystore, tomoda-android-keystore-passwords |
manual | Android signing |
tomoda-play-service-account |
Google Play Console | Play uploads |
tomoda-asc-api-key, tomoda-ios-match-passphrase |
App Store Connect, fastlane match | iOS signing + uploads |
Release workflows on main read tomoda-release-app-private-key and tomoda-ota-prod-upload-key as the tomoda-release service account through WIF; neither value is a GitHub secret. See IAM.
People who run native releases locally get secretAccessor on just the release secrets (local.native_release_secrets in infrastructure/gcp/people.tf) and read on tomoda-ios-certs; see GCP.
Per-environment vs shared secrets¶
Dev and prod share app-level identities (the same Google, Stripe and Apple apps) but never share infra credentials or signing keys. Only these carry a -dev / -prod suffix:
| GCP SM key | Why per env |
|---|---|
tomoda-db-password, tomoda-redis-password, tomoda-pgadmin-password |
Each env runs its own instance |
tomoda-jwt-secret |
A dev token must be invalid in prod |
tomoda-asset-signing-key |
A dev-signed media URL must not open prod media |
tomoda-r2-app-{env}-credentials |
Each token touches only its env's assets bucket |
tomoda-ota-{env}-db-password, tomoda-ota-{env}-upload-key |
Each env runs its own OTA server |
Everything else is one shared, unsuffixed secret. Don't add a suffix to any other key; the shared model is intentional.
Backend environment¶
Secrets¶
Pulled by the backend-secrets-{dev|prod} ExternalSecret in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml. NATS_URL comes from the separate backend-nats ExternalSecret in the same file (tomoda-nats-url-{env}), mounted as a required envFrom so a pod never starts on the backend's built-in default URL. WORKER_AUTH_SECRET comes from the backend-worker-auth ExternalSecret (tomoda-worker-auth-secret), mounted as an optional envFrom on the api pods, so an unset value leaves the backend treating Worker calls like any other client.
| Env var | GCP SM key | How generated | Used for |
|---|---|---|---|
JWT_SECRET |
tomoda-jwt-secret |
openssl rand -hex 32 (auto) |
Signs access/refresh tokens |
DB_PASSWORD |
tomoda-db-password |
openssl rand -base64 24 (auto) |
Postgres authentication |
REDIS_PASSWORD |
tomoda-redis-password |
openssl rand -base64 24 (auto) |
Redis authentication |
ASSET_SIGNING_KEY |
tomoda-asset-signing-key |
auto | Media URL signing |
SYNTHETIC_PROBE_PASSWORD (prod only) |
tomoda-synthetic-probe-password |
openssl rand -base64 32 |
Backend upserts the synthetic+probe@tomoda.life user with its hash at startup |
REVIEW_ACCOUNT_PASSWORD |
tomoda-review-account-password |
openssl rand -base64 24 (auto, only when the secret has no version) |
Backend upserts the review@tomoda.life user with its hash at startup. Rotate by adding a secret version, then restarting tomoda-api in the target namespace |
GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET |
tomoda-google-client-id / -secret |
GCP Console > OAuth 2.0 Client IDs (Web) | Google Sign-In |
GOOGLE_IOS_CLIENT_ID |
tomoda-google-ios-client-id |
OAuth client (iOS, bundle com.tomoda.app) |
iOS Google Sign-In |
GOOGLE_ANDROID_CLIENT_ID |
tomoda-google-android-client-id |
OAuth client (Android, package + SHA-1) | Android Google Sign-In |
GOOGLE_PLACES_API_KEY |
tomoda-google-places-api-key |
Copy of google_apikeys_key.places |
Nearby Search, Place Details |
EMAIL_APIKEY |
tomoda-email-api-key |
Email provider dashboard | OTP, welcome, deactivation emails |
STRIPE_SECRET_KEY / STRIPE_WEBHOOK_SECRET |
tomoda-stripe-secret-key / -webhook-secret |
Stripe Dashboard | Payments, webhook verification |
APPLE_CLIENT_ID |
tomoda-apple-client-id |
App ID bundle ID | Apple Sign-In (iOS) |
APPLE_SERVICE_ID |
tomoda-apple-service-id |
Services IDs | Apple Sign-In (web + Android) |
APPLE_TEAM_ID / APPLE_KEY_ID |
tomoda-apple-team-id / -key-id |
Membership / Keys | Apple Sign-In JWT verification |
APPLE_AUTH_KEY_P8 |
tomoda-apple-auth-key-p8 |
Keys > download .p8 (one-time) |
APNs push |
KLIPY_API_KEY |
tomoda-klipy-api-key |
Klipy dashboard | GIF/sticker/clip search; the backend hands it to signed-in apps via GET /klipy/config and the apps call Klipy directly |
SEMANTIC_API_KEY |
tomoda-deepseek-api-key |
DeepSeek console | Semantic resolver (shared across envs) |
SERPER_API_KEY |
tomoda-serper-api-key |
Serper dashboard | Search enrichment; optional, the pipeline degrades without it |
Object storage (R2)¶
The object-storage-credentials ExternalSecret builds the backend's storage env from tomoda-r2-app-{env}-credentials:
| Env var | Value |
|---|---|
STORAGE_ACCESS_KEY_ID / STORAGE_SECRET_ACCESS_KEY |
R2 token keys (S3-compatible API) |
STORAGE_ENDPOINT |
https://<account>.r2.cloudflarestorage.com |
STORAGE_PUBLIC_URL |
https://assets[-dev].tomoda.life |
Bucket and region come from the backend's config.{dev,prod}.yaml.
Non-secret config¶
ConfigMaps backend-config-dev / backend-config-prod, defined in the same external-secret.yaml files:
| Env var | Dev | Prod |
|---|---|---|
DB_HOST |
postgres-postgresql.data.svc.cluster.local |
postgres-prod-postgresql.data.svc.cluster.local |
DB_USER / DB_NAME |
tomoda_dev_user / tomoda_dev |
tomoda_prod_user / tomoda_prod |
DB_SSLMODE / DB_SSLROOTCERT |
verify-full / /etc/postgres-ca/ca.crt |
same |
DB_READ_HOST, DB_MAX_CONNS, DB_POOLER |
"", 20, false |
same |
REDIS_HOST |
redis-dev-master.data.svc.cluster.local |
redis-prod-master.data.svc.cluster.local |
WEBAUTHN_RP_ID |
api-dev.tomoda.life |
tomoda.life |
WEBAUTHN_RP_ORIGIN, FRONTEND_URL |
https://app-dev.tomoda.life |
https://tomoda.life |
Postgres credential coupling¶
tomoda-db-password-{env} feeds two ExternalSecrets: backend-secrets-{env} (the backend's DB_PASSWORD) and postgres-{env}-credentials-eso (the owner password CNPG sets). Rotate both together with the DB password procedure.
Platform secret values¶
Where the hand-set values come from. The observability manual setup walks through the Sentry, Discord and Cloudflare ones.
| GCP SM key | How generated |
|---|---|
tomoda-sentry-dsn |
Sentry project → Client Keys (DSN). Embedded in the app bundle at release build time by frontend/scripts/release/build-with-secrets.sh (tomoda repo) |
tomoda-sentry-auth-token |
Sentry auth token sourcemap-upload (scopes). Source map upload at release build; never embedded |
tomoda-sentry-grafana-token |
Sentry auth token grafana-readonly (event:read, org:read, project:read, member:read). Lands as monitoring/sentry-grafana-credentials and as CRASHES_API_TOKEN in tomoda-prod/backend-secrets-prod; see Sentry |
tomoda-grafana-admin-password |
openssl rand -base64 32, or scripts/setup-gcp-secrets.sh. Lands as monitoring/grafana-admin-credentials (admin-user, admin-password); see Monitoring for the reset after a rotation |
tomoda-cloudflare-api-token |
Cloudflare API token: Zone DNS, Workers Routes; Account Workers Scripts, R2 Storage, Tunnel, Access Apps and Policies (edit); Account Settings (read) |
tomoda-alert-webhook-primary |
Discord channel → Webhooks. Provider-neutral name, so the value can point elsewhere without a rename |
tomoda-cloudflare-tunnel-token |
Zero Trust → Tunnels → tomoda-prod-tunnel. Backs loki-push.tomoda.life |
tomoda-tailscale-oauth-client-{id,secret} |
Tailscale OAuth client; scopes in Tailnet policy |
tomoda-github-app-{id,installation-id,private-key} |
tomoda-ci-automation App: ID from its settings page, installation ID from the org installation URL, private key .pem |
tomoda-github-terraform-app-private-key |
tomoda-terraform App private key; see GitHub. Its App ID is the TERRAFORM_APP_ID repo variable |
tomoda-release-app-private-key |
tomoda-release App private key .pem (App settings → Private keys). Its App ID is the RELEASE_APP_ID repo variable in both repos |
tomoda-r2-cluster-credentials |
R2 API token, Object Read & Write on tomoda-db-backups, tomoda-observability, tomoda-ota-*, tomoda-location-seed, tomoda-photon-index. JSON {"access_key_id","secret_access_key"} |
tomoda-r2-app-{env}-credentials |
R2 API token, Object Read & Write on tomoda-assets-{env} only. Same JSON shape |
tomoda-ghcr-read-credentials |
<github-user>:<classic PAT with read:packages>, one line. k3s splits it at the colon; ghcr-pull and Image Updater use it whole |
Setup¶
GCP secret values¶
gcloud auth login
gcloud config set project development-485000
# Auto-generate what can be, leave manual secrets empty
./scripts/setup-gcp-secrets.sh
# Or pass values via env vars (names in the script header)
TOMODA_GOOGLE_CLIENT_ID=... TOMODA_STRIPE_SECRET_KEY=sk_live_... ./scripts/setup-gcp-secrets.sh
# Set one manual value
gcloud secrets versions add tomoda-google-client-id \
--project=development-485000 --data-file=- # paste the value, then Ctrl-D
The script is safe to re-run: a differing env var adds a version and disables the old one, a matching or empty one is a no-op, an empty secret with no env var is auto-generated where possible, and --rotate regenerates the auto-generated set.
Provisioning the Google OAuth client¶
The OAuth client used by Argo CD Dex, oauth2-proxy and Grafana is created by hand in the Console (Terraform has no supported resource for it).
- APIs & Services → OAuth consent screen in
development-485000: Internal user type (restricts totomoda.life), app nameTomoda DevOps. - Credentials → Create credentials → OAuth client ID, type Web application.
- Authorized redirect URIs:
https://argo-app.tomoda.life/api/dex/callbackhttps://auth.tomoda.life/oauth2/callback
-
Copy the client ID and secret (the secret is shown once) and store them:
pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-google-client-id \ --project=development-485000 --data-file=- pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-google-client-secret \ --project=development-485000 --data-file=- pbcopy </dev/null -
Force ESO to re-sync (
k8s/envs/platform/manifests/external-secrets-config.yaml) and restart consumers:kubectl delete externalsecret google-oauth-secret -n argocd kubectl delete externalsecret google-oauth-secret -n monitoring kubectl rollout restart deployment/argocd-dex-server -n argocd kubectl rollout restart deployment/monitoring-grafana -n monitoring
Rotating the client secret is steps 4 and 5 with a regenerated secret.
R2 and GHCR tokens¶
R2 tokens can only be scoped to buckets that exist, so create them after the Cloudflare stack has applied (R2 → Manage API tokens), then store each as JSON:
gcloud secrets versions add tomoda-r2-cluster-credentials \
--project=development-485000 --data-file=-
# paste {"access_key_id":"…","secret_access_key":"…"}, then Ctrl-D
Repeat for tomoda-r2-app-dev-credentials and tomoda-r2-app-prod-credentials.
Store the GHCR token with its username prefix; the token never shows on screen:
pbpaste | tr -d '\n' | sed 's/^/<github-user>:/' \
| gcloud secrets versions add tomoda-ghcr-read-credentials --project=development-485000 --data-file=-
pbcopy </dev/null
# Check the shape without printing it
G=$(gcloud secrets versions access latest --secret=tomoda-ghcr-read-credentials --project=development-485000)
case "$G" in *:ghp_*) echo "ok (user=${G%%:*})";; *) echo "wrong: expected <github-user>:ghp_..." ;; esac; unset G
After rotating it, disable the old version and push the new value to every node (in-cluster consumers follow within ESO's 1h refresh):
G=$(gcloud secrets versions access latest --secret=tomoda-ghcr-read-credentials --project=development-485000)
printf 'configs:\n ghcr.io:\n auth:\n username: %s\n password: %s\n' "${G%%:*}" "${G#*:}" \
| ssh <login>@tomoda-bhs-1 'sudo install -m 600 /dev/stdin /etc/rancher/k3s/registries.yaml && sudo systemctl restart k3s'
unset G
ESO bootstrap key (per cluster)¶
scripts/ovh/bootstrap-argocd.sh creates external-secrets/gcpsm-credentials from a key for eso-reader (roles/secretmanager.secretAccessor only). Rotate it per IAM.
Verify¶
kubectl get externalsecret -A # all SecretSynced
kubectl get secret backend-secrets-dev object-storage-credentials -n tomoda-dev
kubectl get secret backend-secrets-prod object-storage-credentials -n tomoda-prod
Rotating secrets¶
ESO refreshes every hour (refreshInterval: 1h). To pick up a new value immediately, delete the ExternalSecret; Argo CD recreates it and ESO syncs at once.
App secrets (JWT, OAuth, Stripe, email): no downtime¶
# 1. New version in GCP SM
openssl rand -base64 48 | tr -d '\n' | gcloud secrets versions add tomoda-jwt-secret-prod \
--project=development-485000 --data-file=-
# vendor keys: --data-file=- and paste, then Ctrl-D
# 2. Force ESO refresh
kubectl delete externalsecret backend-secrets-prod -n tomoda-prod
# 3. Restart the backend
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-prod
For auto-generated secrets, run ./scripts/setup-gcp-secrets.sh --rotate.
| Secret | Impact of rotation |
|---|---|
JWT_SECRET |
All sessions invalidated; users sign in again |
GOOGLE_*, APPLE_* |
Next OAuth login uses the new key; sessions unaffected |
STRIPE_* |
Stripe supports key grace periods |
EMAIL_APIKEY |
Next call uses the new key |
KLIPY_API_KEY |
Apps pick up the new key on their next launch; revoke the old key in the Klipy dashboard once its traffic stops |
DB password rotation: requires brief downtime¶
# 1. New password in GCP SM
openssl rand -base64 24 | tr -d '\n' | gcloud secrets versions add tomoda-db-password-prod \
--project=development-485000 --data-file=-
# 2. Force ESO refresh on both consumers
kubectl delete externalsecret backend-secrets-prod -n tomoda-prod
kubectl delete externalsecret postgres-prod-credentials-eso -n data
# 3. CNPG applies the new credential; restart the primary if it hasn't within a minute
kubectl cnpg restart postgres-prod -n data
# 4. Restart the backend
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-prod
Expect 1-3 min of failing health checks during the Postgres restart.
Redis password rotation: brief disruption¶
Same pattern: update GCP SM, force ESO refresh, restart Redis, restart the backend. Redis holds cache and sessions, so users may need to sign in again.
File reference¶
| File | Purpose |
|---|---|
infrastructure/gcp/secrets.tf |
Secret containers |
scripts/setup-gcp-secrets.sh |
Set / rotate secret values |
k8s/envs/platform/manifests/external-secrets-config.yaml |
gsm-tomoda ClusterSecretStore + platform ExternalSecrets |
k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml |
Backend ExternalSecrets + ConfigMap |
k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml |
CNPG cluster + its credential ExternalSecret |
scripts/ovh/bootstrap-argocd.sh |
Creates the gcpsm-credentials bootstrap Secret |