Skip to content

Secrets Management

How application and platform secrets are stored, synced, and rotated.

Overview

One external secret store, GCP Secret Manager, bridged into Kubernetes by the External Secrets Operator (ESO) through the gsm-tomoda ClusterSecretStore:

GCP Secret Manager --> ClusterSecretStore (gsm-tomoda) --> backend-secrets-{dev|prod}, object-storage-credentials,
                                                           r2-*-credentials, platform secrets

ESO authenticates with a key for the eso-reader service account, stored as the cluster's only hand-created Secret (external-secrets/gcpsm-credentials). No secret value is stored in Git. ESO refreshes every hour.

Inventory

Every secret in project development-485000. The containers are Terraform (infrastructure/gcp/secrets.tf); add a new one there first (Adding a secret). "Auto" values come from scripts/setup-gcp-secrets.sh; everything else is set by hand with gcloud secrets versions add.

GCP SM key Value Consumer
Backend
tomoda-db-password-{dev,prod} auto Backend + CNPG owner password
tomoda-redis-password-{dev,prod} auto Backend + Redis
tomoda-jwt-secret-{dev,prod} auto Backend token signing
tomoda-asset-signing-key-{dev,prod} auto Backend media URL signing + assets Workers (Terraform reads it)
tomoda-r2-app-{dev,prod}-credentials R2 token JSON Backend media uploads
tomoda-nats-password-{dev,prod} openssl rand -hex 32 NATS server (nats-bus-auth)
tomoda-nats-url-{dev,prod} nats://tomoda-<env>:<password>@nats-bus.platform.svc.cluster.local:4222 Backend NATS_URL (backend-nats)
tomoda-google-client-id, -client-secret, -ios-client-id, -android-client-id Google Cloud Console Backend sign-in; the web client is also the SSO client for Argo CD Dex, oauth2-proxy and Grafana
tomoda-google-places-api-key Google Cloud Console Backend places lookups
tomoda-apple-{client-id,service-id,team-id,key-id,auth-key-p8} Apple Developer Backend sign-in + push
tomoda-stripe-secret-key, -webhook-secret Stripe Backend payments
tomoda-email-api-key email provider Backend email
tomoda-klipy-api-key, tomoda-deepseek-api-key, tomoda-serper-api-key vendor dashboards Backend
Cluster and platform
tomoda-r2-cluster-credentials R2 token JSON CNPG backups, k3s etcd snapshots, Loki, Tempo, OTA (bundles + DB dumps), seed loader
tomoda-ghcr-read-credentials <github-user>:<classic PAT> (username prefix required) Node image pulls (bootstrap-node.sh), Image Updater, Trivy (ghcr-pull)
tomoda-github-app-{id,installation-id,private-key} tomoda-ci-automation GitHub App Argo CD repo access, ARC runners, trivy-issue-reporter
tomoda-github-terraform-app-private-key tomoda-terraform GitHub App Terraform CI (infrastructure/github)
tomoda-oauth2-proxy-cookie-secret openssl rand -base64 32 oauth2-proxy
tomoda-pgadmin-password-{dev,prod} auto pgAdmin
tomoda-ota-admin-password, tomoda-ota-signing-key manual Xavia OTA (both envs)
tomoda-ota-{dev,prod}-db-password, tomoda-ota-{dev,prod}-upload-key manual Xavia OTA per env; the prod upload key is also read by tomoda ota.yml on main
tomoda-release-app-private-key manual tomoda-release GitHub App: tomoda release.yml and ota.yml, devops notify-prod-deploy.yml, all on main
Observability and edge
tomoda-cloudflare-api-token Cloudflare Terraform CI (DNS, Workers, R2, Tunnel, Access), external-dns, cert-manager DNS-01
tomoda-cloudflare-tunnel-token Cloudflare Zero Trust cloudflared (Loki push only)
tomoda-tailscale-oauth-client-{id,secret} Tailscale OAuth client Terraform tailscale stack (CI plan, apply on merge)
tomoda-alert-webhook-primary Discord webhook Alertmanager, Sentry, synthetic Worker, Cloudflare billing notifications
tomoda-synthetic-probe-password auto Synthetic Worker, prod smoke test, prod backend (seeds the probe user)
tomoda-review-account-password auto Dev and prod backends (seed the app-store review account); also handed to store reviewers in the Play Console and App Store Connect sign-in details
tomoda-worker-auth-secret openssl rand -hex 32, set by hand Synthetic and OG preview Workers sign API calls; dev and prod backends verify them (backend-worker-auth)
tomoda-sentry-dsn, tomoda-sentry-auth-token Sentry App release builds (tomoda repo)
tomoda-sentry-grafana-token Sentry Grafana Sentry data source, Backstage Crashes (CRASHES_API_TOKEN)
tomoda-grafana-admin-password auto Grafana local admin login
tomoda-backstage-master-password-{dev,prod} auto, by setup-gcp-secrets.sh Read by an operator's task backstage:bootstrap (tomoda repo) with their own gcloud credentials to create or rotate the first Backstage admin. Never mounted in a pod
Mobile release builds (tomoda repo, not the cluster)
tomoda-android-upload-keystore, tomoda-android-keystore-passwords manual Android signing
tomoda-play-service-account Google Play Console Play uploads
tomoda-asc-api-key, tomoda-ios-match-passphrase App Store Connect, fastlane match iOS signing + uploads

Release workflows on main read tomoda-release-app-private-key and tomoda-ota-prod-upload-key as the tomoda-release service account through WIF; neither value is a GitHub secret. See IAM.

People who run native releases locally get secretAccessor on just the release secrets (local.native_release_secrets in infrastructure/gcp/people.tf) and read on tomoda-ios-certs; see GCP.

Per-environment vs shared secrets

Dev and prod share app-level identities (the same Google, Stripe and Apple apps) but never share infra credentials or signing keys. Only these carry a -dev / -prod suffix:

GCP SM key Why per env
tomoda-db-password, tomoda-redis-password, tomoda-pgadmin-password Each env runs its own instance
tomoda-jwt-secret A dev token must be invalid in prod
tomoda-asset-signing-key A dev-signed media URL must not open prod media
tomoda-r2-app-{env}-credentials Each token touches only its env's assets bucket
tomoda-ota-{env}-db-password, tomoda-ota-{env}-upload-key Each env runs its own OTA server

Everything else is one shared, unsuffixed secret. Don't add a suffix to any other key; the shared model is intentional.

Backend environment

Secrets

Pulled by the backend-secrets-{dev|prod} ExternalSecret in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml. NATS_URL comes from the separate backend-nats ExternalSecret in the same file (tomoda-nats-url-{env}), mounted as a required envFrom so a pod never starts on the backend's built-in default URL. WORKER_AUTH_SECRET comes from the backend-worker-auth ExternalSecret (tomoda-worker-auth-secret), mounted as an optional envFrom on the api pods, so an unset value leaves the backend treating Worker calls like any other client.

Env var GCP SM key How generated Used for
JWT_SECRET tomoda-jwt-secret openssl rand -hex 32 (auto) Signs access/refresh tokens
DB_PASSWORD tomoda-db-password openssl rand -base64 24 (auto) Postgres authentication
REDIS_PASSWORD tomoda-redis-password openssl rand -base64 24 (auto) Redis authentication
ASSET_SIGNING_KEY tomoda-asset-signing-key auto Media URL signing
SYNTHETIC_PROBE_PASSWORD (prod only) tomoda-synthetic-probe-password openssl rand -base64 32 Backend upserts the synthetic+probe@tomoda.life user with its hash at startup
REVIEW_ACCOUNT_PASSWORD tomoda-review-account-password openssl rand -base64 24 (auto, only when the secret has no version) Backend upserts the review@tomoda.life user with its hash at startup. Rotate by adding a secret version, then restarting tomoda-api in the target namespace
GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET tomoda-google-client-id / -secret GCP Console > OAuth 2.0 Client IDs (Web) Google Sign-In
GOOGLE_IOS_CLIENT_ID tomoda-google-ios-client-id OAuth client (iOS, bundle com.tomoda.app) iOS Google Sign-In
GOOGLE_ANDROID_CLIENT_ID tomoda-google-android-client-id OAuth client (Android, package + SHA-1) Android Google Sign-In
GOOGLE_PLACES_API_KEY tomoda-google-places-api-key Copy of google_apikeys_key.places Nearby Search, Place Details
EMAIL_APIKEY tomoda-email-api-key Email provider dashboard OTP, welcome, deactivation emails
STRIPE_SECRET_KEY / STRIPE_WEBHOOK_SECRET tomoda-stripe-secret-key / -webhook-secret Stripe Dashboard Payments, webhook verification
APPLE_CLIENT_ID tomoda-apple-client-id App ID bundle ID Apple Sign-In (iOS)
APPLE_SERVICE_ID tomoda-apple-service-id Services IDs Apple Sign-In (web + Android)
APPLE_TEAM_ID / APPLE_KEY_ID tomoda-apple-team-id / -key-id Membership / Keys Apple Sign-In JWT verification
APPLE_AUTH_KEY_P8 tomoda-apple-auth-key-p8 Keys > download .p8 (one-time) APNs push
KLIPY_API_KEY tomoda-klipy-api-key Klipy dashboard GIF/sticker/clip search; the backend hands it to signed-in apps via GET /klipy/config and the apps call Klipy directly
SEMANTIC_API_KEY tomoda-deepseek-api-key DeepSeek console Semantic resolver (shared across envs)
SERPER_API_KEY tomoda-serper-api-key Serper dashboard Search enrichment; optional, the pipeline degrades without it

Object storage (R2)

The object-storage-credentials ExternalSecret builds the backend's storage env from tomoda-r2-app-{env}-credentials:

Env var Value
STORAGE_ACCESS_KEY_ID / STORAGE_SECRET_ACCESS_KEY R2 token keys (S3-compatible API)
STORAGE_ENDPOINT https://<account>.r2.cloudflarestorage.com
STORAGE_PUBLIC_URL https://assets[-dev].tomoda.life

Bucket and region come from the backend's config.{dev,prod}.yaml.

Non-secret config

ConfigMaps backend-config-dev / backend-config-prod, defined in the same external-secret.yaml files:

Env var Dev Prod
DB_HOST postgres-postgresql.data.svc.cluster.local postgres-prod-postgresql.data.svc.cluster.local
DB_USER / DB_NAME tomoda_dev_user / tomoda_dev tomoda_prod_user / tomoda_prod
DB_SSLMODE / DB_SSLROOTCERT verify-full / /etc/postgres-ca/ca.crt same
DB_READ_HOST, DB_MAX_CONNS, DB_POOLER "", 20, false same
REDIS_HOST redis-dev-master.data.svc.cluster.local redis-prod-master.data.svc.cluster.local
WEBAUTHN_RP_ID api-dev.tomoda.life tomoda.life
WEBAUTHN_RP_ORIGIN, FRONTEND_URL https://app-dev.tomoda.life https://tomoda.life

Postgres credential coupling

tomoda-db-password-{env} feeds two ExternalSecrets: backend-secrets-{env} (the backend's DB_PASSWORD) and postgres-{env}-credentials-eso (the owner password CNPG sets). Rotate both together with the DB password procedure.

Platform secret values

Where the hand-set values come from. The observability manual setup walks through the Sentry, Discord and Cloudflare ones.

GCP SM key How generated
tomoda-sentry-dsn Sentry project → Client Keys (DSN). Embedded in the app bundle at release build time by frontend/scripts/release/build-with-secrets.sh (tomoda repo)
tomoda-sentry-auth-token Sentry auth token sourcemap-upload (scopes). Source map upload at release build; never embedded
tomoda-sentry-grafana-token Sentry auth token grafana-readonly (event:read, org:read, project:read, member:read). Lands as monitoring/sentry-grafana-credentials and as CRASHES_API_TOKEN in tomoda-prod/backend-secrets-prod; see Sentry
tomoda-grafana-admin-password openssl rand -base64 32, or scripts/setup-gcp-secrets.sh. Lands as monitoring/grafana-admin-credentials (admin-user, admin-password); see Monitoring for the reset after a rotation
tomoda-cloudflare-api-token Cloudflare API token: Zone DNS, Workers Routes; Account Workers Scripts, R2 Storage, Tunnel, Access Apps and Policies (edit); Account Settings (read)
tomoda-alert-webhook-primary Discord channel → Webhooks. Provider-neutral name, so the value can point elsewhere without a rename
tomoda-cloudflare-tunnel-token Zero Trust → Tunnels → tomoda-prod-tunnel. Backs loki-push.tomoda.life
tomoda-tailscale-oauth-client-{id,secret} Tailscale OAuth client; scopes in Tailnet policy
tomoda-github-app-{id,installation-id,private-key} tomoda-ci-automation App: ID from its settings page, installation ID from the org installation URL, private key .pem
tomoda-github-terraform-app-private-key tomoda-terraform App private key; see GitHub. Its App ID is the TERRAFORM_APP_ID repo variable
tomoda-release-app-private-key tomoda-release App private key .pem (App settings → Private keys). Its App ID is the RELEASE_APP_ID repo variable in both repos
tomoda-r2-cluster-credentials R2 API token, Object Read & Write on tomoda-db-backups, tomoda-observability, tomoda-ota-*, tomoda-location-seed, tomoda-photon-index. JSON {"access_key_id","secret_access_key"}
tomoda-r2-app-{env}-credentials R2 API token, Object Read & Write on tomoda-assets-{env} only. Same JSON shape
tomoda-ghcr-read-credentials <github-user>:<classic PAT with read:packages>, one line. k3s splits it at the colon; ghcr-pull and Image Updater use it whole

Setup

GCP secret values

gcloud auth login
gcloud config set project development-485000

# Auto-generate what can be, leave manual secrets empty
./scripts/setup-gcp-secrets.sh

# Or pass values via env vars (names in the script header)
TOMODA_GOOGLE_CLIENT_ID=... TOMODA_STRIPE_SECRET_KEY=sk_live_... ./scripts/setup-gcp-secrets.sh

# Set one manual value
gcloud secrets versions add tomoda-google-client-id \
  --project=development-485000 --data-file=-        # paste the value, then Ctrl-D

The script is safe to re-run: a differing env var adds a version and disables the old one, a matching or empty one is a no-op, an empty secret with no env var is auto-generated where possible, and --rotate regenerates the auto-generated set.

Provisioning the Google OAuth client

The OAuth client used by Argo CD Dex, oauth2-proxy and Grafana is created by hand in the Console (Terraform has no supported resource for it).

  1. APIs & Services → OAuth consent screen in development-485000: Internal user type (restricts to tomoda.life), app name Tomoda DevOps.
  2. Credentials → Create credentials → OAuth client ID, type Web application.
  3. Authorized redirect URIs:
    • https://argo-app.tomoda.life/api/dex/callback
    • https://auth.tomoda.life/oauth2/callback
  4. Copy the client ID and secret (the secret is shown once) and store them:

    pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-google-client-id \
        --project=development-485000 --data-file=-
    pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-google-client-secret \
        --project=development-485000 --data-file=-
    pbcopy </dev/null
    
  5. Force ESO to re-sync (k8s/envs/platform/manifests/external-secrets-config.yaml) and restart consumers:

    kubectl delete externalsecret google-oauth-secret -n argocd
    kubectl delete externalsecret google-oauth-secret -n monitoring
    kubectl rollout restart deployment/argocd-dex-server -n argocd
    kubectl rollout restart deployment/monitoring-grafana -n monitoring
    

Rotating the client secret is steps 4 and 5 with a regenerated secret.

R2 and GHCR tokens

R2 tokens can only be scoped to buckets that exist, so create them after the Cloudflare stack has applied (R2 → Manage API tokens), then store each as JSON:

gcloud secrets versions add tomoda-r2-cluster-credentials \
  --project=development-485000 --data-file=-
# paste {"access_key_id":"…","secret_access_key":"…"}, then Ctrl-D

Repeat for tomoda-r2-app-dev-credentials and tomoda-r2-app-prod-credentials.

Store the GHCR token with its username prefix; the token never shows on screen:

pbpaste | tr -d '\n' | sed 's/^/<github-user>:/' \
  | gcloud secrets versions add tomoda-ghcr-read-credentials --project=development-485000 --data-file=-
pbcopy </dev/null

# Check the shape without printing it
G=$(gcloud secrets versions access latest --secret=tomoda-ghcr-read-credentials --project=development-485000)
case "$G" in *:ghp_*) echo "ok (user=${G%%:*})";; *) echo "wrong: expected <github-user>:ghp_..." ;; esac; unset G

After rotating it, disable the old version and push the new value to every node (in-cluster consumers follow within ESO's 1h refresh):

G=$(gcloud secrets versions access latest --secret=tomoda-ghcr-read-credentials --project=development-485000)
printf 'configs:\n  ghcr.io:\n    auth:\n      username: %s\n      password: %s\n' "${G%%:*}" "${G#*:}" \
  | ssh <login>@tomoda-bhs-1 'sudo install -m 600 /dev/stdin /etc/rancher/k3s/registries.yaml && sudo systemctl restart k3s'
unset G

ESO bootstrap key (per cluster)

scripts/ovh/bootstrap-argocd.sh creates external-secrets/gcpsm-credentials from a key for eso-reader (roles/secretmanager.secretAccessor only). Rotate it per IAM.

Verify

kubectl get externalsecret -A          # all SecretSynced
kubectl get secret backend-secrets-dev object-storage-credentials -n tomoda-dev
kubectl get secret backend-secrets-prod object-storage-credentials -n tomoda-prod

Rotating secrets

ESO refreshes every hour (refreshInterval: 1h). To pick up a new value immediately, delete the ExternalSecret; Argo CD recreates it and ESO syncs at once.

App secrets (JWT, OAuth, Stripe, email): no downtime

# 1. New version in GCP SM
openssl rand -base64 48 | tr -d '\n' | gcloud secrets versions add tomoda-jwt-secret-prod \
  --project=development-485000 --data-file=-
# vendor keys: --data-file=- and paste, then Ctrl-D

# 2. Force ESO refresh
kubectl delete externalsecret backend-secrets-prod -n tomoda-prod

# 3. Restart the backend
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-prod

For auto-generated secrets, run ./scripts/setup-gcp-secrets.sh --rotate.

Secret Impact of rotation
JWT_SECRET All sessions invalidated; users sign in again
GOOGLE_*, APPLE_* Next OAuth login uses the new key; sessions unaffected
STRIPE_* Stripe supports key grace periods
EMAIL_APIKEY Next call uses the new key
KLIPY_API_KEY Apps pick up the new key on their next launch; revoke the old key in the Klipy dashboard once its traffic stops

DB password rotation: requires brief downtime

# 1. New password in GCP SM
openssl rand -base64 24 | tr -d '\n' | gcloud secrets versions add tomoda-db-password-prod \
  --project=development-485000 --data-file=-

# 2. Force ESO refresh on both consumers
kubectl delete externalsecret backend-secrets-prod          -n tomoda-prod
kubectl delete externalsecret postgres-prod-credentials-eso -n data

# 3. CNPG applies the new credential; restart the primary if it hasn't within a minute
kubectl cnpg restart postgres-prod -n data

# 4. Restart the backend
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-prod

Expect 1-3 min of failing health checks during the Postgres restart.

Redis password rotation: brief disruption

Same pattern: update GCP SM, force ESO refresh, restart Redis, restart the backend. Redis holds cache and sessions, so users may need to sign in again.

File reference

File Purpose
infrastructure/gcp/secrets.tf Secret containers
scripts/setup-gcp-secrets.sh Set / rotate secret values
k8s/envs/platform/manifests/external-secrets-config.yaml gsm-tomoda ClusterSecretStore + platform ExternalSecrets
k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml Backend ExternalSecrets + ConfigMap
k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml CNPG cluster + its credential ExternalSecret
scripts/ovh/bootstrap-argocd.sh Creates the gcpsm-credentials bootstrap Secret