Secrets Management¶

Authoritative reference for how application secrets are stored, synced, and rotated.

Overview¶

Two external secret stores, bridged into Kubernetes by the External Secrets Operator (ESO):

GCP Secret Manager --> ClusterSecretStore (gsm-tomoda)  --> backend-secrets-{dev|prod}
GCP Secret Manager --> ClusterSecretStore (gsm-tomoda)  --> aws-eso-credentials-{env}
                                                              |
                                                              v
AWS Secrets Manager --> SecretStore (aws-sm-{env}, per ns) --> s3-uploader-secret

The AWS store's own credentials are themselves sourced from GCP SM (via the Workload-Identity gsm-tomoda store) and projected into each tenant namespace — so every secret ultimately bootstraps from GCP Workload Identity, with no hand-created Kubernetes secret. See AWS Secrets Manager access.

Store	What lives there	Why
GCP Secret Manager	App secrets — JWT, OAuth, Stripe, email, etc.	App runs on GKE; native integration via Workload Identity
AWS Secrets Manager	S3 uploader credentials	Assets hosted on AWS S3; credentials generated by Terraform

No secret value is ever stored in Git. All sensitive values live in GCP SM or AWS SM and are pulled into K8s Secrets automatically. ESO refreshes every 1 hour by default.

Per-environment vs shared secrets¶

Dev and prod share app-level identities (the same Google, Stripe, Apple, LINE apps) but must not share infra credentials or auth-signing keys. So only genuinely per-environment secrets carry a -dev / -prod suffix in GCP Secret Manager; everything else is a single shared, unsuffixed secret.

GCP SM key	Suffixed?	Why
`tomoda-db-password`	`-dev` / `-prod`	Each env runs its own Postgres
`tomoda-redis-password`	`-dev` / `-prod`	Each env runs its own Redis
`tomoda-pgadmin-password`	`-dev` / `-prod`	Each env runs its own pgAdmin
`tomoda-jwt-secret`	`-dev` / `-prod`	Cryptographic isolation — a dev token must be invalid in prod
`tomoda-encryption-key`	`-dev` / `-prod`	Cryptographic isolation — dev ciphertext must be invalid in prod
Everything else	No — shared	Same third-party app / platform identity across both envs

Shared (unsuffixed) covers Google OAuth, Stripe, Apple, LINE, email, Klipy, DeepSeek, Serper, and all platform secrets (Sentry, Cloudflare, GitHub App, alert webhook, synthetic-probe).

scripts/setup-gcp-secrets.sh creates the five per-env secrets under both dev and prod, and the shared set once under global. The backend ExternalSecret overlays in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml reference the suffixed keys only for those five, and the unsuffixed keys for everything else.

Don't re-suffix

Only those five secrets are per-environment. Resist adding -dev / -prod to any other key — the shared model is intentional.

Complete variable reference¶

Secrets from GCP Secret Manager¶

Pulled by the backend-secrets-{dev|prod} ExternalSecret in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml.

Env Var	GCP SM Key	How Generated	Used For
`JWT_SECRET`	`tomoda-jwt-secret`	`openssl rand -hex 32` (auto)	Signs JWT access/refresh tokens
`ENCRYPTION_KEY`	`tomoda-encryption-key`	`openssl rand -hex 16` (auto)	Reserved for future field-level encryption
`DB_PASSWORD`	`tomoda-db-password`	`openssl rand -base64 24` (auto)	Postgres authentication
`REDIS_PASSWORD`	`tomoda-redis-password`	`openssl rand -base64 24` (auto)	Redis authentication
`GOOGLE_CLIENT_ID`	`tomoda-google-client-id`	GCP Console > OAuth 2.0 Client IDs (Web)	Google Sign-In (backend + web)
`GOOGLE_CLIENT_SECRET`	`tomoda-google-client-secret`	Same	Google Sign-In token validation
`GOOGLE_IOS_CLIENT_ID`	`tomoda-google-ios-client-id`	OAuth 2.0 Client (iOS, bundle `com.tomoda.app`)	iOS Google Sign-In
`GOOGLE_ANDROID_CLIENT_ID`	`tomoda-google-android-client-id`	OAuth 2.0 Client (Android, package + SHA-1)	Android Google Sign-In
`GOOGLE_PLACES_API_KEY`	`tomoda-google-places-api-key`	API Key (restrict to Places API New)	Nearby Search, Place Details
`EMAIL_APIKEY`	`tomoda-email-api-key`	Dispatch dashboard > API Keys	OTP, welcome, deactivation emails
`STRIPE_SECRET_KEY`	`tomoda-stripe-secret-key`	Stripe Dashboard > API Keys	Payments, subscriptions
`STRIPE_WEBHOOK_SECRET`	`tomoda-stripe-webhook-secret`	Stripe Dashboard > Webhooks	Webhook signature verification
`LINE_CLIENT_ID`	`tomoda-line-client-id`	LINE Dev Console > Channel ID	LINE Login
`LINE_CLIENT_SECRET`	`tomoda-line-client-secret`	Same	LINE Login token verification
`APPLE_CLIENT_ID`	`tomoda-apple-client-id`	Apple Dev > App ID Bundle ID	Apple Sign-In (iOS native)
`APPLE_SERVICE_ID`	`tomoda-apple-service-id`	Apple Dev > Services IDs	Apple Sign-In (web + Android)
`APPLE_TEAM_ID`	`tomoda-apple-team-id`	Apple Dev > Membership	Apple Sign-In
`APPLE_KEY_ID`	`tomoda-apple-key-id`	Apple Dev > Keys	Apple Sign-In JWT verification
`APPLE_AUTH_KEY_P8`	`tomoda-apple-auth-key-p8`	Apple Dev > Keys > Download .p8 (one-time)	APNs push notifications
`KLIPY_API_KEY`	`tomoda-klipy-api-key`	Klipy dashboard	GIF/sticker/clip search
`SEMANTIC_API_KEY`	`tomoda-deepseek-api-key`	DeepSeek console → API keys	DeepSeek chat completions for the semantic resolver (link entity extraction + place selection). Shared across dev + prod today; split per env if blast-radius isolation matters.
`SERPER_API_KEY`	`tomoda-serper-api-key`	Serper dashboard → API keys	Search-enrichment provider (Serper today; swappable via the `SearchEnricher` interface). Optional — the pipeline degrades without it.
`CLOUDFRONT_PRIVATE_KEY_PEM`	`tomoda-cloudfront-signing-private-key-{env}`	Generated by `infrastructure/aws/cloudfront_signing.tf` (`tls_private_key`) and pushed by the same apply	RSA private key the backend uses to sign CloudFront URLs
`CLOUDFRONT_KEY_PAIR_ID`	`tomoda-cloudfront-key-pair-id-{env}`	Same — comes from `aws_cloudfront_public_key.signing.id`	The `Key-Pair-Id` query parameter CloudFront uses to look up the matching public key

Observability secrets (GCP Secret Manager)¶

Consumed by Alertmanager (Discord routing), the Cloudflare synthetic Worker (probes + tunneled Loki push), and the synthetic test user seed migration. Created via the observability manual setup — work through that page once before the observability stack can run end-to-end.

GCP SM Key	How Generated	Used For
`tomoda-sentry-dsn`	Sentry project → Settings → Client Keys (DSN)	Embedded in the JS bundle at native + web release-build time; tells `@sentry/react-native` where to send events. Pulled by `frontend/scripts/build-with-secrets.sh` in the tomoda repo
`tomoda-sentry-auth-token`	Sentry → Settings → Account → Auth Tokens (scopes `project:releases` + `org:read`) — name `sourcemap-upload`	Used by the Sentry CLI during the native + web release build to upload source maps. Pulled by `build-with-secrets.sh` — never embedded in the bundle
`tomoda-sentry-grafana-token`	Sentry → Settings → Account → Auth Tokens (scopes `event:read` + `org:read` + `project:read` + `member:read`) — name `grafana-readonly`	Projected via ESO into `monitoring/sentry-grafana-credentials` so Grafana's `grafana-sentry-datasource` plugin can query frontend metrics (issue rate, crash-free %, transaction p50/p95) — see Sentry data source
`tomoda-cloudflare-api-token`	Cloudflare dashboard → Profile → API Tokens (scopes: Zone:DNS:Edit, Account:Workers Scripts:Edit, Account:Cloudflare Tunnel:Edit, Account:Account Settings:Read)	Terraform `cloudflare` provider — DNS, Workers (synthetic monitoring), Tunnel management
`tomoda-alert-webhook-primary`	Discord channel → Webhooks → New Webhook	Alertmanager Discord receiver, Sentry Discord integration, synthetic Worker failure notifications. Provider-portable name — value can later be a Slack/PagerDuty/Opsgenie URL without renaming the secret
`tomoda-cloudflare-tunnel-token`	Cloudflare Zero Trust → Networks → Tunnels → Create tunnel `tomoda-prod-tunnel`	The `cloudflared` Deployment in `monitoring` authenticates to Cloudflare's edge with this token; backs the public `loki-push.tomoda.life` hostname
`tomoda-cloudflare-access-client-id`	Cloudflare Zero Trust → Access → Service Auth → token `synthetic-worker`	Worker → Loki push request includes this in `CF-Access-Client-Id` header so Access lets the request through
`tomoda-cloudflare-access-client-secret`	Same as above (pair)	Worker → Loki push request includes this in `CF-Access-Client-Secret` header
`tomoda-synthetic-probe-password`	`openssl rand -base64 32` (operator picks the value)	Cleartext password the synthetic Worker POSTs to `/api/v1/auth/login`; the seed migration in `backend/internal/database/migrations/` writes the hashed value to the synthetic test user row
`tomoda-github-app-id`	GitHub App settings page (`Tomoda ARC Runners`) — visible at the top after creation	ARC reads this to identify the GitHub App when minting installation tokens. Not strictly secret but kept here for symmetry
`tomoda-github-app-installation-id`	URL of the App's installation page in `tomoda-labs` org settings (numeric path segment after `installations/`)	ARC pairs this with the App ID to mint installation-scoped GitHub tokens
`tomoda-github-app-private-key`	GitHub App settings → Generate a private key (downloads `.pem`)	Full PEM contents. ARC uses this to sign JWTs that exchange for short-lived installation tokens — the actual sensitive credential for the runner pool
`tomoda-deepseek-api-key`	DeepSeek platform → API keys (shared with the backend)	Projected via ESO into `monitoring/cost-exporter-deepseek` so the cost-exporter can `GET /user/balance` — see Cost monitoring
`tomoda-aws-cost-explorer-creds`	JSON `{"AWS_ACCESS_KEY_ID","AWS_SECRET_ACCESS_KEY"}` mirrored from AWS SM `tomoda-cost-explorer-{env}` (written by `infrastructure/aws/cost_explorer_iam.tf`)	Projected via ESO into `monitoring/cost-exporter-aws` so the cost-exporter can call `ce:GetCostAndUsage`. IAM user is scoped to that one action

Secrets from AWS Secrets Manager¶

Pulled by the s3-uploader-secret ExternalSecret. The AWS SM secret is a single JSON blob; ESO splits it into individual K8s Secret keys via the property selector.

Env Var	AWS SM Key	Source	Used For
`AWS_ACCESS_KEY_ID`	`tomoda-s3-uploader-{env}`	Terraform	S3 asset uploads
`AWS_SECRET_ACCESS_KEY`	`tomoda-s3-uploader-{env}`	Terraform	S3 asset uploads
`S3_BUCKET`	`tomoda-s3-uploader-{env}`	Terraform	`tomoda-assets-{env}`
`S3_REGION`	`tomoda-s3-uploader-{env}`	Terraform	`us-east-2`
`S3_BASE_URL`	`tomoda-s3-uploader-{env}`	Terraform	CloudFront domain (`https://assets[-dev].tomoda.life`)

Non-secret config (ConfigMap)¶

Values that are not sensitive (DB host, ports, URLs) live in the per-env ConfigMap defined alongside the ExternalSecrets:

backend-config-dev in tomoda
backend-config-prod in prod

Env Var	Dev	Prod	Purpose
`ENV`	`dev`	`prod`	Environment label
`DB_HOST`	`postgres-postgresql.data.svc.cluster.local`	`postgres-prod-postgresql.data.svc.cluster.local`	Postgres alias
`DB_PORT`	`5432`	`5432`
`DB_USER`	`tomoda_dev_user`	`tomoda_prod_user`
`DB_NAME`	`tomoda_dev`	`tomoda_prod`
`DB_SSLMODE`	`disable`	`require`
`REDIS_HOST`	`redis-master.data.svc.cluster.local`	`redis-prod-master.data.svc.cluster.local`
`REDIS_PORT`	`6379`	`6379`
`WEBAUTHN_RP_ID`	`api-dev.tomoda.life`	`api.tomoda.life`	Passkey RP
`WEBAUTHN_RP_ORIGIN`	`https://app-dev.tomoda.life`	`https://app.tomoda.life`	Passkey origin
`FRONTEND_URL`	`https://app-dev.tomoda.life`	`https://app.tomoda.life`	Email link base

Postgres credential coupling¶

The DB_PASSWORD value in GCP SM is consumed by two ExternalSecrets:

backend-secrets-{dev|prod} — exposes it as the backend's DB_PASSWORD env var
postgres-{dev|prod}-credentials-eso — populates the K8s Secret CNPG uses to set the Postgres owner password at initdb time

Both reference tomoda-db-password. Keep them aligned by rotating via the procedure below.

AWS Secrets Manager access¶

ESO needs an AWS key to read AWS Secrets Manager. Rather than a hand-created Kubernetes secret, that key lives in GCP Secret Manager and is projected into each tenant namespace via the Workload-Identity gsm-tomoda store — the same store used for every other app secret. This breaks the chicken-and-egg (the AWS store needs a key to authenticate; the key now arrives through GitOps).

Per environment, three moving parts, all in k8s/apps/tomoda/overlays/<env>/external-secret.yaml:

ExternalSecret aws-eso-credentials-<env> — pulls tomoda-aws-eso-access-key-<env> + tomoda-aws-eso-secret-key-<env> from GCP SM (store gsm-tomoda) into a K8s Secret with keys access-key / secret-access-key.
SecretStore aws-sm-<env> — namespace-scoped AWS Secrets Manager store, authenticating with the secret above. dev and prod own separate stores so a key rotation stays inside one namespace.
ExternalSecret s3-uploader-secret — reads tomoda-s3-uploader-<env> from aws-sm-<env>.

GCP SM secret	Source
`tomoda-aws-eso-access-key-<env>`	`terraform output -raw eso_reader_access_key_id`
`tomoda-aws-eso-secret-key-<env>`	`terraform output -raw eso_reader_secret_access_key`

See setup step 3 below.

Setup guide¶

Step 1 — GCP secrets (one-time)¶

gcloud auth login
gcloud config set project development-485000

# Option A — bootstrap with empty values, fill manual secrets one by one
./scripts/setup-gcp-secrets.sh
printf 'YOUR_VALUE' | gcloud secrets versions add tomoda-google-client-id \
  --project=development-485000 --data-file=-

# Option B — pass everything via env vars
TOMODA_GOOGLE_CLIENT_ID=... \
TOMODA_GOOGLE_CLIENT_SECRET=GOCSPX-... \
TOMODA_STRIPE_SECRET_KEY=sk_live_... \
TOMODA_EMAIL_APIKEY=eyJhbG... \
./scripts/setup-gcp-secrets.sh

OAuth clients are NOT managed by Terraform

The Google IAP OAuth client used by Argo CD/Dex (and by Grafana's auth-proxy flow) is created manually via the Google Cloud Console. The google_iap_brand and google_iap_client Terraform resources were removed because the underlying APIs are deprecated and being shut down (Jan 19, 2026 / Mar 19, 2026). The client secret and ID are then stored in GCP Secret Manager as tomoda-google-client-secret and tomoda-google-client-id and pulled into the argocd and monitoring namespaces by the ExternalSecret resources in k8s/envs/platform/manifests/external-secrets-config.yaml. See Provisioning the Google OAuth client below for the click path.

Provisioning the Google OAuth client¶

For Argo CD SSO and any future browser-facing SSO flows.

Go to GCP Console → APIs & Services → OAuth consent screen in project development-485000.
Configure the consent screen if not already configured: Internal user type (restricts to tomoda.life Workspace), App name Tomoda DevOps, support email, developer contact email.
Go to APIs & Services → Credentials → + Create credentials → OAuth client ID.
Application type: Web application. Name: Tomoda Argo CD (or similar).
Authorized redirect URIs — add both:
- https://argo-app.tomoda.life/api/dex/callback
- https://auth.tomoda.life/oauth2/callback
Click Create. Copy the client ID and client secret immediately (the secret is only shown once).

Store them in GCP Secret Manager:

printf '$CLIENT_ID' | gcloud secrets versions add tomoda-google-client-id \
    --project=development-485000 --data-file=-

printf '$CLIENT_SECRET' | gcloud secrets versions add tomoda-google-client-secret \
    --project=development-485000 --data-file=-

Force ESO to project the new value into the target namespaces:

kubectl delete externalsecret google-oauth-secret -n argocd
kubectl delete externalsecret google-oauth-secret -n monitoring
# Argo CD recreates both within seconds; ESO re-syncs.

Restart the consumers so they pick up the new value:

kubectl rollout restart deployment/argocd-dex-server -n argocd
kubectl rollout restart deployment/monitoring-grafana -n monitoring   # if Grafana auth is enabled

The same flow works for rotating the client secret — re-do steps 6-9 with a regenerated secret from the Console.

The script's behaviour:

Env var set + differs from current: creates a new version, disables the old
Env var set + matches current: no-op
Env var empty + secret already has a value: leaves it alone
Env var empty + secret empty: auto-generates for JWT/encryption/DB/Redis, leaves blank for manual secrets
--rotate flag: force-regenerates auto-generated secrets regardless

Safe to re-run anytime.

Step 2 — AWS Terraform (one-time per environment)¶

cd infrastructure/aws

# Dev
terraform apply -var="environment=dev"

# Prod
terraform apply -var="environment=prod"

This creates:

The S3 IAM uploader user + access key
The AWS Secrets Manager secret with the credentials
The ESO reader IAM user

Step 3 — Push the eso-reader key to GCP Secret Manager (per environment)¶

No kubectl create secret — the key goes into GCP SM and ESO projects it via GitOps (see AWS Secrets Manager access).

cd infrastructure/aws   # select the right workspace first (dev / prod)
terraform output -raw eso_reader_access_key_id \
  | gcloud secrets create tomoda-aws-eso-access-key-<env> \
      --project=development-485000 --replication-policy=automatic --data-file=-
terraform output -raw eso_reader_secret_access_key \
  | gcloud secrets create tomoda-aws-eso-secret-key-<env> \
      --project=development-485000 --replication-policy=automatic --data-file=-

On the next Argo sync, aws-eso-credentials-<env>, the aws-sm-<env> store, and s3-uploader-secret come up automatically in the tenant namespace.

Step 4 — Verify¶

# All ExternalSecrets in SecretSynced state
kubectl get externalsecret -A

# Per-env synced K8s Secrets exist
kubectl get secret backend-secrets-dev   -n tomoda
kubectl get secret s3-uploader-secret    -n tomoda
kubectl get secret backend-secrets-prod  -n prod
kubectl get secret s3-uploader-secret    -n prod

How it works¶

Secret refresh¶

ESO refreshes every 1 hour (refreshInterval: 1h). To force an immediate refresh after rotating a value:

kubectl delete externalsecret backend-secrets-dev -n tomoda
# Argo CD recreates the resource within seconds; ESO syncs immediately

Workload Identity¶

The ESO controller's K8s SA is bound (via the iam.gke.io/gcp-service-account annotation) to a GCP service account with roles/secretmanager.secretAccessor. That's how it authenticates to GCP SM without static keys.

AWS has no GCP-Workload-Identity-style mechanism reachable from inside GKE, so the AWS store needs a static key. That key isn't hand-created: it's stored in GCP SM and projected into each namespace as aws-eso-credentials-<env> through the Workload-Identity gsm-tomoda store — so the AWS path still bootstraps from Workload Identity, one hop removed. See AWS Secrets Manager access.

Rotating secrets¶

App secrets (JWT, OAuth, Stripe, email) — no downtime¶

# 1. New version in GCP SM
printf 'NEW_VALUE' | gcloud secrets versions add tomoda-jwt-secret \
  --project=development-485000 --data-file=-

# 2. Force ESO refresh
kubectl delete externalsecret backend-secrets-prod -n prod

# 3. Restart backend to pick up new env vars
kubectl rollout restart deployment/tomoda-api -n prod
kubectl rollout restart deployment/tomoda-async -n prod

For auto-generated secrets, run ./scripts/setup-gcp-secrets.sh --rotate.

Impact by secret:

Secret	Impact of rotation
`JWT_SECRET`	All active sessions invalidated — users re-login
`ENCRYPTION_KEY`	Currently unused — no impact
`GOOGLE_`, `LINE_`, `APPLE_*`	Next OAuth login uses new key; existing sessions unaffected
`STRIPE_*`	Stripe supports key grace periods; no immediate impact
`EMAIL_APIKEY`	Next email send uses new key
`KLIPY_API_KEY`	Next API call uses new key

DB password rotation — requires brief downtime¶

CNPG's owner password is set from tomoda-db-password via the postgres-{env}-credentials-eso ExternalSecret. Backend reads the same value. Rotation:

# 1. New password in GCP SM
NEW_PW=$(openssl rand -base64 24)
printf "$NEW_PW" | gcloud secrets versions add tomoda-db-password \
  --project=development-485000 --data-file=-

# 2. Force ESO refresh on both consumers
kubectl delete externalsecret backend-secrets-prod        -n prod
kubectl delete externalsecret postgres-prod-credentials-eso -n data

# 3. CNPG will pick up the new credential secret and apply it.
#    Force a primary restart if CNPG hasn't applied it within a minute:
kubectl cnpg restart cluster postgres-prod -n data

# 4. Restart backend to pick up the new env var
kubectl rollout restart deployment/tomoda-api -n prod
kubectl rollout restart deployment/tomoda-async -n prod

Downtime: 1-3 min during the Postgres restart. Backend health checks will fail until Postgres is back.

Redis password rotation — brief disruption¶

Same pattern. Update GCP SM, force ESO refresh, restart Redis, restart backend. Redis data is ephemeral (cache + sessions); users may need to re-login.

File reference¶

File	Purpose
`scripts/setup-gcp-secrets.sh`	Create / rotate GCP SM secrets
`k8s/envs/platform/manifests/external-secrets-config.yaml`	`gsm-tomoda` `ClusterSecretStore` (GCP) + platform ExternalSecrets
`k8s/apps/tomoda/overlays/dev/external-secret.yaml`	Dev `ExternalSecret`s, `aws-sm-dev` `SecretStore`, ConfigMap
`k8s/apps/tomoda/overlays/prod/external-secret.yaml`	Prod `ExternalSecret`s, `aws-sm-prod` `SecretStore`, ConfigMap
`k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml`	CNPG cluster + per-cluster credential ExternalSecret
`infrastructure/aws/iam_uploader.tf`	S3 IAM user, AWS SM secret, ESO reader IAM user

Manual setup summary¶

Item	Manual?	When
Run `scripts/setup-gcp-secrets.sh`	Yes	Once
Set ~16 GCP secret values	Yes	Once per secret
`terraform apply` in `infrastructure/aws/`	Yes	Once per environment
Push eso-reader key to GCP SM (`tomoda-aws-eso-*-<env>`)	Yes	Once per environment
S3 credentials in AWS SM	No	Terraform
JWT / DB / Redis passwords	No	Script auto-generates
K8s Secret resource creation	No	ESO
Secret refresh	No	ESO every 1 hour