Secrets Management¶

Authoritative reference for how application secrets are stored, synced, and rotated.

Overview¶

Two external secret stores, bridged into Kubernetes by the External Secrets Operator (ESO):

GCP Secret Manager --> ClusterSecretStore (gsm-tomoda)   --> backend-secrets-{dev|prod}
AWS Secrets Manager --> ClusterSecretStore (aws-sm-tomoda) --> s3-uploader-secret

Store	What lives there	Why
GCP Secret Manager	App secrets — JWT, OAuth, Stripe, email, etc.	App runs on GKE; native integration via Workload Identity
AWS Secrets Manager	S3 uploader credentials	Assets hosted on AWS S3; credentials generated by Terraform

No secret value is ever stored in Git. All sensitive values live in GCP SM or AWS SM and are pulled into K8s Secrets automatically. ESO refreshes every 1 hour by default.

Complete variable reference¶

Secrets from GCP Secret Manager¶

Pulled by the backend-secrets-{dev|prod} ExternalSecret in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml.

Env Var	GCP SM Key	How Generated	Used For
`JWT_SECRET`	`tomoda-jwt-secret`	`openssl rand -hex 32` (auto)	Signs JWT access/refresh tokens
`ENCRYPTION_KEY`	`tomoda-encryption-key`	`openssl rand -hex 16` (auto)	Reserved for future field-level encryption
`DB_PASSWORD`	`tomoda-db-password`	`openssl rand -base64 24` (auto)	Postgres authentication
`REDIS_PASSWORD`	`tomoda-redis-password`	`openssl rand -base64 24` (auto)	Redis authentication
`GOOGLE_CLIENT_ID`	`tomoda-google-client-id`	GCP Console > OAuth 2.0 Client IDs (Web)	Google Sign-In (backend + web)
`GOOGLE_CLIENT_SECRET`	`tomoda-google-client-secret`	Same	Google Sign-In token validation
`GOOGLE_IOS_CLIENT_ID`	`tomoda-google-ios-client-id`	OAuth 2.0 Client (iOS, bundle `com.tomoda.app`)	iOS Google Sign-In
`GOOGLE_ANDROID_CLIENT_ID`	`tomoda-google-android-client-id`	OAuth 2.0 Client (Android, package + SHA-1)	Android Google Sign-In
`GOOGLE_PLACES_API_KEY`	`tomoda-google-places-api-key`	API Key (restrict to Places API New)	Nearby Search, Place Details
`EMAIL_APIKEY`	`tomoda-email-api-key`	Dispatch dashboard > API Keys	OTP, welcome, deactivation emails
`STRIPE_SECRET_KEY`	`tomoda-stripe-secret-key`	Stripe Dashboard > API Keys	Payments, subscriptions
`STRIPE_WEBHOOK_SECRET`	`tomoda-stripe-webhook-secret`	Stripe Dashboard > Webhooks	Webhook signature verification
`LINE_CLIENT_ID`	`tomoda-line-client-id`	LINE Dev Console > Channel ID	LINE Login
`LINE_CLIENT_SECRET`	`tomoda-line-client-secret`	Same	LINE Login token verification
`APPLE_CLIENT_ID`	`tomoda-apple-client-id`	Apple Dev > App ID Bundle ID	Apple Sign-In (iOS native)
`APPLE_SERVICE_ID`	`tomoda-apple-service-id`	Apple Dev > Services IDs	Apple Sign-In (web + Android)
`APPLE_TEAM_ID`	`tomoda-apple-team-id`	Apple Dev > Membership	Apple Sign-In
`APPLE_KEY_ID`	`tomoda-apple-key-id`	Apple Dev > Keys	Apple Sign-In JWT verification
`APPLE_AUTH_KEY_P8`	`tomoda-apple-auth-key-p8`	Apple Dev > Keys > Download .p8 (one-time)	APNs push notifications
`KLIPY_API_KEY`	`tomoda-klipy-api-key`	Klipy dashboard	GIF/sticker/clip search

Observability secrets (GCP Secret Manager)¶

Consumed by Alertmanager (Discord routing), the Cloudflare synthetic Worker (probes + tunneled Loki push), and the synthetic test user seed migration. Created via the observability manual setup — work through that page once before the observability stack can run end-to-end.

GCP SM Key	How Generated	Used For
`tomoda-sentry-dsn`	Sentry project → Settings → Client Keys (DSN)	Embedded in the JS bundle at native + web release-build time; tells `@sentry/react-native` where to send events. Pulled by `frontend/scripts/build-with-secrets.sh` in the tomoda repo
`tomoda-sentry-auth-token`	Sentry → Settings → Account → Auth Tokens (scopes `project:releases` + `org:read`) — name `sourcemap-upload`	Used by the Sentry CLI during the native + web release build to upload source maps. Pulled by `build-with-secrets.sh` — never embedded in the bundle
`tomoda-sentry-grafana-token`	Sentry → Settings → Account → Auth Tokens (scopes `event:read` + `org:read` + `project:read` + `member:read`) — name `grafana-readonly`	Projected via ESO into `monitoring/sentry-grafana-credentials` so Grafana's `grafana-sentry-datasource` plugin can query frontend metrics (issue rate, crash-free %, transaction p50/p95) — see Sentry data source
`tomoda-cloudflare-api-token`	Cloudflare dashboard → Profile → API Tokens (scopes: Zone:DNS:Edit, Account:Workers Scripts:Edit, Account:Cloudflare Tunnel:Edit, Account:Account Settings:Read)	Terraform `cloudflare` provider — DNS, Workers (synthetic monitoring), Tunnel management
`tomoda-alert-webhook-primary`	Discord channel → Webhooks → New Webhook	Alertmanager Discord receiver, Sentry Discord integration, synthetic Worker failure notifications. Provider-portable name — value can later be a Slack/PagerDuty/Opsgenie URL without renaming the secret
`tomoda-cloudflare-tunnel-token`	Cloudflare Zero Trust → Networks → Tunnels → Create tunnel `tomoda-prod-tunnel`	The `cloudflared` Deployment in `monitoring` authenticates to Cloudflare's edge with this token; backs the public `loki-push.tomoda.life` hostname
`tomoda-cloudflare-access-client-id`	Cloudflare Zero Trust → Access → Service Auth → token `synthetic-worker`	Worker → Loki push request includes this in `CF-Access-Client-Id` header so Access lets the request through
`tomoda-cloudflare-access-client-secret`	Same as above (pair)	Worker → Loki push request includes this in `CF-Access-Client-Secret` header
`tomoda-synthetic-probe-password`	`openssl rand -base64 32` (operator picks the value)	Cleartext password the synthetic Worker POSTs to `/api/v1/auth/login`; the seed migration in `backend/internal/database/migrations/` writes the hashed value to the synthetic test user row
`tomoda-github-app-id`	GitHub App settings page (`Tomoda ARC Runners`) — visible at the top after creation	ARC reads this to identify the GitHub App when minting installation tokens. Not strictly secret but kept here for symmetry
`tomoda-github-app-installation-id`	URL of the App's installation page in `tomoda-labs` org settings (numeric path segment after `installations/`)	ARC pairs this with the App ID to mint installation-scoped GitHub tokens
`tomoda-github-app-private-key`	GitHub App settings → Generate a private key (downloads `.pem`)	Full PEM contents. ARC uses this to sign JWTs that exchange for short-lived installation tokens — the actual sensitive credential for the runner pool

Secrets from AWS Secrets Manager¶

Pulled by the s3-uploader-secret ExternalSecret. The AWS SM secret is a single JSON blob; ESO splits it into individual K8s Secret keys via the property selector.

Env Var	AWS SM Key	Source	Used For
`AWS_ACCESS_KEY_ID`	`tomoda-s3-uploader-{env}`	Terraform	S3 asset uploads
`AWS_SECRET_ACCESS_KEY`	`tomoda-s3-uploader-{env}`	Terraform	S3 asset uploads
`S3_BUCKET`	`tomoda-s3-uploader-{env}`	Terraform	`tomoda-assets-{env}`
`S3_REGION`	`tomoda-s3-uploader-{env}`	Terraform	`ap-northeast-1`
`S3_BASE_URL`	`tomoda-s3-uploader-{env}`	Terraform	CloudFront domain (`https://assets[-dev].tomoda.life`)

Non-secret config (ConfigMap)¶

Values that are not sensitive (DB host, ports, URLs) live in the per-env ConfigMap defined alongside the ExternalSecrets:

backend-config-dev in tomoda
backend-config-prod in prod

Env Var	Dev	Prod	Purpose
`ENV`	`dev`	`prod`	Environment label
`DB_HOST`	`postgres-postgresql.data.svc.cluster.local`	`prod-postgres-postgresql.data.svc.cluster.local`	Postgres alias
`DB_PORT`	`5432`	`5432`
`DB_USER`	`tomoda_dev_user`	`tomoda_prod_user`
`DB_NAME`	`tomoda_dev`	`tomoda_prod`
`DB_SSLMODE`	`disable`	`require`
`REDIS_HOST`	`redis-master.data.svc.cluster.local`	`prod-redis-master.data.svc.cluster.local`
`REDIS_PORT`	`6379`	`6379`
`WEBAUTHN_RP_ID`	`api-dev.tomoda.life`	`api.tomoda.life`	Passkey RP
`WEBAUTHN_RP_ORIGIN`	`https://app-dev.tomoda.life`	`https://app.tomoda.life`	Passkey origin
`FRONTEND_URL`	`https://app-dev.tomoda.life`	`https://app.tomoda.life`	Email link base
`GIN_MODE`	`debug`	`release`	Gin framework mode (patched in prod overlay)

Postgres credential coupling¶

The DB_PASSWORD value in GCP SM is consumed by two ExternalSecrets:

backend-secrets-{dev|prod} — exposes it as the backend's DB_PASSWORD env var
postgres-{dev|prod}-credentials-eso — populates the K8s Secret CNPG uses to set the Postgres owner password at initdb time

Both reference tomoda-db-password. Keep them aligned by rotating via the procedure below.

ESO auth secret (one-time)¶

ESO needs AWS credentials to read AWS Secrets Manager. The K8s Secret aws-eso-credentials in the external-secrets namespace holds them.

Key	Source
`access-key`	`terraform output -raw eso_reader_access_key_id`
`secret-access-key`	`terraform output -raw eso_reader_secret_access_key`

See setup step 3 below.

Setup guide¶

Step 1 — GCP secrets (one-time)¶

gcloud auth login
gcloud config set project development-485000

# Option A — bootstrap with empty values, fill manual secrets one by one
./scripts/setup-gcp-secrets.sh
printf 'YOUR_VALUE' | gcloud secrets versions add tomoda-google-client-id \
  --project=development-485000 --data-file=-

# Option B — pass everything via env vars
TOMODA_GOOGLE_CLIENT_ID=... \
TOMODA_GOOGLE_CLIENT_SECRET=GOCSPX-... \
TOMODA_STRIPE_SECRET_KEY=sk_live_... \
TOMODA_EMAIL_APIKEY=eyJhbG... \
./scripts/setup-gcp-secrets.sh

OAuth clients are NOT managed by Terraform

The Google IAP OAuth client used by Argo CD/Dex (and by Grafana's auth-proxy flow) is created manually via the Google Cloud Console. The google_iap_brand and google_iap_client Terraform resources were removed because the underlying APIs are deprecated and being shut down (Jan 19, 2026 / Mar 19, 2026). The client secret and ID are then stored in GCP Secret Manager as tomoda-google-client-secret and tomoda-google-client-id and pulled into the argocd and monitoring namespaces by the ExternalSecret resources in k8s/envs/dev/sys/manifests/external-secrets-config.yaml. See Provisioning the Google OAuth client below for the click path.

Provisioning the Google OAuth client¶

For Argo CD SSO and any future browser-facing SSO flows.

Go to GCP Console → APIs & Services → OAuth consent screen in project development-485000.
Configure the consent screen if not already configured: Internal user type (restricts to tomoda.life Workspace), App name Tomoda DevOps, support email, developer contact email.
Go to APIs & Services → Credentials → + Create credentials → OAuth client ID.
Application type: Web application. Name: Tomoda Argo CD (or similar).
Authorized redirect URIs — add both:
- https://argo-app.tomoda.life/api/dex/callback
- https://auth.tomoda.life/oauth2/callback
Click Create. Copy the client ID and client secret immediately (the secret is only shown once).

Store them in GCP Secret Manager:

printf '$CLIENT_ID' | gcloud secrets versions add tomoda-google-client-id \
    --project=development-485000 --data-file=-

printf '$CLIENT_SECRET' | gcloud secrets versions add tomoda-google-client-secret \
    --project=development-485000 --data-file=-

Force ESO to project the new value into the target namespaces:

kubectl delete externalsecret google-oauth-secret -n argocd
kubectl delete externalsecret google-oauth-secret -n monitoring
# Argo CD recreates both within seconds; ESO re-syncs.

Restart the consumers so they pick up the new value:

kubectl rollout restart deployment/argocd-dex-server -n argocd
kubectl rollout restart deployment/monitoring-grafana -n monitoring   # if Grafana auth is enabled

The same flow works for rotating the client secret — re-do steps 6-9 with a regenerated secret from the Console.

The script's behaviour:

Env var set + differs from current: creates a new version, disables the old
Env var set + matches current: no-op
Env var empty + secret already has a value: leaves it alone
Env var empty + secret empty: auto-generates for JWT/encryption/DB/Redis, leaves blank for manual secrets
--rotate flag: force-regenerates auto-generated secrets regardless

Safe to re-run anytime.

Step 2 — AWS Terraform (one-time per environment)¶

cd infrastructure/aws

# Dev
terraform apply -var="environment=dev"

# Prod
terraform apply -var="environment=prod"

This creates:

The S3 IAM uploader user + access key
The AWS Secrets Manager secret with the credentials
The ESO reader IAM user

Step 3 — ESO auth secret (one-time)¶

cd infrastructure/aws
ESO_KEY=$(terraform output -raw eso_reader_access_key_id)
ESO_SECRET=$(terraform output -raw eso_reader_secret_access_key)

kubectl create secret generic aws-eso-credentials \
  -n external-secrets \
  --from-literal=access-key="$ESO_KEY" \
  --from-literal=secret-access-key="$ESO_SECRET"

Step 4 — Verify¶

# All ExternalSecrets in SecretSynced state
kubectl get externalsecret -A

# Per-env synced K8s Secrets exist
kubectl get secret backend-secrets-dev   -n tomoda
kubectl get secret s3-uploader-secret    -n tomoda
kubectl get secret backend-secrets-prod  -n prod
kubectl get secret s3-uploader-secret    -n prod

How it works¶

Secret refresh¶

ESO refreshes every 1 hour (refreshInterval: 1h). To force an immediate refresh after rotating a value:

kubectl delete externalsecret backend-secrets-dev -n tomoda
# Argo CD recreates the resource within seconds; ESO syncs immediately

Workload Identity¶

The ESO controller's K8s SA is bound (via the iam.gke.io/gcp-service-account annotation) to a GCP service account with roles/secretmanager.secretAccessor. That's how it authenticates to GCP SM without static keys.

For AWS, ESO uses the aws-eso-credentials K8s Secret (created in step 3) because AWS doesn't have a GCP-Workload-Identity-style mechanism reachable from inside GKE.

Rotating secrets¶

App secrets (JWT, OAuth, Stripe, email) — no downtime¶

# 1. New version in GCP SM
printf 'NEW_VALUE' | gcloud secrets versions add tomoda-jwt-secret \
  --project=development-485000 --data-file=-

# 2. Force ESO refresh
kubectl delete externalsecret backend-secrets-prod -n prod

# 3. Restart backend to pick up new env vars
kubectl rollout restart deployment/tomoda-api -n prod
kubectl rollout restart deployment/tomoda-async -n prod

For auto-generated secrets, run ./scripts/setup-gcp-secrets.sh --rotate.

Impact by secret:

Secret	Impact of rotation
`JWT_SECRET`	All active sessions invalidated — users re-login
`ENCRYPTION_KEY`	Currently unused — no impact
`GOOGLE_`, `LINE_`, `APPLE_*`	Next OAuth login uses new key; existing sessions unaffected
`STRIPE_*`	Stripe supports key grace periods; no immediate impact
`EMAIL_APIKEY`	Next email send uses new key
`KLIPY_API_KEY`	Next API call uses new key

DB password rotation — requires brief downtime¶

CNPG's owner password is set from tomoda-db-password via the postgres-{env}-credentials-eso ExternalSecret. Backend reads the same value. Rotation:

# 1. New password in GCP SM
NEW_PW=$(openssl rand -base64 24)
printf "$NEW_PW" | gcloud secrets versions add tomoda-db-password \
  --project=development-485000 --data-file=-

# 2. Force ESO refresh on both consumers
kubectl delete externalsecret backend-secrets-prod        -n prod
kubectl delete externalsecret postgres-prod-credentials-eso -n data

# 3. CNPG will pick up the new credential secret and apply it.
#    Force a primary restart if CNPG hasn't applied it within a minute:
kubectl cnpg restart cluster postgres-prod -n data

# 4. Restart backend to pick up the new env var
kubectl rollout restart deployment/tomoda-api -n prod
kubectl rollout restart deployment/tomoda-async -n prod

Downtime: 1-3 min during the Postgres restart. Backend health checks will fail until Postgres is back.

Redis password rotation — brief disruption¶

Same pattern. Update GCP SM, force ESO refresh, restart Redis, restart backend. Redis data is ephemeral (cache + sessions); users may need to re-login.

File reference¶

File	Purpose
`scripts/setup-gcp-secrets.sh`	Create / rotate GCP SM secrets
`k8s/envs/dev/sys/manifests/external-secrets-config.yaml`	`ClusterSecretStore` for GCP + AWS
`k8s/apps/tomoda/overlays/dev/external-secret.yaml`	Dev `ExternalSecret`s + ConfigMap
`k8s/apps/tomoda/overlays/prod/external-secret.yaml`	Prod `ExternalSecret`s + ConfigMap
`k8s/envs/{dev,prod}/middleware/postgres/manifests/cluster.yaml`	CNPG cluster + per-cluster credential ExternalSecret
`infrastructure/aws/iam_uploader.tf`	S3 IAM user, AWS SM secret, ESO reader IAM user

Manual setup summary¶

Item	Manual?	When
Run `scripts/setup-gcp-secrets.sh`	Yes	Once
Set ~16 GCP secret values	Yes	Once per secret
`terraform apply` in `infrastructure/aws/`	Yes	Once per environment
Create `aws-eso-credentials` K8s secret	Yes	Once
S3 credentials in AWS SM	No	Terraform
JWT / DB / Redis passwords	No	Script auto-generates
K8s Secret resource creation	No	ESO
Secret refresh	No	ESO every 1 hour