Skip to content

Tomoda

The Tomoda application — Go backend plus Expo/React frontend — deployed as a Kustomize base with dev and prod overlays under k8s/apps/tomoda/. Argo CD reconciles the overlay output into the tomoda-dev namespace (dev) or prod namespace (prod) of the single GKE cluster.

For backend code, schemas, and migrations, see docs/backend/ in the tomoda repo.

Topology

The backend is one image, two Deployments: an API pool serving HTTP + WebSocket (tomoda-api, mode multi-hub) and an async pool running the Asynq server + scheduler (tomoda-async, mode async). Mode is selected at startup via SERVER_MODE. Rationale in the tomoda repo's docs/architecture/scaling.md and docs/architecture/decisions.md ("single-image-multiple-modes").

tomoda binary one image, two roles via SERVER_MODE
↙   ↘
tomoda-api SERVER_MODE=multi-hub · HTTP + WS
tomoda-async SERVER_MODE=async · Asynq server + scheduler
Each pool gets its own HPA, Service (or none), and NetworkPolicy. Same image, no rebuild between them.

NetworkPolicies restrict ingress on every pod to the traefik-system namespace (api / frontend) or to nothing at all (tomoda-async, which has no inbound traffic — it pulls from Redis). See Network policies.

Base (k8s/apps/tomoda/base/)

kustomization.yaml aggregates five manifests and declares the dev image refs (tomoda-backend:latest / tomoda-frontend:latest from tomoda-dev-repo). Overlays rewrite the image registry for prod.

backend-api-deployment.yaml

Deployment tomoda-api plus the ClusterIP Service backend-service.

  • Replicas: 2 (base); the dev overlay pins to 1 (its HPA floor), so dev doesn't hold a second node via anti-affinity. Prod HPA 2–3; dev HPA 1–1.
  • Mode: SERVER_MODE=multi-hub — HTTP API + WebSocket Hub. Cross-pod WS fanout via Redis pub/sub on chat:event:* (no ingress session affinity).
  • Probes: /health — liveness (15s initial, 30s period), readiness (10s initial, 10s period). Served regardless of mode.
  • Resources: requests 64Mi / 150m, limits 512Mi / 500m (I/O-bound). The 150m CPU request keeps HPA utilisation stable.
  • Anti-affinity: required hostname anti-affinity in the base, so 2 replicas land on distinct hosts (and zones); a single node/zone loss can't take out both.

Service backend-service, selector app: tomoda-api (async pods don't serve traffic).

backend-async-deployment.yaml

Deployment tomoda-async — no Service.

  • Replicas: 2 (base); the dev overlay pins to 1 (its HPA floor), so dev doesn't hold a second node via anti-affinity. Prod HPA 2–3; dev HPA 1–1.
  • Mode: SERVER_MODE=async — Asynq server + scheduler, both on every replica. The scheduler's asynq.Unique per cron entry dedups enqueues so only one wins per tick; all replicas consume the shared queues.
  • Probes: same /health endpoint.
  • Resources: requests 128Mi / 150m, limits 768Mi / 500m — more memory headroom than api because purge/cleanup tasks spike.
  • Anti-affinity: same required hostname anti-affinity as api.

Every container on both pools sets requests and limits.

Both Deployments share the same configuration sources:

  1. backend-secrets (envFrom.secretRef) — overlay-renamed to backend-secrets-dev or backend-secrets-prod. Synced from GCP Secret Manager via ExternalSecrets. Contains JWT, encryption, DB and Redis passwords, OAuth client secrets (Google / LINE / Apple), Stripe keys, email API key, KLIPY API key.
  2. s3-uploader-secret (envFrom.secretRef) — synced from AWS Secrets Manager. Holds the AWS access key, bucket name, region, and base URL for user-uploaded media.
  3. backend-config-<env> (added by the overlay patch as a configMapRef) — non-secret runtime config: DB host/port/user/name, Redis host/port, WebAuthn RP ID, frontend URL, ENV.

PHOTON_URL=http://photon.platform.svc.cluster.local:2322 is set inline.

frontend-deployment.yaml

Single-replica Deployment serving the Expo web bundle on port 8081. EXPO_PUBLIC_API_URL is baked into the base as https://api-dev.tomoda.life; prod uses a different image (rebuilt against https://api.tomoda.life), not a runtime env override. Liveness and readiness both probe /.

ingress.yaml

A single Traefik-class Ingress with two rules in the base — api-dev.tomoda.life routes to backend-service:8080, app-dev.tomoda.life routes to frontend-service:8081. TLS is terminated by cert-manager via the letsencrypt-prod ClusterIssuer; the certificate is stored in tomoda-app-tls.

network-policy.yaml

Three NetworkPolicies:

  • tomoda-api-policy — ingress from traefik-system only, port 8080.
  • tomoda-async-policy — no ingress rules (default-deny). Async pods initiate connections to Redis and Postgres; nothing dials them.
  • frontend-policy — ingress from traefik-system only, port 8081.

Dev overlay (k8s/apps/tomoda/overlays/dev/)

namespace: tomoda
commonLabels:
  env: dev

Adds two ExternalSecret resources, one ConfigMap, and two HPAs on top of the base:

  • backend-secrets-dev — references ClusterSecretStore/gsm-tomoda (GCP Secret Manager) and maps every key the backend needs from a tomoda-* GSM secret. Refresh interval: 1h.
  • s3-uploader-secret — references the namespace-scoped SecretStore/aws-sm-dev (AWS Secrets Manager) and pulls the tomoda-s3-uploader-dev JSON, splitting it into the AWS env vars the backend reads. The store's own key is projected from GCP SM via aws-eso-credentials-dev — see Secrets Management.
  • backend-config-dev — DB host postgres-postgresql.data.svc.cluster.local, DB name tomoda_dev, user tomoda_dev_user, DB_SSLMODE=disable; Redis host redis-dev-master.data.svc.cluster.local; WebAuthn RP ID api-dev.tomoda.life; frontend URL https://app-dev.tomoda.life.
  • hpa-api.yaml / hpa-async.yaml — CPU 70%, memory 80%, bounds 1–3 (dev gets less traffic; lower floor saves cost).

JSON-patches rewire envFrom[0] to the dev secret name and append the ConfigMap as an extra envFrom entry, on both tomoda-api and tomoda-async. A further patch sets nodeSelector: {pool: burst} on both pools (dev tolerates spot preemption). Ingress hostnames stay as the base defaults.

Prod overlay (k8s/apps/tomoda/overlays/prod/)

namespace: prod
commonLabels:
  env: prod

Adds five extra resources and rewrites images plus the Ingress:

  • backend-secrets-prod + s3-uploader-secret — same shape as dev, but the S3 ESO pulls tomoda-s3-uploader-prod and uses the prod tenant's own SecretStore/aws-sm-prod. App secrets still come through the cluster-wide gsm-tomoda store; only the source secret names + the per-env AWS store differ.
  • backend-config-prod — DB host postgres-prod-postgresql.data.svc.cluster.local, DB name tomoda_prod, user tomoda_prod_user, DB_SSLMODE=require; Redis host redis-prod-master.data.svc.cluster.local; WebAuthn RP ID api.tomoda.life; frontend URL https://app.tomoda.life.
  • pdb.yaml — two PodDisruptionBudgets (tomoda-api, tomoda-async), minAvailable: 1 in prod. With replicas ≥ 2, voluntary disruption (node drain, GKE upgrade) keeps one pod of each pool serving throughout. The dev overlay uses maxUnavailable: 1 instead so its single replica can be evicted and idle nodes can scale down (see scaling).
  • hpa-api.yaml / hpa-async.yaml — CPU 70%, memory 80%, bounds 2–3. The tomoda-api HPA also scales on WS connections per pod (tomoda_ws_connections_active, target 8k). The async HPA scales down more slowly (600s window vs api's 300s) so in-flight worker tasks aren't cut short. See Scaling for tuning.

The Kustomize images block points both images at tomoda-prod-repo. Patches set nodeSelector: {pool: app} on both backend pools (on-demand, off spot), flip imagePullPolicy to IfNotPresent, and rewrite the Ingress to add api.tomoda.life, app.tomoda.life, and a www.tomoda.life rule to the frontend.

Operations

  • Deploy — Argo CD Image Updater bumps the :latest tag on dev sync; prod deploys are gated on Cloud Build firing from a semver Git tag. See Argo CD.
  • Datastore deps — Postgres and Redis must exist in the data namespace before backend pods become ready. The DSNs are baked into the ConfigMaps above.
  • Secrets rotation — rotate the value in GSM or AWS SM; ExternalSecrets refreshes within an hour. Pod restart is needed for changes that are read at boot — restart both pools.
  • Adding env vars — non-secret defaults go in the overlay's ConfigMap; secrets go in the matching ExternalSecret + the corresponding GSM key. Avoid hardcoding values in the base.
  • Scaling — see Operations → Scaling for HPA tuning, playbooks, and the trigger criteria for splitting WS into its own pool (design doc).