Skip to content

Tomoda

The Tomoda application (Go backend plus Expo/React frontend), deployed as a Kustomize base with dev and prod overlays under k8s/apps/tomoda/. Argo CD reconciles the overlay output into the tomoda-dev namespace (dev) or tomoda-prod namespace (prod) of the k3s cluster.

For backend code, schemas, and migrations, see docs/backend/ in the tomoda repo.

Topology

The backend is one image, two Deployments: an API pool serving HTTP + WebSocket (tomoda-api, mode multi-hub) and an async pool running the Asynq server + scheduler (tomoda-async, mode async). Mode is selected at startup via SERVER_MODE. Rationale in the tomoda repo's docs/architecture/scaling.md and docs/architecture/decisions.md ("single-image-multiple-modes").

tomoda binary one image, two roles via SERVER_MODE
↙   ↘
tomoda-api SERVER_MODE=multi-hub · HTTP + WS
tomoda-async SERVER_MODE=async · Asynq server + scheduler
Each pool gets its own HPA, Service (or none), and NetworkPolicy. Same image, no rebuild between them.

NetworkPolicies limit ingress to Traefik, the Prometheus scrape and the smoke-test Job; tomoda-async takes no traffic beyond the scrape (it pulls from Redis). See Network policies.

Base (k8s/apps/tomoda/base/)

kustomization.yaml aggregates the Deployments, Ingress, NetworkPolicies, a ServiceMonitor (tomoda-backend) and PodMonitor (tomoda-async), the ghcr-pull ExternalSecret and the smoke-test Job. Images are ghcr.io/tomoda-labs/tomoda-{backend,frontend}, tagged dev in the base; the prod overlay pins the release tag.

backend-api-deployment.yaml

Deployment tomoda-api plus the ClusterIP Service backend-service.

  • Replicas: 2 (base); the dev overlay pins to 1. Prod HPA 2–3; dev HPA 1–1.
  • Mode: SERVER_MODE=multi-hub: HTTP API + WebSocket Hub. Cross-pod WS fanout goes through NATS (WS_BACKPLANE=nats, NATS_URL with per-env credentials from the required backend-nats Secret), so there is no ingress session affinity.
  • Probes: liveness on /health (15s initial, 30s period); readiness on /ready (10s initial, 10s period, 2s timeout), which pings Postgres and Redis and returns 503 if either fails. Both are served regardless of mode.
  • Resources: requests 256Mi / 500m, memory limit 1Gi, no CPU limit (I/O-bound). The CPU request is the HPA utilisation baseline. Dev overrides the CPU request to 100m.
  • Anti-affinity: preferred hostname anti-affinity, so replicas spread across servers once there is more than one; on one server they co-locate.

Service backend-service, selector app: tomoda-api (async pods don't serve traffic).

backend-async-deployment.yaml

Deployment tomoda-async, no Service.

  • Replicas: 2 (base); the dev overlay pins to 1. Prod HPA 2–3; dev HPA 1–1.
  • Mode: SERVER_MODE=async: Asynq server + scheduler, both on every replica. The scheduler's asynq.Unique per cron entry dedups enqueues so only one wins per tick; all replicas consume the shared queues. Workers emit realtime events (item.parsed, expired chat messages), so async also sets WS_BACKPLANE=nats and publishes on the same bus the api pods subscribe to. Like api, it reads NATS_URL from the required backend-nats Secret.
  • Probes: same as api (/health liveness, /ready readiness).
  • Resources: requests 256Mi / 250m, memory limit 1536Mi, no CPU limit. More memory headroom than api because purge/cleanup tasks spike. Dev overrides the CPU request to 50m.
  • Anti-affinity: same preferred hostname anti-affinity as api.

Every container on both pools sets requests and a memory limit; there are no CPU limits, so bursts aren't throttled.

Both Deployments share the same configuration sources:

  1. backend-secrets (envFrom.secretRef), overlay-renamed to backend-secrets-dev or backend-secrets-prod. Synced from GCP Secret Manager via ExternalSecrets. Contains JWT, DB and Redis passwords, OAuth client secrets (Google / Apple), payment keys, email API key, GIF search API key.
  2. object-storage-credentials (envFrom.secretRef), synced from GCP Secret Manager (tomoda-r2-app-<env>-credentials). Holds the R2 keys (as STORAGE_ACCESS_KEY_ID / STORAGE_SECRET_ACCESS_KEY), STORAGE_ENDPOINT and STORAGE_PUBLIC_URL for user-uploaded media.
  3. backend-config-<env> (added by the overlay patch as a configMapRef): non-secret runtime config such as DB host/port/user/name, Redis host/port, WebAuthn RP ID, frontend URL, ENV.

The postgres-ca Secret is mounted at /etc/postgres-ca for DB_SSLMODE=verify-full. PHOTON_URL=http://photon.platform.svc.cluster.local:2322 is set inline. Pods list imagePullSecrets: [ghcr-pull].

frontend-deployment.yaml

Single-replica Deployment serving the Expo web bundle on port 8081. EXPO_PUBLIC_API_URL is set in the base as https://api-dev.tomoda.life. Liveness and readiness both probe /.

ingress.yaml

A single Traefik-class Ingress tomoda-ingress, Cloudflare-proxied, with two rules in the base: api-dev.tomoda.life routes to backend-service:8080, app-dev.tomoda.life routes to frontend-service:8081. TLS is issued by cert-manager via the letsencrypt-prod ClusterIssuer into tomoda-app-tls.

network-policy.yaml

Three NetworkPolicies:

  • tomoda-api-policy: ingress on 8080 from traefik-system, monitoring and the smoke-test pod.
  • tomoda-async-policy: ingress on 8080 from monitoring only (metrics). Async pods initiate connections to Redis and Postgres; nothing else dials them.
  • frontend-policy: ingress on 8081 from traefik-system and the smoke-test pod.

Dev overlay (k8s/apps/tomoda/overlays/dev/)

namespace: tomoda-dev
commonLabels:
  env: dev

Adds on top of the base:

  • backend-secrets-dev: references ClusterSecretStore/gsm-tomoda (GCP Secret Manager) and maps every key the backend needs from a tomoda-* GSM secret. Refresh interval: 1h.
  • object-storage-credentials: references gsm-tomoda and templates the R2 keys from tomoda-r2-app-dev-credentials plus the R2 endpoint and STORAGE_PUBLIC_URL=https://assets-dev.tomoda.life. See Secrets Management.
  • backend-config-dev: DB host postgres-postgresql.data.svc.cluster.local, DB name tomoda_dev, user tomoda_dev_user, DB_SSLMODE=verify-full; Redis host redis-dev-master.data.svc.cluster.local; WebAuthn RP ID api-dev.tomoda.life; frontend URL https://app-dev.tomoda.life.
  • ca-sync.yaml: mirrors the CNPG server CA (data/postgres-dev-ca) into the namespace as postgres-ca, through a Kubernetes-provider SecretStore whose ServiceAccount may read only that one Secret.
  • pdb.yaml: maxUnavailable: 1 on both pools, so the single replica can be evicted and a drain doesn't hang.
  • hpa-api.yaml / hpa-async.yaml: bounds 1–1, which keeps the HPA objects but disables scale-out.

JSON-patches rewire envFrom[0] to the dev secret name, append the ConfigMap, and enable metrics and tracing to Tempo (10% sampling) on both tomoda-api and tomoda-async. Ingress hostnames stay as the base defaults.

Prod overlay (k8s/apps/tomoda/overlays/prod/)

namespace: tomoda-prod
commonLabels:
  env: prod

Same shape as dev, plus:

  • backend-secrets-prod + object-storage-credentials: with tomoda-r2-app-prod-credentials, tomoda-asset-signing-key-prod and STORAGE_PUBLIC_URL=https://assets.tomoda.life.
  • backend-config-prod: DB host postgres-prod-postgresql.data.svc.cluster.local, DB name tomoda_prod, user tomoda_prod_user, DB_SSLMODE=verify-full; Redis host redis-prod-master.data.svc.cluster.local; WebAuthn RP ID tomoda.life; frontend URL https://tomoda.life.
  • pdb.yaml: minAvailable: 1 on both pools. With replicas ≥ 2, a node drain keeps one pod of each pool serving (see Scaling).
  • hpa-api.yaml / hpa-async.yaml: CPU 70%, memory 80%, bounds 2–3. The tomoda-api HPA also scales on WS connections per pod (tomoda_ws_connections_active, target 8k). The async HPA scales down more slowly (600s window vs api's 300s) so in-flight worker tasks aren't cut short.
  • cloudflare-aop.yaml: Authenticated Origin Pulls. A TLSOption makes Traefik require and verify the Cloudflare origin-pull client certificate, so a request that did not come through the Cloudflare edge fails at TLS.

The Kustomize images block pins the release (newTag: vX.Y.Z); the deploy(prod) PR from the tomoda release bumps it. Patches enable the prod smoke-test login, set observability, attach the AOP TLSOption to the Ingress and rewrite its hosts: api.tomoda.life to the backend, and tomoda.life, app.tomoda.life and www.tomoda.life to the frontend.

Operations

  • Deploy: Image Updater tracks dev-* tags for dev; prod moves when a deploy(prod) PR is merged. A PostSync smoke-test Job runs after every sync. See Deploy.
  • Datastore deps: Postgres and Redis must exist in the data namespace before backend pods become ready. The DSNs are baked into the ConfigMaps above.
  • Secrets rotation: rotate the value in GSM; ExternalSecrets refreshes within an hour. Pod restart is needed for changes that are read at boot: restart both pools.
  • Adding env vars: non-secret defaults go in the overlay's ConfigMap; secrets go in the matching ExternalSecret + the corresponding GSM key. Avoid hardcoding values in the base.
  • Scaling: see Operations → Scaling for HPA tuning and playbooks, and WS Pool Split for the planned dedicated WebSocket pool.