Tomoda¶
The Tomoda application (Go backend plus Expo/React frontend), deployed as a
Kustomize base with dev and prod overlays under k8s/apps/tomoda/. Argo CD
reconciles the overlay output into the tomoda-dev namespace (dev) or
tomoda-prod namespace (prod) of the k3s cluster.
For backend code, schemas, and migrations, see docs/backend/ in the
tomoda repo.
Topology¶
The backend is one image, two Deployments: an API pool serving HTTP +
WebSocket (tomoda-api, mode multi-hub) and an async pool running the
Asynq server + scheduler (tomoda-async, mode async). Mode is selected at
startup via SERVER_MODE. Rationale in the tomoda repo's
docs/architecture/scaling.md and docs/architecture/decisions.md
("single-image-multiple-modes").
NetworkPolicies limit ingress to Traefik, the Prometheus scrape and the
smoke-test Job; tomoda-async takes no traffic beyond the scrape (it pulls
from Redis). See Network policies.
Base (k8s/apps/tomoda/base/)¶
kustomization.yaml aggregates the Deployments, Ingress, NetworkPolicies,
a ServiceMonitor (tomoda-backend) and PodMonitor (tomoda-async), the
ghcr-pull ExternalSecret and the smoke-test Job. Images are
ghcr.io/tomoda-labs/tomoda-{backend,frontend}, tagged dev in the base;
the prod overlay pins the release tag.
backend-api-deployment.yaml¶
Deployment tomoda-api plus the ClusterIP Service backend-service.
- Replicas: 2 (base); the dev overlay pins to 1. Prod HPA 2–3; dev HPA 1–1.
- Mode:
SERVER_MODE=multi-hub: HTTP API + WebSocket Hub. Cross-pod WS fanout goes through NATS (WS_BACKPLANE=nats,NATS_URLwith per-env credentials from the requiredbackend-natsSecret), so there is no ingress session affinity. - Probes: liveness on
/health(15s initial, 30s period); readiness on/ready(10s initial, 10s period, 2s timeout), which pings Postgres and Redis and returns 503 if either fails. Both are served regardless of mode. - Resources: requests
256Mi/500m, memory limit1Gi, no CPU limit (I/O-bound). The CPU request is the HPA utilisation baseline. Dev overrides the CPU request to100m. - Anti-affinity: preferred hostname anti-affinity, so replicas spread across servers once there is more than one; on one server they co-locate.
Service backend-service, selector app: tomoda-api (async pods don't
serve traffic).
backend-async-deployment.yaml¶
Deployment tomoda-async, no Service.
- Replicas: 2 (base); the dev overlay pins to 1. Prod HPA 2–3; dev HPA 1–1.
- Mode:
SERVER_MODE=async: Asynq server + scheduler, both on every replica. The scheduler'sasynq.Uniqueper cron entry dedups enqueues so only one wins per tick; all replicas consume the shared queues. Workers emit realtime events (item.parsed, expired chat messages), so async also setsWS_BACKPLANE=natsand publishes on the same bus the api pods subscribe to. Like api, it readsNATS_URLfrom the requiredbackend-natsSecret. - Probes: same as api (
/healthliveness,/readyreadiness). - Resources: requests
256Mi/250m, memory limit1536Mi, no CPU limit. More memory headroom than api because purge/cleanup tasks spike. Dev overrides the CPU request to50m. - Anti-affinity: same preferred hostname anti-affinity as api.
Every container on both pools sets requests and a memory limit; there are no CPU limits, so bursts aren't throttled.
Both Deployments share the same configuration sources:
backend-secrets(envFrom.secretRef), overlay-renamed tobackend-secrets-devorbackend-secrets-prod. Synced from GCP Secret Manager via ExternalSecrets. Contains JWT, DB and Redis passwords, OAuth client secrets (Google / Apple), payment keys, email API key, GIF search API key.object-storage-credentials(envFrom.secretRef), synced from GCP Secret Manager (tomoda-r2-app-<env>-credentials). Holds the R2 keys (asSTORAGE_ACCESS_KEY_ID/STORAGE_SECRET_ACCESS_KEY),STORAGE_ENDPOINTandSTORAGE_PUBLIC_URLfor user-uploaded media.backend-config-<env>(added by the overlay patch as aconfigMapRef): non-secret runtime config such as DB host/port/user/name, Redis host/port, WebAuthn RP ID, frontend URL,ENV.
The postgres-ca Secret is mounted at /etc/postgres-ca for
DB_SSLMODE=verify-full. PHOTON_URL=http://photon.platform.svc.cluster.local:2322 is set inline. Pods list imagePullSecrets: [ghcr-pull].
frontend-deployment.yaml¶
Single-replica Deployment serving the Expo web bundle on port 8081.
EXPO_PUBLIC_API_URL is set in the base as https://api-dev.tomoda.life.
Liveness and readiness both probe /.
ingress.yaml¶
A single Traefik-class Ingress tomoda-ingress, Cloudflare-proxied, with two
rules in the base: api-dev.tomoda.life routes to backend-service:8080,
app-dev.tomoda.life routes to frontend-service:8081. TLS is issued by
cert-manager via the letsencrypt-prod ClusterIssuer into tomoda-app-tls.
network-policy.yaml¶
Three NetworkPolicies:
tomoda-api-policy: ingress on 8080 fromtraefik-system,monitoringand the smoke-test pod.tomoda-async-policy: ingress on 8080 frommonitoringonly (metrics). Async pods initiate connections to Redis and Postgres; nothing else dials them.frontend-policy: ingress on 8081 fromtraefik-systemand the smoke-test pod.
Dev overlay (k8s/apps/tomoda/overlays/dev/)¶
namespace: tomoda-dev
commonLabels:
env: dev
Adds on top of the base:
backend-secrets-dev: referencesClusterSecretStore/gsm-tomoda(GCP Secret Manager) and maps every key the backend needs from atomoda-*GSM secret. Refresh interval: 1h.object-storage-credentials: referencesgsm-tomodaand templates the R2 keys fromtomoda-r2-app-dev-credentialsplus the R2 endpoint andSTORAGE_PUBLIC_URL=https://assets-dev.tomoda.life. See Secrets Management.backend-config-dev: DB hostpostgres-postgresql.data.svc.cluster.local, DB nametomoda_dev, usertomoda_dev_user,DB_SSLMODE=verify-full; Redis hostredis-dev-master.data.svc.cluster.local; WebAuthn RP IDapi-dev.tomoda.life; frontend URLhttps://app-dev.tomoda.life.ca-sync.yaml: mirrors the CNPG server CA (data/postgres-dev-ca) into the namespace aspostgres-ca, through a Kubernetes-provider SecretStore whose ServiceAccount may read only that one Secret.pdb.yaml:maxUnavailable: 1on both pools, so the single replica can be evicted and a drain doesn't hang.hpa-api.yaml/hpa-async.yaml: bounds 1–1, which keeps the HPA objects but disables scale-out.
JSON-patches rewire envFrom[0] to the dev secret name, append the
ConfigMap, and enable metrics and tracing to Tempo (10% sampling) on both
tomoda-api and tomoda-async. Ingress hostnames stay as the base defaults.
Prod overlay (k8s/apps/tomoda/overlays/prod/)¶
namespace: tomoda-prod
commonLabels:
env: prod
Same shape as dev, plus:
backend-secrets-prod+object-storage-credentials: withtomoda-r2-app-prod-credentials,tomoda-asset-signing-key-prodandSTORAGE_PUBLIC_URL=https://assets.tomoda.life.backend-config-prod: DB hostpostgres-prod-postgresql.data.svc.cluster.local, DB nametomoda_prod, usertomoda_prod_user,DB_SSLMODE=verify-full; Redis hostredis-prod-master.data.svc.cluster.local; WebAuthn RP IDtomoda.life; frontend URLhttps://tomoda.life.pdb.yaml:minAvailable: 1on both pools. With replicas ≥ 2, a node drain keeps one pod of each pool serving (see Scaling).hpa-api.yaml/hpa-async.yaml: CPU 70%, memory 80%, bounds 2–3. Thetomoda-apiHPA also scales on WS connections per pod (tomoda_ws_connections_active, target 8k). The async HPA scales down more slowly (600s window vs api's 300s) so in-flight worker tasks aren't cut short.cloudflare-aop.yaml: Authenticated Origin Pulls. ATLSOptionmakes Traefik require and verify the Cloudflare origin-pull client certificate, so a request that did not come through the Cloudflare edge fails at TLS.
The Kustomize images block pins the release (newTag: vX.Y.Z); the
deploy(prod) PR from the tomoda release bumps it. Patches enable the prod
smoke-test login, set observability, attach the AOP TLSOption to the Ingress
and rewrite its hosts: api.tomoda.life to the backend, and tomoda.life,
app.tomoda.life and www.tomoda.life to the frontend.
Operations¶
- Deploy: Image Updater tracks
dev-*tags for dev; prod moves when adeploy(prod)PR is merged. A PostSync smoke-test Job runs after every sync. See Deploy. - Datastore deps: Postgres and
Redis must exist in the
datanamespace before backend pods become ready. The DSNs are baked into the ConfigMaps above. - Secrets rotation: rotate the value in GSM; ExternalSecrets refreshes within an hour. Pod restart is needed for changes that are read at boot: restart both pools.
- Adding env vars: non-secret defaults go in the overlay's ConfigMap; secrets go in the matching ExternalSecret + the corresponding GSM key. Avoid hardcoding values in the base.
- Scaling: see Operations → Scaling for HPA tuning and playbooks, and WS Pool Split for the planned dedicated WebSocket pool.