Skip to content

Scaling

The backend pools autoscale horizontally; data services scale by manifest edits. This page is the ops playbook: which manifest, which command, zero-downtime or not.

The app-side companion, Long-term Scaling & Growth in the tomoda repo, covers the code seams (DB_READ_HOST, DB_MAX_CONNS, the ws-hub / api-hub process modes, chat_messages partitioning) and the "why".

Just-in-time playbooks

Each step plugs into a seam the backend already ships, so it's a config or deployment change, not a rewrite. Build it when the trigger fires. The early-warning alerts live in k8s/envs/platform/manifests/alerting-rules.yaml (group tomoda.scaling, prod only).

Step Trigger Zero-downtime Playbook
Read replica PostgresCPUHigh (primary CPU high under steady read load) Yes below
Connection pooler PostgresConnectionsNearMax (connections near max_connections) Yes, rolling below
Dedicated WS tier HPAPinnedAtMax while CPU looks idle, or PodMemoryNearLimit on api pods Yes, rolling below
chat_messages archival Hot-storage cost of retained history grows Yes below

Read replica

A replica needs a second server to be worth it: on one node it shares the disk and CPU with the primary (Adding a server).

  1. Raise instances in k8s/envs/prod/postgres/manifests/cluster.yaml to 2 (1 primary + 1 streaming replica). CNPG creates the replica and serves postgres-prod-ro.data.svc.cluster.local.
  2. Set DB_READ_HOST: "postgres-prod-ro.data.svc.cluster.local" in the prod ConfigMap in k8s/apps/tomoda/overlays/prod/external-secret.yaml.
  3. Commit; Argo CD syncs and the backend rolls (kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-prod if it didn't).

Zero-downtime: yes. Empty DB_READ_HOST means reads use the primary, so instances: 2 alone gives a warm failover standby.

Connection pooler

Trigger: total backend connections (pods × DB_MAX_CONNS, 20 per pod today) approach the primary's max_connections (200).

Add a CNPG PgBouncer Pooler in transaction mode in front of the primary:

# k8s/envs/prod/postgres/manifests/pooler.yaml (new file in the dir Argo syncs)
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
  name: postgres-prod-pooler
  namespace: data
spec:
  cluster:
    name: postgres-prod
  instances: 2
  type: rw
  pgbouncer:
    poolMode: transaction
    parameters:
      max_client_conn: "500"
      default_pool_size: "40"   # server-side conns; keep the total well under max_connections

Then point the backend at it in k8s/apps/tomoda/overlays/prod/external-secret.yaml:

DB_HOST: "postgres-prod-pooler.data.svc.cluster.local"
DB_MAX_CONNS: "10"     # per pod; pods × this must fit the pooler, not the DB
DB_POOLER: "true"      # pgx CacheDescribe exec mode, no server-side prepared statements

Transaction pooling breaks pgx's default prepared-statement cache; DB_POOLER=true switches it off (backend/internal/database/pgx.go), so the cutover is config-only. Set DB_HOST and DB_POOLER together, dev first.

Zero-downtime: yes, rolling restart.

Dedicated WS tier

Trigger: the api HPA sits at maxReplicas while HTTP load looks idle, or api memory rises from per-socket buffers. Both mean WebSocket load the CPU-based HPA can't see.

The backend already ships ws-hub and api-hub modes; this is a deployment split. Manifests, migration sequence and the custom-metric HPA are in Tomoda: WS Pool Split. In short:

  1. Add a tomoda-ws Deployment (SERVER_MODE=ws-hub) and Service at 0 replicas; confirm it reconciles.
  2. Scale it to 2 and add its HPA on tomoda_ws_connections_active (already exposed by prometheus-adapter).
  3. Route the ingress /ws path to the WS Service.
  4. Flip SERVER_MODE on backend-api-deployment.yaml from multi-hub to api-hub; sockets on api pods close and clients reconnect to the WS pool.

Zero-downtime: yes, rolling. Cross-pod fan-out keeps working across the mixed fleet during the roll.

chat_messages partition archival

chat_messages is PARTITION BY RANGE by month; the backend creates upcoming partitions at boot. History is retained by design (deletion is per user, on account deletion, in the app), so there is no prune job and the only ops step is archival.

Trigger: chat_messages storage on the primary costs more than it's worth on hot storage (cnpg_pg_database_size_bytes).

  • ALTER TABLE chat_messages DETACH PARTITION chat_messages_YYYY_MM (online; writes land in the current month).
  • pg_dump the detached table to R2, then drop the local copy.

Zero-downtime: yes.

Backend (api + async pools)

Deployment Mode Prod HPA Dev HPA
tomoda-api multi-hub (HTTP + WS hub) min 2, max 3 1 (pinned)
tomoda-async async (worker + scheduler) min 2, max 3 1 (pinned)

HPAs target CPU 70% and memory 80%. Prod tomoda-api also scales on WebSocket connections per pod (tomoda_ws_connections_active, target 8000) through prometheus-adapter. Scale-down windows: 300s for api, 600s for async so in-flight tasks finish. Prod max 3 fits the single server; raise it together with server capacity.

Files: k8s/apps/tomoda/overlays/{dev,prod}/hpa-{api,async}.yaml.

Day-to-day HPA inspection

kubectl get hpa -n tomoda-prod                          # TARGETS: current vs target
kubectl describe hpa tomoda-api-hpa -n tomoda-prod      # recent scaling events
kubectl top pods -n tomoda-prod -l app=tomoda-api
Symptom Cause
<unknown>/70% in TARGETS metrics-server not returning data (kubectl -n kube-system get pods \| grep metrics-server), or the container has no resources.requests
Stuck at maxReplicas Real load; raise the ceiling if the server has room and watch Postgres and Redis next
Flapping Widen behavior.scaleDown.stabilizationWindowSeconds

Changing bounds or metrics

Edit the HPA manifest and commit; never kubectl scale or kubectl edit hpa in prod (Argo CD reverts it). To pin for a planned event, set minReplicas and maxReplicas to the same value. To scale on another series (Asynq queue depth, say), add a rule to k8s/envs/platform/prometheus-adapter/values.yaml first, then reference it as an External metric.

PDB interaction

Env Policy File
prod minAvailable: 1 k8s/apps/tomoda/overlays/prod/pdb.yaml
dev maxUnavailable: 1 k8s/apps/tomoda/overlays/dev/pdb.yaml

Prod runs at least 2 pods per pool, so minAvailable: 1 keeps one serving through a drain. Dev runs 1, where minAvailable: 1 would block every drain, so it uses maxUnavailable: 1. A drain blocked by a PDB says "violates pod disruption budget": wait for replicas, then re-drain.

Switching a PDB between minAvailable and maxUnavailable needs a delete

An in-place update keeps the original field while Argo reports success (disruptionsAllowed stays 0). After such a change, delete the PDB once so Argo recreates it:

kubectl delete pdb -n tomoda-dev tomoda-api-pdb tomoda-async-pdb
kubectl get pdb -n tomoda-dev      # ALLOWED DISRUPTIONS should be 1

Postgres (CNPG)

Both clusters run one instance.

  • CPU / memory: edit spec.resources in k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml (prod: 8Gi request, 16Gi limit). CNPG restarts the pod; with one instance expect ~30s of downtime.
  • Storage: local-path volumes are directories on the server's NVMe; spec.storage.size (prod 150Gi, dev 20Gi) is a budget, not an enforced quota, and resizeInUseVolumes: false stops CNPG from trying to expand a bound PVC. Growth is bounded by the disk budget.
  • Read replicas and pooling: Read replica, Connection pooler.
  • Location catalog growth: the seed load is the curated catalog (about 1M rows). Long-tail places are added on demand by the app's place harvester, so the table grows with usage rather than in one bulk load. Watch the locations table size on the Postgres dashboard.

See Postgres Operations for backup, restore and tuning.

Redis

Standalone Bitnami chart, one master, no replicas (k8s/envs/{dev,prod}/redis/values.yaml). It holds cache and the Asynq job queues, so it runs with --maxmemory below the container limit and noeviction: at the cap writes fail rather than jobs being dropped. Raise --maxmemory and master.resources.limits.memory together. See Redis.

Server capacity

The cluster is one fixed bare-metal server (64 GB RAM, 16 threads, ~890 GB NVMe) with no autoscaler: pods that don't fit stay Pending. Check headroom before raising requests or replicas:

kubectl describe node | sed -n '/Allocated resources/,/Events/p'
kubectl top node

CPU requests run out first, memory second (see capacity and the disk budget). Real use is far below requests: the node peaks near 9 of 16 threads and 20 GB with CI running, and prod and platform together use about 1 CPU. The reservation is prod launch headroom (Postgres 2 CPU / 8Gi, embeddings 1 CPU / 3Gi, Photon 8Gi, api and async) plus CI.

CI capacity budget

CI runners request what a job really uses, so the scheduler reserves it. Requests are also the CPU guarantee: while the sum of requests fits in allocatable, every pod gets at least its requested CPU even with the node saturated, so a CI burst cannot slow prod. Check this before raising any maxRunners.

Line CPU Memory
Allocatable 15,000m 61,537Mi
Prod, platform and dev, steady 6,985m 29,018Mi
Prod api and async at HPA max (one extra pod each) 750m 512Mi
Cron and scan jobs running at once 300m 700Mi
CI at max: tomoda-arc 3, tomoda-arc-db 2 (plus Postgres sidecars), tomoda-arc-trusted 1 6,200m 15,872Mi
Total 14,235m 46,102Mi
Margin 765m (5%) 15,435Mi (25%)

CPU is the binding line: each runner adds 1 CPU, so the current 3 + 2 + 1 caps already use the margin. Do not raise maxRunners until a CPU request is released (for example prod headroom once launch load is known) and the table still shows at least 5% margin. Memory is not the constraint.

Beyond the numbers:

  • Runner pods use the ci-runner PriorityClass (-1000, preemptionPolicy: Never): when prod needs room the scheduler preempts runners, never the other way round. See ARC runners.
  • Runner memory limits (6Gi, 4Gi on tomoda-arc-db, plus 1Gi per Postgres sidecar) sum to about 35 GB at the worst case, below allocatable minus the non-CI working set.
  • Postgres and the prod app stay Burstable with requests well above their use, so node-pressure eviction and the OOM killer reach CI first. Guaranteed QoS would need CPU limits, which this cluster avoids on services (see OVH).

Adding a server

The second machine joins as an agent: two k3s servers are less available than one, because etcd then needs both. HA means three servers, with instances: 2 or more on postgres-prod and higher HPA ceilings. Steps are in Node Lifecycle.

Workloads have no node selectors, so the scheduler spreads them as soon as a node joins. Add soft tomoda.life/role affinity if placement matters.

Cost

The server is a flat monthly bill (see Cost monitoring); right-sizing requests changes headroom, not the bill. Grow by adding servers or moving to a larger model.