Scaling¶
The backend pools autoscale horizontally; data services scale by manifest edits. This page is the ops playbook: which manifest, which command, zero-downtime or not.
The app-side companion,
Long-term Scaling & Growth
in the tomoda repo, covers the code seams (DB_READ_HOST, DB_MAX_CONNS, the
ws-hub / api-hub process modes, chat_messages partitioning) and the "why".
Just-in-time playbooks¶
Each step plugs into a seam the backend already ships, so it's a config or deployment change, not a rewrite. Build it when the trigger fires. The early-warning alerts live in k8s/envs/platform/manifests/alerting-rules.yaml (group tomoda.scaling, prod only).
| Step | Trigger | Zero-downtime | Playbook |
|---|---|---|---|
| Read replica | PostgresCPUHigh (primary CPU high under steady read load) |
Yes | below |
| Connection pooler | PostgresConnectionsNearMax (connections near max_connections) |
Yes, rolling | below |
| Dedicated WS tier | HPAPinnedAtMax while CPU looks idle, or PodMemoryNearLimit on api pods |
Yes, rolling | below |
| chat_messages archival | Hot-storage cost of retained history grows | Yes | below |
Read replica¶
A replica needs a second server to be worth it: on one node it shares the disk and CPU with the primary (Adding a server).
- Raise
instancesink8s/envs/prod/postgres/manifests/cluster.yamlto2(1 primary + 1 streaming replica). CNPG creates the replica and servespostgres-prod-ro.data.svc.cluster.local. - Set
DB_READ_HOST: "postgres-prod-ro.data.svc.cluster.local"in the prod ConfigMap ink8s/apps/tomoda/overlays/prod/external-secret.yaml. - Commit; Argo CD syncs and the backend rolls (
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-prodif it didn't).
Zero-downtime: yes. Empty DB_READ_HOST means reads use the primary, so instances: 2 alone gives a warm failover standby.
Connection pooler¶
Trigger: total backend connections (pods × DB_MAX_CONNS, 20 per pod today) approach the primary's max_connections (200).
Add a CNPG PgBouncer Pooler in transaction mode in front of the primary:
# k8s/envs/prod/postgres/manifests/pooler.yaml (new file in the dir Argo syncs)
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
name: postgres-prod-pooler
namespace: data
spec:
cluster:
name: postgres-prod
instances: 2
type: rw
pgbouncer:
poolMode: transaction
parameters:
max_client_conn: "500"
default_pool_size: "40" # server-side conns; keep the total well under max_connections
Then point the backend at it in k8s/apps/tomoda/overlays/prod/external-secret.yaml:
DB_HOST: "postgres-prod-pooler.data.svc.cluster.local"
DB_MAX_CONNS: "10" # per pod; pods × this must fit the pooler, not the DB
DB_POOLER: "true" # pgx CacheDescribe exec mode, no server-side prepared statements
Transaction pooling breaks pgx's default prepared-statement cache; DB_POOLER=true switches it off (backend/internal/database/pgx.go), so the cutover is config-only. Set DB_HOST and DB_POOLER together, dev first.
Zero-downtime: yes, rolling restart.
Dedicated WS tier¶
Trigger: the api HPA sits at maxReplicas while HTTP load looks idle, or api memory rises from per-socket buffers. Both mean WebSocket load the CPU-based HPA can't see.
The backend already ships ws-hub and api-hub modes; this is a deployment split. Manifests, migration sequence and the custom-metric HPA are in Tomoda: WS Pool Split. In short:
- Add a
tomoda-wsDeployment (SERVER_MODE=ws-hub) and Service at 0 replicas; confirm it reconciles. - Scale it to 2 and add its HPA on
tomoda_ws_connections_active(already exposed by prometheus-adapter). - Route the ingress
/wspath to the WS Service. - Flip
SERVER_MODEonbackend-api-deployment.yamlfrommulti-hubtoapi-hub; sockets on api pods close and clients reconnect to the WS pool.
Zero-downtime: yes, rolling. Cross-pod fan-out keeps working across the mixed fleet during the roll.
chat_messages partition archival¶
chat_messages is PARTITION BY RANGE by month; the backend creates upcoming partitions at boot. History is retained by design (deletion is per user, on account deletion, in the app), so there is no prune job and the only ops step is archival.
Trigger: chat_messages storage on the primary costs more than it's worth on hot storage (cnpg_pg_database_size_bytes).
ALTER TABLE chat_messages DETACH PARTITION chat_messages_YYYY_MM(online; writes land in the current month).pg_dumpthe detached table to R2, then drop the local copy.
Zero-downtime: yes.
Backend (api + async pools)¶
| Deployment | Mode | Prod HPA | Dev HPA |
|---|---|---|---|
tomoda-api |
multi-hub (HTTP + WS hub) |
min 2, max 3 | 1 (pinned) |
tomoda-async |
async (worker + scheduler) |
min 2, max 3 | 1 (pinned) |
HPAs target CPU 70% and memory 80%. Prod tomoda-api also scales on WebSocket connections per pod (tomoda_ws_connections_active, target 8000) through prometheus-adapter. Scale-down windows: 300s for api, 600s for async so in-flight tasks finish. Prod max 3 fits the single server; raise it together with server capacity.
Files: k8s/apps/tomoda/overlays/{dev,prod}/hpa-{api,async}.yaml.
Day-to-day HPA inspection¶
kubectl get hpa -n tomoda-prod # TARGETS: current vs target
kubectl describe hpa tomoda-api-hpa -n tomoda-prod # recent scaling events
kubectl top pods -n tomoda-prod -l app=tomoda-api
| Symptom | Cause |
|---|---|
<unknown>/70% in TARGETS |
metrics-server not returning data (kubectl -n kube-system get pods \| grep metrics-server), or the container has no resources.requests |
Stuck at maxReplicas |
Real load; raise the ceiling if the server has room and watch Postgres and Redis next |
| Flapping | Widen behavior.scaleDown.stabilizationWindowSeconds |
Changing bounds or metrics¶
Edit the HPA manifest and commit; never kubectl scale or kubectl edit hpa in prod (Argo CD reverts it). To pin for a planned event, set minReplicas and maxReplicas to the same value. To scale on another series (Asynq queue depth, say), add a rule to k8s/envs/platform/prometheus-adapter/values.yaml first, then reference it as an External metric.
PDB interaction¶
| Env | Policy | File |
|---|---|---|
| prod | minAvailable: 1 |
k8s/apps/tomoda/overlays/prod/pdb.yaml |
| dev | maxUnavailable: 1 |
k8s/apps/tomoda/overlays/dev/pdb.yaml |
Prod runs at least 2 pods per pool, so minAvailable: 1 keeps one serving through a drain. Dev runs 1, where minAvailable: 1 would block every drain, so it uses maxUnavailable: 1. A drain blocked by a PDB says "violates pod disruption budget": wait for replicas, then re-drain.
Switching a PDB between minAvailable and maxUnavailable needs a delete
An in-place update keeps the original field while Argo reports success (disruptionsAllowed stays 0). After such a change, delete the PDB once so Argo recreates it:
kubectl delete pdb -n tomoda-dev tomoda-api-pdb tomoda-async-pdb
kubectl get pdb -n tomoda-dev # ALLOWED DISRUPTIONS should be 1
Postgres (CNPG)¶
Both clusters run one instance.
- CPU / memory: edit
spec.resourcesink8s/envs/{dev,prod}/postgres/manifests/cluster.yaml(prod: 8Gi request, 16Gi limit). CNPG restarts the pod; with one instance expect ~30s of downtime. - Storage:
local-pathvolumes are directories on the server's NVMe;spec.storage.size(prod 150Gi, dev 20Gi) is a budget, not an enforced quota, andresizeInUseVolumes: falsestops CNPG from trying to expand a bound PVC. Growth is bounded by the disk budget. - Read replicas and pooling: Read replica, Connection pooler.
- Location catalog growth: the seed load is the curated catalog (about 1M rows). Long-tail places are added on demand by the app's place harvester, so the table grows with usage rather than in one bulk load. Watch the
locationstable size on the Postgres dashboard.
See Postgres Operations for backup, restore and tuning.
Redis¶
Standalone Bitnami chart, one master, no replicas (k8s/envs/{dev,prod}/redis/values.yaml). It holds cache and the Asynq job queues, so it runs with --maxmemory below the container limit and noeviction: at the cap writes fail rather than jobs being dropped. Raise --maxmemory and master.resources.limits.memory together. See Redis.
Server capacity¶
The cluster is one fixed bare-metal server (64 GB RAM, 16 threads, ~890 GB NVMe) with no autoscaler: pods that don't fit stay Pending. Check headroom before raising requests or replicas:
kubectl describe node | sed -n '/Allocated resources/,/Events/p'
kubectl top node
CPU requests run out first, memory second (see capacity and the disk budget). Real use is far below requests: the node peaks near 9 of 16 threads and 20 GB with CI running, and prod and platform together use about 1 CPU. The reservation is prod launch headroom (Postgres 2 CPU / 8Gi, embeddings 1 CPU / 3Gi, Photon 8Gi, api and async) plus CI.
CI capacity budget¶
CI runners request what a job really uses, so the scheduler reserves it. Requests are also the CPU guarantee: while the sum of requests fits in allocatable, every pod gets at least its requested CPU even with the node saturated, so a CI burst cannot slow prod. Check this before raising any maxRunners.
| Line | CPU | Memory |
|---|---|---|
| Allocatable | 15,000m | 61,537Mi |
| Prod, platform and dev, steady | 6,985m | 29,018Mi |
Prod api and async at HPA max (one extra pod each) |
750m | 512Mi |
| Cron and scan jobs running at once | 300m | 700Mi |
CI at max: tomoda-arc 3, tomoda-arc-db 2 (plus Postgres sidecars), tomoda-arc-trusted 1 |
6,200m | 15,872Mi |
| Total | 14,235m | 46,102Mi |
| Margin | 765m (5%) | 15,435Mi (25%) |
CPU is the binding line: each runner adds 1 CPU, so the current 3 + 2 + 1 caps already use the margin. Do not raise maxRunners until a CPU request is released (for example prod headroom once launch load is known) and the table still shows at least 5% margin. Memory is not the constraint.
Beyond the numbers:
- Runner pods use the
ci-runnerPriorityClass (-1000,preemptionPolicy: Never): when prod needs room the scheduler preempts runners, never the other way round. See ARC runners. - Runner memory limits (6Gi, 4Gi on
tomoda-arc-db, plus 1Gi per Postgres sidecar) sum to about 35 GB at the worst case, below allocatable minus the non-CI working set. - Postgres and the prod app stay Burstable with requests well above their use, so node-pressure eviction and the OOM killer reach CI first. Guaranteed QoS would need CPU limits, which this cluster avoids on services (see OVH).
Adding a server¶
The second machine joins as an agent: two k3s servers are less available than one, because etcd then needs both. HA means three servers, with instances: 2 or more on postgres-prod and higher HPA ceilings. Steps are in Node Lifecycle.
Workloads have no node selectors, so the scheduler spreads them as soon as a node joins. Add soft tomoda.life/role affinity if placement matters.
Cost¶
The server is a flat monthly bill (see Cost monitoring); right-sizing requests changes headroom, not the bill. Grow by adding servers or moving to a larger model.