Environments¶
Tomoda runs on a production cluster with a dev tenant: one k3s cluster on one OVH bare-metal server, and three logical tiers: platform (cluster-wide shared services), dev (a deliberately small tenant for testing changes before they hit prod), and prod (the canonical environment, sized for real traffic).
One cluster, three tiers¶
| Layer | Value |
|---|---|
| Server | OVH RISE-GAME-2, Beauharnois (BHS) |
| Cluster | k3s, single server (embedded etcd) |
| Pod / service CIDRs | 10.42.0.0/16 / 10.43.0.0/16 |
| Dev app namespace | tomoda-dev |
| Prod app namespace | tomoda-prod |
| Shared data namespace | data (both postgres-dev/postgres-prod, redis-dev/redis-prod) |
| Shared platform namespace | platform (Photon, NATS, Xavia OTA, embeddings) |
| Shared system namespaces | argocd, cert-manager, external-secrets, traefik-system, monitoring, arc-systems, etc. |
The server and k3s setup are in OVH bare metal.
Bootstrap: the app-of-apps pattern¶
Argo CD is installed once by scripts/ovh/bootstrap-argocd.sh, which then applies the single bootstrap manifest; everything else flows from git, Argo CD's own release included:
kubectl apply -f k8s/envs/bootstrap.yaml
k8s/envs/bootstrap.yaml creates three Argo Applications:
platformrecurses intok8s/envs/platform/: Argo CD itself, cert-manager, Traefik, external-dns, external-secrets, oauth2-proxy, monitoring (Prometheus, Grafana, Alertmanager), Loki, Tempo, blackbox-exporter, Photon, NATS and Xavia OTA. Required; everything else depends on it.devrecurses intok8s/envs/dev/:postgres-dev(CNPG, no backups),redis-dev, pgAdmin, Redis Insight and the tomoda dev overlay. Optional; delete it to free dev's footprint.prodrecurses intok8s/envs/prod/:postgres-prod(WAL + daily backups to R2),redis-prod, pgAdmin, Redis Insight and the tomoda prod overlay (HPA-driven).
Bootstrapping a fresh cluster
Apply k8s/envs/bootstrap.yaml once. Argo CD owns everything afterward. Don't kubectl apply anything else by hand; change git and let Argo reconcile.
What's shared vs per-environment¶
Shared (one instance, in k8s/envs/platform/):
- All system services: cert-manager, Traefik, external-dns, external-secrets, monitoring (Prometheus, Grafana, Alertmanager, Loki, Tempo), oauth2-proxy, blackbox-exporter, alerting rules and dashboards.
- Photon geocoder, a singleton in
platform. Both backends callphoton.platform.svc.cluster.local:2322; the planet index is read-only, takes days to rebuild and gives the same answers to either tenant. - The Argo CD control plane.
- GHCR pull credentials (node-level) and GCP Secret Manager access via ESO.
Per-environment (two instances, separated by name):
- Postgres
Cluster(CNPG):postgres-dev(tomoda_dev, no backups) andpostgres-prod(tomoda_prod, WAL + daily backups to R2, 30d retention), both indata. Seek8s/envs/{dev,prod}/postgres/manifests/cluster.yaml. - Redis (Bitnami chart):
redis-devandredis-prod, both indata, so a dev runaway can't degrade prod's queues. - The tomoda app: Argo apps
tomoda-dev(namespacetomoda-dev, HPA pinned at 1) andtomoda-prod(namespacetomoda-prod, HPA scale-out). - pgAdmin and Redis Insight.
Per-environment outside the cluster:
- Image tags in one GHCR package per image:
dev-<sha>/devvsvX.Y.Z. - R2 media buckets and assets Workers:
tomoda-assets-dev/assets-dev.tomoda.lifevstomoda-assets-prod/assets.tomoda.life. - Xavia OTA buckets:
tomoda-ota-devvstomoda-ota-prod.
Dev is deliberately small: enough to answer "is this change broken", with memory limits so a runaway can't starve prod. Exact sizing per env is in the Environments matrix.
Promotion¶
Every change reaches dev first: a merge to tomoda main builds dev-<sha> images and Image Updater rolls tomoda-dev. Prod's tag is pinned in git; a Release run opens a deploy(prod) PR here, merging it is the prod gate and reverting it is the rollback. Details in Deploy.
Trade-offs¶
The single-cluster model keeps cost and operational complexity low: one server, one Argo CD UI, one Photon install. The cost is a shared blast radius: a server failure, an Argo CD outage, or a buggy ClusterRole affects both tenants at once, and both share one Traefik.
The growth path is a second server joining the same cluster (Postgres standby, workloads spread), then a third for an HA control plane. Moving dev to its own cluster only needs the bootstrap pointed elsewhere; the overlays already separate everything.