Skip to content

Environments

Tomoda runs on a production cluster with a dev tenant: one k3s cluster on one OVH bare-metal server, and three logical tiers: platform (cluster-wide shared services), dev (a deliberately small tenant for testing changes before they hit prod), and prod (the canonical environment, sized for real traffic).

One cluster, three tiers

Layer Value
Server OVH RISE-GAME-2, Beauharnois (BHS)
Cluster k3s, single server (embedded etcd)
Pod / service CIDRs 10.42.0.0/16 / 10.43.0.0/16
Dev app namespace tomoda-dev
Prod app namespace tomoda-prod
Shared data namespace data (both postgres-dev/postgres-prod, redis-dev/redis-prod)
Shared platform namespace platform (Photon, NATS, Xavia OTA, embeddings)
Shared system namespaces argocd, cert-manager, external-secrets, traefik-system, monitoring, arc-systems, etc.

The server and k3s setup are in OVH bare metal.

Bootstrap: the app-of-apps pattern

Argo CD is installed once by scripts/ovh/bootstrap-argocd.sh, which then applies the single bootstrap manifest; everything else flows from git, Argo CD's own release included:

kubectl apply -f k8s/envs/bootstrap.yaml

k8s/envs/bootstrap.yaml creates three Argo Applications:

  • platform recurses into k8s/envs/platform/: Argo CD itself, cert-manager, Traefik, external-dns, external-secrets, oauth2-proxy, monitoring (Prometheus, Grafana, Alertmanager), Loki, Tempo, blackbox-exporter, Photon, NATS and Xavia OTA. Required; everything else depends on it.
  • dev recurses into k8s/envs/dev/: postgres-dev (CNPG, no backups), redis-dev, pgAdmin, Redis Insight and the tomoda dev overlay. Optional; delete it to free dev's footprint.
  • prod recurses into k8s/envs/prod/: postgres-prod (WAL + daily backups to R2), redis-prod, pgAdmin, Redis Insight and the tomoda prod overlay (HPA-driven).
One k3s cluster · OVH Beauharnois
platform tier shared, required · cert-manager, ESO, Traefik, monitoring, …
dev tier small PVCs, no backup, 1 replica
prod tier 150 Gi Postgres, backups to R2, HPA
Three tiers inside one cluster: three Argo CD Applications differing in sizing and backup policy, not separate clusters or projects.

Bootstrapping a fresh cluster

Apply k8s/envs/bootstrap.yaml once. Argo CD owns everything afterward. Don't kubectl apply anything else by hand; change git and let Argo reconcile.

What's shared vs per-environment

Shared (one instance, in k8s/envs/platform/):

  • All system services: cert-manager, Traefik, external-dns, external-secrets, monitoring (Prometheus, Grafana, Alertmanager, Loki, Tempo), oauth2-proxy, blackbox-exporter, alerting rules and dashboards.
  • Photon geocoder, a singleton in platform. Both backends call photon.platform.svc.cluster.local:2322; the planet index is read-only, takes days to rebuild and gives the same answers to either tenant.
  • The Argo CD control plane.
  • GHCR pull credentials (node-level) and GCP Secret Manager access via ESO.

Per-environment (two instances, separated by name):

  • Postgres Cluster (CNPG): postgres-dev (tomoda_dev, no backups) and postgres-prod (tomoda_prod, WAL + daily backups to R2, 30d retention), both in data. See k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml.
  • Redis (Bitnami chart): redis-dev and redis-prod, both in data, so a dev runaway can't degrade prod's queues.
  • The tomoda app: Argo apps tomoda-dev (namespace tomoda-dev, HPA pinned at 1) and tomoda-prod (namespace tomoda-prod, HPA scale-out).
  • pgAdmin and Redis Insight.

Per-environment outside the cluster:

  • Image tags in one GHCR package per image: dev-<sha> / dev vs vX.Y.Z.
  • R2 media buckets and assets Workers: tomoda-assets-dev / assets-dev.tomoda.life vs tomoda-assets-prod / assets.tomoda.life.
  • Xavia OTA buckets: tomoda-ota-dev vs tomoda-ota-prod.

Dev is deliberately small: enough to answer "is this change broken", with memory limits so a runaway can't starve prod. Exact sizing per env is in the Environments matrix.

Promotion

Every change reaches dev first: a merge to tomoda main builds dev-<sha> images and Image Updater rolls tomoda-dev. Prod's tag is pinned in git; a Release run opens a deploy(prod) PR here, merging it is the prod gate and reverting it is the rollback. Details in Deploy.

Trade-offs

The single-cluster model keeps cost and operational complexity low: one server, one Argo CD UI, one Photon install. The cost is a shared blast radius: a server failure, an Argo CD outage, or a buggy ClusterRole affects both tenants at once, and both share one Traefik.

The growth path is a second server joining the same cluster (Postgres standby, workloads spread), then a third for an HA control plane. Moving dev to its own cluster only needs the bootstrap pointed elsewhere; the overlays already separate everything.