Skip to content

Decisions

ADR-lite log: the load-bearing choices that shape the platform, why, and what we gave up.

Each entry: Decision / Context / Rationale / Trade-offs.


OVH bare metal, k3s and R2

Decision. The cluster runs on one OVH Rise bare-metal server in Beauharnois with k3s. All object storage is Cloudflare R2. GCP keeps only Secret Manager, the Terraform state bucket, Google OAuth clients and the Places API, managed in infrastructure/gcp/. See OVH bare metal.

Context. Managed Kubernetes cost several times more for far less memory, and the API ran CPU-throttled under small CPU limits.

Rationale. About CAD 134/month (RISE-GAME-2) buys 64 GB, 16 dedicated Zen 3 threads and 2× 960 GB NVMe, close to Canadian users. Argo CD, Kustomize, CNPG, Traefik and the Cloudflare edge are portable to any Kubernetes. R2 has no egress fees, which matters for the 83 GB Photon index and for restores.

Trade-offs. One machine is a single point of failure until a second server joins (RAID1 covers disks only). We run the k3s control plane and OS patching ourselves. Memory is the first limit, sized for about 100k MAU.


Argo CD + Kustomize GitOps

Decision. Every workload is described declaratively in this repository. Argo CD reconciles cluster state to git, with Kustomize as the templating layer (and Helm for upstream charts).

Context. The alternatives were raw kubectl apply from CI, or Helm-only with a release controller (e.g. Flux + Helm).

Rationale. GitOps gives an audit trail by default: every change is a git commit, and live state is diffable against main at any time. Kustomize keeps the overlay model simple (base + per-env patches). Argo's UI is a strong operational asset for a small team.

Trade-offs. Two more systems to learn (Argo, Kustomize). A bad commit to main is a production change with selfHeal: true. Drift detection pulls the cluster back to git rather than alerting.

In-cluster Postgres via CloudNativePG

Decision. Postgres is a CNPG Cluster running in the cluster on local NVMe, with our ghcr.io/tomoda-labs/tomoda-postgres image (PostGIS, PGroonga, pgvector). There is no managed database.

Context. A managed Postgres was the alternative.

Rationale. A managed instance is about an order of magnitude more expensive than the equivalent CNPG pod, and PGroonga rules out most managed offerings. Barman WAL archiving to R2 (tomoda-db-backups) gives off-cluster point-in-time recovery. The manifests run on any Kubernetes.

Trade-offs. We own backup verification, version upgrades, failover. Prod runs a single instance until a second server exists; the HA switch to a standby is a deliberate flip. Restores from Barman are slower than a managed clone.


In-cluster Redis via Bitnami Helm chart

Decision. Redis runs as a Bitnami standalone Helm release in the data namespace, password auth, ClusterIP only.

Context. A managed Redis was the alternative.

Rationale. Same cost/portability story as Postgres. Redis is never exposed outside the cluster (only backend pods reach it); NetworkPolicy in the data namespace plus the ESO-projected password (see Redis) are the boundary.

Trade-offs. No managed failover, no automatic patching. The PersistentVolumeClaim size has to be increased manually when it fills.


Traefik for cluster ingress

Decision. Ingress is Traefik from the upstream Helm chart, a LoadBalancer Service bound to the node's public IP by k3s ServiceLB with externalTrafficPolicy: Local. TLS terminates at Traefik with cert-manager certificates.

Context. The alternatives were ingress-nginx or the Gateway API.

Rationale. Traefik works the same on any Kubernetes. Its middleware (oauth2-proxy chains, rate limits, basic-auth) and CRDs fit the Argo-managed manifests. Trusting forwarded headers only from Cloudflare's ranges keeps the real client IP for proxied hosts.

Trade-offs. We maintain Traefik's config ourselves, and edge protection comes from the Cloudflare proxy, not the ingress.

Media on R2 behind Workers

Decision. User media lives in private R2 buckets (tomoda-assets-{env}), served by the assets Workers with signed URLs. See Cloudflare.

Context. The alternative was a separate object store and CDN on another cloud account.

Rationale. R2 has no egress fees and the Workers sit on the same edge that already fronts tomoda.life. One fewer account means one fewer IAM system and bill.

Trade-offs. Media delivery depends on Cloudflare alongside DNS and ingress.

Cloudflare proxy for app hosts

Decision. The app hostnames are proxied through Cloudflare (external-dns.alpha.kubernetes.io/cloudflare-proxied on the Ingress), and prod is origin-locked with authenticated origin pulls. Admin UIs stay DNS-only. See Cloudflare.

Context. The server's public IP is otherwise directly reachable, and media already runs on Cloudflare Workers.

Rationale. The proxy gives DDoS protection, WAF and edge TLS at no cost, and origin pulls stop clients from bypassing it in prod.

Trade-offs. The edge sits in the request path for every app call, and client IPs reach the cluster only through trusted forwarded headers.

Single GCP project for dev and prod

Decision. The GCP footprint (Secret Manager, the Terraform state bucket, Google OAuth clients, the Places API) lives in one project, development-485000, for both envs. Dev and prod are separated by Kubernetes namespace and by -dev / -prod secrets, not by project.

Context. A per-environment project (or dedicated tomoda-prod) is the conventional cloud organization pattern.

Rationale. A small team and one set of IAM bindings keep operations simple, and billing for the Places API stays in one place.

Trade-offs. Bigger blast radius: a bad IAM grant can expose prod secrets. Places API cost isn't split between dev and prod.


Dex + Google OAuth for Argo CD SSO

Decision. Argo CD's bundled Dex is configured with a Google OAuth connector, restricted to the tomoda.life hosted domain. Default RBAC policy is role:readonly with an email allow-list for role:admin. See k8s/envs/platform/argocd/values.yaml.

Context. Argo CD ships with local user auth; SSO is opt-in.

Rationale. Tying access to the company Google Workspace means joiners and leavers are handled by HR, not Argo CD config. hostedDomains: ["tomoda.life"] enforces company-account-only sign-in.

Trade-offs. A Google outage takes down Argo CD login. Admins are listed by email in policy.csv, so promoting someone is a values change.


External Secrets Operator for secret distribution

Decision. Secrets live in GCP Secret Manager. External Secrets Operator runs in-cluster and materializes them as native Secret objects via ExternalSecret and the gsm-tomoda ClusterSecretStore.

Context. Alternatives: encrypted secrets committed to git, a self-hosted vault, or kubectl create secret by hand.

Rationale. Secrets live in the cloud KMS and never enter git, encrypted or otherwise. ESO picks up Secret Manager rotation automatically (refreshInterval: 1h). Pods reference them like any K8s Secret. ESO authenticates with one read-only service account key, the only hand-made Secret besides Argo CD's repo credential.

Trade-offs. ESO is one more controller to monitor. A bug in ClusterSecretStore config can stall secret materialization cluster-wide.


App-of-apps GitOps pattern

Decision. One bootstrap manifest (k8s/envs/bootstrap.yaml) creates the platform, dev and prod Applications, each recursing (recurse: true) into k8s/envs/<tier>/ to pick up every application.yaml there. There is no flat manifest list.

Context. The simpler model is to list every Application explicitly in one file.

Rationale. Adding an app is one application.yaml in the right directory, no central-registry edit: the directory structure is the registry. Pruning is automatic: delete the directory and Argo prunes the resources.

Trade-offs. "What is deployed?" means walking the tree, not reading one file. A typo in a sub-directory's application.yaml can silently fail to render; argocd app list is the source of truth, not the filesystem.