Decisions¶
ADR-lite log: the load-bearing choices that shape the platform, why, and what we gave up.
Each entry: Decision / Context / Rationale / Trade-offs.
GKE + Argo CD + Kustomize for the control plane¶
Decision. All workloads run on GKE. Every workload is described declaratively in this git repository. Argo CD reconciles cluster state to git, with Kustomize as the templating layer (and Helm for upstream charts).
Context. The alternatives were raw kubectl apply from CI, Helm-only with a release controller (e.g. Flux + Helm), or running everything on managed services (Cloud Run + Cloud SQL + Memorystore).
Rationale. GitOps gives an audit trail by default: every change is a git commit, live state diffable against main any time. Kustomize keeps the overlay model simple (base + per-env patches) without Helm's variable-substitution complexity. Argo's UI is a strong operational asset for a small team.
Trade-offs. Two more systems to learn (Argo, Kustomize). A bad commit to main is a production change with selfHeal: true. Drift detection often pulls the cluster back to git rather than alerting.
In-cluster Postgres via CloudNativePG¶
Decision. Postgres is a CNPG Cluster running inside GKE with the ghcr.io/cloudnative-pg/postgis:17-3.5 image. There is no Cloud SQL instance.
Context. Cloud SQL was the obvious default; CNPG won on cost and operational model.
Rationale. A db-custom-1-3840 Cloud SQL instance is ~an order of magnitude more expensive than the equivalent CNPG pod. CNPG ships PostGIS in its image (we use the geo features). Barman WAL archiving to a GCS bucket (tomoda-db-backups-*) gives off-cluster point-in-time recovery. The manifests run on any Kubernetes, no Cloud SQL API lock-in.
Trade-offs. We own backup verification, version upgrades, failover. CNPG runs on the on-demand data pool (tainted, off spot); lean is single-instance, and the HA switch to a synchronous standby is a deliberate flip. Barman restores aren't as instant as a Cloud SQL clone.
In-cluster Redis via Bitnami Helm chart¶
Decision. Redis runs as a Bitnami standalone Helm release in the data namespace, password auth, ClusterIP only.
Context. Memorystore would have been the managed alternative.
Rationale. Same cost/portability story as Postgres. Redis is never exposed outside the cluster (only backend pods reach it); NetworkPolicy in the data namespace plus the ESO-projected password (see Redis) are the boundary.
Trade-offs. No managed failover, no automatic patching. The PersistentVolumeClaim size has to be increased manually when it fills.
Traefik for cluster ingress¶
Decision. Ingress is Traefik, deployed via the upstream Helm chart, exposed as a GCP external L4 LoadBalancer.
Context. GKE's bundled GCE Ingress (using a GCLB at L7) was the obvious GCP-native alternative, as was the GKE Gateway API implementation.
Rationale. Traefik works identically on any Kubernetes, keeping multi-cloud open. Its middleware ecosystem (oauth2-proxy chains, rate limit, basic-auth) and CRD config map cleanly onto the Argo-managed manifests. cert-manager + Let's Encrypt is straightforward. The L4 LoadBalancer terminates TLS inside the cluster, easier to introspect.
Trade-offs. No GCLB Cloud Armor / WAF for free. One external L4 LB instead of GCE Ingress's HTTPS LB, and we maintain Traefik's config ourselves.
AWS S3 + CloudFront for static assets¶
Decision. Static assets live in S3 buckets (tomoda-assets-{env}) in us-east-2 and are served by CloudFront with assets.tomoda.life / assets-dev.tomoda.life as aliases.
Context. GCS + Cloud CDN sits next to all our other GCP infrastructure and would have been the consistent choice.
Rationale. The assets pipeline predates the GKE setup and was battle-tested before GCP became the primary cloud. CloudFront's caching, OAC, and ACM integration are mature; no operational reason to migrate.
Trade-offs. A second cloud account for a structurally small surface: two IAM systems, two billing dashboards. Documented and stable, so the tax is mostly cognitive.
Cloudflare DNS-only (no orange-cloud proxy)¶
Decision. Cloudflare hosts the tomoda.life zone. Every record is proxied = false. The Cloudflare CDN and WAF are not used.
Context. Cloudflare's proxy mode would give us free DDoS protection and a global edge.
Rationale. CloudFront already fronts the asset path; proxying it through Cloudflare double-caches and complicates invalidation. Proxying the API path adds latency and obscures the real client IP without a trust-header chain. DNS-only lets external-dns and Terraform both manage records cleanly, and origin behaviour is identical to direct.
Trade-offs. No Cloudflare L7 protection. To add it back, do it per-host: leave assets.* unproxied (CloudFront protects it), proxy only api.* / app.*.
Single GCP project for dev and prod¶
Decision. Everything runs in GCP project development-485000. Dev and prod are separated by Kubernetes namespace and by per-env Artifact Registry repos, not by project.
Context. A per-environment project (or dedicated tomoda-prod) is the conventional cloud organization pattern.
Rationale. A small team, one cluster, one VPC, one set of IAM bindings — much simpler operations. Terraform stays in a single state file. Cross-env queries (cost reports, audit logs, billing alerts) work without federation.
Trade-offs. Bigger blast radius: a bad IAM grant or a wrong gcloud config set project can affect prod. Cost attribution between dev and prod relies on labels, not project boundaries.
Multi-zonal ARM node pools + 4-pool split¶
Decision. Nodes spread across three zones (us-central1-a / -b / -f) under a single zonal control plane, and split into four pools by workload class: on-demand system, app, and tainted data, plus a spot burst pool. Every node is ARM (t2a-standard-2). See infrastructure/gcp/gke.tf and GKE.
Context. The conventional GKE default is a single regional pool, or per-workload pools without preemption isolation.
Rationale. Multi-zonal nodes let the autoscaler draw spot capacity from three zones and let multi-replica workloads survive a zone loss, with a zonal (free) control plane. ARM is a separate Compute Engine inventory pool from x86, adding capacity diversity against an x86 spot stockout; all images are multi-arch, so var.node_machine_type flips back to e2-* in one variable. The four-pool split isolates by preemption tolerance: databases and prod app on-demand, only dev + CI on spot, and the data taint keeps everything else off the DB nodes.
Trade-offs. More pools and a per-pool nodeSelector contract. ARM adds a QEMU-emulated arm64 leg to CGO images. Two-tier posture: lean (~$300/mo, single-instance databases) vs resilient HA (~$470/mo, CNPG synchronous standby + Redis Sentinel, data pool across zones). A regional control plane (~$73/mo more) is a further tier, not built.
Dex + Google OAuth for Argo CD SSO¶
Decision. Argo CD's bundled Dex is configured with a Google OAuth connector, restricted to the tomoda.life hosted domain. Default RBAC policy is role:admin. See infrastructure/gcp/argocd.tf.
Context. Argo CD ships with local user auth; SSO is opt-in.
Rationale. Tying access to the company Google Workspace means joiners and leavers are handled by HR, not Argo CD config. hostedDomains: ["tomoda.life"] enforces company-account-only sign-in.
Trade-offs. A Google outage takes down Argo CD login. Default role:admin gives everyone with a tomoda.life account full Argo permissions; tighten with explicit RBAC past a handful of users.
External Secrets Operator for secret distribution¶
Decision. Secrets live in GCP Secret Manager (and a smaller set in AWS Secrets Manager). External Secrets Operator runs in-cluster and materializes them as native Secret objects via ExternalSecret and ClusterSecretStore.
Context. Alternatives: Sealed Secrets (encrypt then commit to git), Bitnami Vault-Secrets, or kubectl create secret by hand.
Rationale. Secrets live in the cloud KMS and never enter git, encrypted or otherwise. ESO picks up Secret Manager rotation automatically (refreshInterval: 1h). Pods reference them like any K8s Secret. Workload Identity removes the operator's bootstrap-auth problem.
Trade-offs. ESO is one more controller to monitor. A bug in ClusterSecretStore config can stall secret materialization cluster-wide.
App-of-apps GitOps pattern¶
Decision. Each environment has a single bootstrap Application manifest (k8s/envs/{dev,prod}/bootstrap.yaml) that, with recurse: true, picks up every other Application in k8s/envs/<env>/{sys,middleware}/. There is no flat manifest list anywhere.
Context. The simpler model is to list every Application explicitly in one file.
Rationale. Adding an app is one application.yaml in the right directory, no central-registry edit — the directory structure is the registry. Pruning is automatic: delete the directory and Argo prunes the resources.
Trade-offs. "What is deployed?" means walking the tree, not reading one file. A typo in a sub-directory's application.yaml can silently fail to render; argocd app list is the source of truth, not the filesystem.