Skip to content

GKE Cluster

The single Kubernetes cluster behind everything Tomoda runs. Provisioned by infrastructure/gcp/gke.tf and infrastructure/gcp/gke_iam.tf.

Cluster shape

Field Value
Name gke-tomoda
Control plane us-central1-a (single zonal, free)
Node zones us-central1-a / -b / -f (multi-zonal, node_locations)
Node arch ARM (t2a-standard-4 general, t2a-standard-2 burst); x86 (e2-standard-2) on data
Network gke-tomoda-vpc (custom)
Subnetwork gke-tomoda-subnet
Pod range pods secondary range (10.1.0.0/16)
Services range services secondary range (10.2.0.0/20)
Workload Identity Enabled (${project_id}.svc.id.goog)
Dataplane V2 (datapath_provider = "ADVANCED_DATAPATH")
Default node pool Removed (remove_default_node_pool = true)
Deletion protection On

Dataplane V2 is create-time only

datapath_provider = "ADVANCED_DATAPATH" (GKE Dataplane V2) uses eBPF networking and enforces Kubernetes NetworkPolicy. It cannot be changed on a live cluster; flipping it forces a cluster replacement. See Network policies.

Deletion protection

deletion_protection = true. terraform destroy fails until you flip the bit in gke.tf and apply. Losing the cluster means losing every workload, every PVC, and the ability to restore CNPG without a manual DR flow.

The cluster name gke-tomoda is load-bearing (CI substitutions, kubeconfig contexts, runbooks). A Terraform rename force-replaces the cluster; don't rename without a coordinated rollout.

Topology: multi-zonal ARM nodes

  • Control plane — single-zonal in us-central1-a (free).
  • Nodes — multi-zonal: every pool sets node_locations = var.node_zones (us-central1-a / -b / -f), so the autoscaler draws from three zones. A single-zone spot stockout doesn't block scheduling; multi-replica workloads survive a zone loss. Zones a/b/f are the ones offering ARM (T2A).
  • Arch — ARM (t2a-standard-2, 2 vCPU / 8 GB) on all pools except data, which is x86 (e2-standard-2). ARM is a separate Compute Engine inventory pool from x86, adding capacity diversity against an x86 spot stockout. data stays x86 because the custom Postgres image builds on cloudnative-pg/postgis (amd64-only upstream). All app/platform images are multi-arch; set var.node_machine_type to an e2-* type to move ARM pools to x86.
  • Autoscaling counts — multi-zonal pools use total_min_node_count / total_max_node_count (cluster-wide totals spread across zones), not per-zone min/max.

Node pools

Three pools, all under the dedicated node service account gke-node-sa (google_service_account.gke_nodes, wired via node_config.service_account on each pool in gke.tf), not the default Compute Engine SA. It is least-privilege, holding only roles/logging.logWriter, roles/monitoring.metricWriter, and roles/artifactregistry.reader. See IAM Overview.

Pool label Mode Machine Total autoscale Carries
pool=general on-demand t2a-standard-4 1–3 Control plane, Traefik, Argo CD, operators, monitoring + prod app (api + async), Photon, embeddings, xavia-prod
pool=data on-demand, tainted workload=stateful:NoSchedule e2-standard-2 (x86) 1–3 CNPG + Redis (prod + dev)
pool=burst spot, location_policy = ANY t2a-standard-2 0–5 Dev app, CI runners, photon-indexer, xavia-dev
  • pool=general — the single on-demand pool for everything except stateful data: control-plane and observability (Traefik, Argo CD, ARC controller + listener, cloudflared, monitoring, cert-manager / external-secrets / external-dns / oauth2-proxy) alongside the user-facing tier (prod api + async, Photon's planet index at 6 Gi / 250 Gi PVC / ~1.8h to rebuild, embeddings, xavia-prod). Never spot. Lean start is one larger node (floor 1); BALANCED spreads the pool across zones as it grows.
  • pool=data — on-demand, x86, tainted so only CNPG + Redis (tolerating workload=stateful) land here. A preempted database is an outage, so never spot. Lean floor is 1; the HA flavour bumps the ceiling for a synchronous CNPG standby across zones.
  • pool=burst — the only spot pool. location_policy = ANY grabs whichever zone has spot capacity, surviving a single-zone stockout. Scales to zero when idle.

Cost posture

Posture Monthly Survives
Lean (runs today) ~$300/mo Node preemption, single-zone spot stockout
Resilient HA ~$470/mo Zone loss (multi-replica DB + app across zones)

The resilient upgrade raises the data pool ceiling and turns on the CNPG synchronous standby + Redis Sentinel (see Postgres, Redis). A regional control plane (survives a control-plane zone outage) adds ~$73/mo — a further tier, not built.

Scaling the cluster

Every pool has a GKE cluster autoscaler (autoscaling {} block in gke.tf): grows on unschedulable pods, shrinks on underutilized nodes, replaces preempted spot nodes.

The cluster runs the OPTIMIZE_UTILIZATION autoscaling profile (cluster_autoscaling in gke.tf). The default BALANCED profile refuses to drain any node hosting a local-storage (emptyDir) pod — which every node does (kube-dns, metrics-server, GMP collectors) — so an underutilized node never becomes a scale-down candidate and pools stay stuck at their high-water mark. OPTIMIZE_UTILIZATION raises the scale-down utilization threshold and evicts those pods, so pools consolidate back down. Trade-off: more aggressive scale-down means more frequent pod reschedules, so multi-replica workloads carry PDBs. (Node auto-provisioning stays off — enabled = false; pools are managed explicitly.)

  • Higher ceiling — bump total_max_node_count in gke.tf and terraform apply. Don't hand-edit node counts.
  • Pod-level — add an HPA against the Deployment. HPA scales replicas; the autoscaler provisions nodes when capacity is exhausted.
  • One-off bursts (e.g. Photon indexer via scripts/photon-index-local.sh) — spin up a separate Compute Engine VM rather than scaling the cluster, to keep Argo CD's drift detection happy.

Workload Identity

Workload Identity is enabled with the pool ${project_id}.svc.id.goog. Every Workload Identity binding in this project follows the pattern:

serviceAccount:${project_id}.svc.id.goog[<namespace>/<ksa-name>]

The actual bindings used by workloads in the cluster are listed in IAM Overview. The relevant ones for this page:

  • The node service account gke-node-sa gets roles/artifactregistry.reader so nodes can pull images from both repos without per-pool config (gke.tf).
  • Per-workload GCP SAs (CNPG backup, photon-indexer, image-updater, cloud-build) are documented on their own pages.

What is not in here

  • No private cluster — control plane endpoint is public; access relies on GCP auth, not network ACLs.
  • No release channel — release_channel unset, so the cluster follows the default channel. Declare RAPID / REGULAR / STABLE for predictable version upgrades.
  • No maintenance window — upgrades happen whenever GCP decides. Watch the GKE console for advance notice.
  • No GPU pools. The only taint is workload=stateful:NoSchedule on pool=data. Every other pool is untainted; placement is by nodeSelector on pool.

Recovery

If the cluster is somehow lost (deletion-protection bypassed, project deleted, etc.):

  1. Restore the cluster with terraform apply from clean state.
  2. Re-bootstrap Argo CD from the repo's argocd/ manifests.
  3. Restore Postgres from the most recent CNPG base backup in the backup bucket; step-by-step in the Postgres operations runbook.

No automated DR — recovery is a manual replay of Terraform + Argo CD + the Postgres restore.