Skip to content

Deploy

How a code change reaches dev and prod. The pipeline is fully GitOps: nothing is kubectl apply'd by hand in steady state.

High-level flow

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub (main)
    participant CB as Cloud Build
    participant AR as Artifact Registry
    participant IU as Image Updater
    participant AC as Argo CD
    participant K8s as GKE

    Dev->>GH: git push origin main
    GH->>CB: trigger (manual approval for dev)
    Note over CB: build + test
    CB->>AR: push image (sha-<commit>)
    IU->>AR: poll for new tags
    IU->>GH: commit Kustomize image override
    AC->>GH: detect manifest change
    AC->>K8s: reconcile (rolling update)
    K8s-->>Dev: new pod ready

Dev pipeline

  1. Developer pushes to main in the tomoda app repo.
  2. Cloud Build trigger fires. The dev trigger is configured with a manual approval gate — open the Cloud Build console and approve, or run: 
    gcloud builds approve <BUILD_ID> --project=development-485000
  3. Once approved, Cloud Build builds the container image, runs tests, and pushes the image to Artifact Registry tomoda-dev-repo tagged with the commit SHA.
  4. Argo CD Image Updater polls Artifact Registry every few minutes. When it sees a new SHA matching the configured tag pattern, it edits the Kustomize image override in this devops repo and pushes a commit (under the Image Updater bot account).
  5. Argo CD detects the change to the manifests, syncs the tomoda application, and triggers a rolling update of the backend Deployment in the tomoda namespace.
  6. The new pod must pass its readiness probe before traffic is shifted. The old pod terminates after the new one is healthy.

Prod pipeline

Prod is gated by Git tags, not approvals.

  1. Cut a release tag in the tomoda app repo: git tag vX.Y.Z && git push --tags.
  2. Cloud Build prod trigger fires automatically on tag push — no manual approval (the tag itself is the gate).
  3. Image is built and pushed to Artifact Registry tomoda-prod-repo tagged with the version.
  4. Image Updater picks it up, commits to this repo's k8s/apps/tomoda/overlays/prod/ Kustomize overlay.
  5. Argo CD syncs the prod-tomoda application. A new pod rolls out in the prod namespace.

Health checks and disruption

Every backend pod has readinessProbe and livenessProbe configured against /health. Argo CD waits for the new pod's readiness probe to pass before the rolling update proceeds.

Prod enforces a PodDisruptionBudget with minAvailable: 1. During a rolling update:

  • Argo CD spins up the new replica
  • Waits for readiness
  • Terminates the old replica

Since prod currently runs a single replica (see Scaling to grow it), the PDB means the old pod stays up until the new one is healthy — preventing a brief 503 window mid-deploy.

PDB with single replica

A minAvailable: 1 PDB with replicas: 1 blocks voluntary disruptions (node drains, autoscaler evictions). Before any planned node maintenance, scale the backend to 2 replicas first.

What you commit, what gets generated

Repo What you commit What the pipeline writes
tomoda (app) Code changes Cloud Build builds image
devops (this repo) Manifest changes, infra Image Updater writes Kustomize image SHA

If you change a manifest directly in devops/k8s/apps/tomoda/overlays/dev/, Argo CD syncs it on the next poll (default: 3 min). Force a sync with argocd app sync tomoda if you need it immediately.

Verifying a deploy

# Watch the rollout (both pools share the same image; check whichever rolled)
kubectl rollout status deployment/tomoda-api -n prod
kubectl rollout status deployment/tomoda-async -n prod

# Confirm the running image SHA
kubectl get deploy tomoda-api -n prod -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl get deploy tomoda-async -n prod -o jsonpath='{.spec.template.spec.containers[0].image}'

# Hit the health endpoint (served by tomoda-api via the ingress)
curl https://api.tomoda.life/health

If anything looks wrong, see Rollback.