Skip to content

Deploy

How a code change reaches dev and prod. The pipeline is fully GitOps: nothing is kubectl apply'd by hand in steady state.

High-level flow (dev)

git push
→
images.yml
push image
GHCR
↓ Image Updater polls for new tags
Image Updater
set image tag
Argo CD
reconcile
k3s rolling swap
Five actors, two halves. The top half (push → build → registry) is the producer side: a merge to the app repo's main builds and lands the dev-* image. The bottom half (poll → set tag → sync → roll) is the consumer side: Image Updater notices the new tag, sets it on the tomoda-dev Application (no git commit), and Argo CD swings dev onto it. Prod never uses Image Updater; see below.

Full release process

Backend/web and the mobile app version independently. A merge auto-deploys to dev; prod deploys when the deploy(prod) PR a Release run opens here is merged; the mobile app ships most changes over-the-air.

merge to main (app repo)
↓ auto
dev (automatic)
images.yml (no approval)
↓ image :dev-sha
Image Updater → Argo CD
↓
k3s tomoda-dev
prod (merge = gate)
Release run (app repo): vX.Y.Z + images
↓ opens
deploy(prod) PR here
↓ merge (the gate)
Argo CD
↓ + prod OTA follows
k3s tomoda-prod
mobile app
JS-only merge → OTA dev
↓ deploy PR merged
OTA prod (Xavia)
↓ native change only
build from the tag → TestFlight / Play
Three lanes. Backend/web: merge → dev auto; a Release run's deploy(prod) PR, once merged → prod (together). Mobile: JS ships over-the-air (dev on merge, prod once the deploy PR is merged); native binaries are built from the release tag only when native code changes.

Dev pipeline (backend / web)

  1. Merge to main in the tomoda app repo.
  2. images.yml (tomoda repo) builds the changed images automatically, with no approval gate, and pushes :dev-<sha> + :dev to GHCR.
  3. Argo CD Image Updater (ImageUpdater tomoda-dev) sees the newest dev-* tag and writes it onto the tomoda-dev Application.
  4. Argo CD syncs and rolls the backend/web Deployment in tomoda-dev. The new pod passes readiness before the old one terminates.

Prod pipeline (backend / web)

Prod's version is pinned in git (newTag in k8s/apps/tomoda/overlays/prod/kustomization.yaml). Backend + web ship together:

  1. Run the Release workflow in the tomoda repo. One run bumps both VERSION files, tags vX.Y.Z, pushes :vX.Y.Z to GHCR and opens a PR here (deploy(prod): tomoda vX.Y.Z) that changes the pinned tag.
  2. Merge the PR. That is the prod deploy. Argo CD syncs tomoda-prod and rolls the Deployments.
  3. Prod OTA follows. .github/workflows/notify-prod-deploy.yml sees the pin change on main and sends a prod-deployed dispatch to the tomoda repo, which re-runs the Release run's OTA gate so it publishes that version's JS to ota.tomoda.life (or runs its ota.yml for the tag when there is no run to re-run). Reverting the PR sends the older tag the same way.
  4. Smoke test. A PostSync Job (k8s/apps/tomoda/base/smoke-test.yaml) checks health, the web app, the auth middleware and, in prod, a real login with the synthetic probe account. A failure marks the sync Failed and fires ArgoCDSyncFailed in Discord; roll back per Rollback.

The release GitHub App opens the PR (Contents and Pull requests write on this repo) and sends the dispatch (Contents write on tomoda); notify-prod-deploy.yml reads its key from Secret Manager as tomoda-release through WIF, on main only (IAM). The tomoda repo's deployment docs cover the Release run.

Migrations

Migrations run when the new pods boot, so the previous release keeps serving against the new schema during the rollout, and after any rollback. The tomoda repo enforces expand/contract migrations in CI (task db:migrate:check); see the app repo's database docs.

Mobile app (native + OTA)

Native binaries are built from the release tag and rarely (only on a native/SDK change); most changes ship over-the-air through the self-hosted Xavia OTA servers. A JS-only merge publishes to ota-dev (internal testers); a release publishes to ota once its deploy PR is merged here. An update reaches only binaries with a matching runtimeVersion, so a native change means a new binary. See the app repo's docs/frontend/native/release.md.

Health checks and disruption

Backend pods use /ready for readiness (it checks Postgres and Redis, so a pod that loses either leaves the Service) and /health for liveness (a data-tier outage doesn't restart pods). A rollout starts the new pod, waits for readiness, then terminates the old one.

Prod runs at least 2 replicas of tomoda-api and tomoda-async (HPA) with a minAvailable: 1 PodDisruptionBudget, so one stays serving through a rollout or a node drain.

What you commit, what gets generated

Repo What you commit What the pipeline writes
tomoda (app) Code changes images.yml builds and pushes images
devops (this repo) Manifest changes, infra Image Updater pins image tags on the Argo Applications (no git commits)

If you change a manifest directly in devops/k8s/apps/tomoda/overlays/dev/, Argo CD syncs it on the next poll (default: 3 min). Force a sync with argocd app sync tomoda-dev if you need it immediately.

Verifying a deploy

# Watch the rollout (both pools share the same image; check whichever rolled)
kubectl rollout status deployment/tomoda-api -n tomoda-prod
kubectl rollout status deployment/tomoda-async -n tomoda-prod

# Confirm the running image SHA
kubectl get deploy tomoda-api -n tomoda-prod -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl get deploy tomoda-async -n tomoda-prod -o jsonpath='{.spec.template.spec.containers[0].image}'

# Hit the health endpoint (served by tomoda-api via the ingress)
curl https://api.tomoda.life/health

If anything looks wrong, see Rollback.