Deploy¶
How a code change reaches dev and prod. The pipeline is fully GitOps: nothing is kubectl apply'd by hand in steady state.
High-level flow (dev)¶
main builds and lands the dev-* image. The bottom half (poll → set tag → sync → roll) is the consumer side: Image Updater notices the new tag, sets it on the tomoda-dev Application (no git commit), and Argo CD swings dev onto it. Prod never uses Image Updater; see below.Full release process¶
Backend/web and the mobile app version independently. A merge auto-deploys
to dev; prod deploys when the deploy(prod) PR a Release run opens here is
merged; the mobile app ships most changes over-the-air.
main (app repo)vX.Y.Z + imagesdeploy(prod) PR heredevprod (Xavia)deploy(prod) PR, once merged → prod (together). Mobile: JS ships over-the-air (dev on merge, prod once the deploy PR is merged); native binaries are built from the release tag only when native code changes.Dev pipeline (backend / web)¶
- Merge to
mainin the tomoda app repo. images.yml(tomoda repo) builds the changed images automatically, with no approval gate, and pushes:dev-<sha>+:devto GHCR.- Argo CD Image Updater (
ImageUpdater tomoda-dev) sees the newestdev-*tag and writes it onto thetomoda-devApplication. - Argo CD syncs and rolls the backend/web
Deploymentintomoda-dev. The new pod passes readiness before the old one terminates.
Prod pipeline (backend / web)¶
Prod's version is pinned in git (newTag in k8s/apps/tomoda/overlays/prod/kustomization.yaml). Backend + web ship together:
- Run the Release workflow in the tomoda repo. One run bumps both
VERSIONfiles, tagsvX.Y.Z, pushes:vX.Y.Zto GHCR and opens a PR here (deploy(prod): tomoda vX.Y.Z) that changes the pinned tag. - Merge the PR. That is the prod deploy. Argo CD syncs
tomoda-prodand rolls the Deployments. - Prod OTA follows.
.github/workflows/notify-prod-deploy.ymlsees the pin change onmainand sends aprod-deployeddispatch to the tomoda repo, which re-runs the Release run's OTA gate so it publishes that version's JS toota.tomoda.life(or runs itsota.ymlfor the tag when there is no run to re-run). Reverting the PR sends the older tag the same way. - Smoke test. A PostSync Job (
k8s/apps/tomoda/base/smoke-test.yaml) checks health, the web app, the auth middleware and, in prod, a real login with the synthetic probe account. A failure marks the sync Failed and firesArgoCDSyncFailedin Discord; roll back per Rollback.
The release GitHub App opens the PR (Contents and Pull requests write on this repo) and sends the dispatch (Contents write on tomoda); notify-prod-deploy.yml reads its key from Secret Manager as tomoda-release through WIF, on main only (IAM). The tomoda repo's deployment docs cover the Release run.
Migrations¶
Migrations run when the new pods boot, so the previous release keeps serving against the new schema during the rollout, and after any rollback. The tomoda repo enforces expand/contract migrations in CI (task db:migrate:check); see the app repo's database docs.
Mobile app (native + OTA)¶
Native binaries are built from the release tag and rarely (only on a native/SDK
change); most changes ship over-the-air through the self-hosted
Xavia OTA servers. A JS-only merge publishes to
ota-dev (internal testers); a release publishes to ota once its deploy PR is
merged here. An update reaches only binaries with a matching runtimeVersion, so
a native change means a new binary. See the app repo's docs/frontend/native/release.md.
Health checks and disruption¶
Backend pods use /ready for readiness (it checks Postgres and Redis, so a pod that loses either leaves the Service) and /health for liveness (a data-tier outage doesn't restart pods). A rollout starts the new pod, waits for readiness, then terminates the old one.
Prod runs at least 2 replicas of tomoda-api and tomoda-async (HPA) with a minAvailable: 1 PodDisruptionBudget, so one stays serving through a rollout or a node drain.
What you commit, what gets generated¶
| Repo | What you commit | What the pipeline writes |
|---|---|---|
tomoda (app) |
Code changes | images.yml builds and pushes images |
devops (this repo) |
Manifest changes, infra | Image Updater pins image tags on the Argo Applications (no git commits) |
If you change a manifest directly in devops/k8s/apps/tomoda/overlays/dev/, Argo CD syncs it on the next poll (default: 3 min). Force a sync with argocd app sync tomoda-dev if you need it immediately.
Verifying a deploy¶
# Watch the rollout (both pools share the same image; check whichever rolled)
kubectl rollout status deployment/tomoda-api -n tomoda-prod
kubectl rollout status deployment/tomoda-async -n tomoda-prod
# Confirm the running image SHA
kubectl get deploy tomoda-api -n tomoda-prod -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl get deploy tomoda-async -n tomoda-prod -o jsonpath='{.spec.template.spec.containers[0].image}'
# Hit the health endpoint (served by tomoda-api via the ingress)
curl https://api.tomoda.life/health
If anything looks wrong, see Rollback.