Deployment¶
Backend and web ship together as one version; the native mobile app
versions separately. A merge to main auto-deploys backend + web to dev; prod
deploys when the deploy PR in devops that a Release run opens is merged.
Everything is GitOps, no human kubectl.
End-to-end flow¶
mainvX.Y.Zdev (Xavia)prodStep by step:
- Merge to
main→images.ymlbuilds the changed images automatically (no approval) and pushes:dev-<sha>+:devto GHCR. Argo CD Image Updater rollstomoda-devforward, andota.ymlpublishes the JS bundle toota-dev. - Run Release (below). One run cuts
vX.Y.Z, builds its images and opens the devops deploy PR. - Merge the deploy PR. That is the prod deploy: Argo CD rolls
tomoda-prod(readiness-gated,PodDisruptionBudgetminAvailable: 1), and the same Release run then publishes the prod OTA bundle. - Smoke test. After every sync Argo CD runs a PostSync Job (health, web, auth middleware, and in prod a real login with the synthetic probe account). A failure marks the sync Failed and alerts Discord.
- Rollback. Revert the deploy PR in devops. Migrations are expand/contract, so the previous release works against the current schema (Migrations).
Releasing¶
Run Release¶
Actions → Release → Run workflow, on main:
| Input | Default | Meaning |
|---|---|---|
version |
blank | vX.Y.Z to cut. Blank bumps the highest existing vX.Y.Z tag by bump. The latest existing tag runs the release jobs again without cutting (repair). |
bump |
patch |
patch, minor or major, used when version is blank. |
The run is green when the images are in GHCR and the devops deploy PR is open (or updated). It does not wait for the PR to merge; that is the deploy, and it stays a human decision. Until the merge, ota-gate shows a notice linking the PR and ota-prod is skipped. Merging re-runs ota-gate in the same run, which then publishes prod OTA (Prod OTA).
What each job does¶
| Job | Does | Safe to re-run because |
|---|---|---|
cut |
Runs scripts/release/release.sh: computes the version, stamps unreleased policy versions (scripts/release/stamp-legal-release.sh, committed as legal: stamp policy release date), commits chore: release vX.Y.Z (bumps backend/VERSION + frontend/VERSION) to main, tags it and creates the GitHub Release. Pushes with the release GitHub App token. |
It reuses a tag this run already pushed, skips a commit main already has, and creates the GitHub Release only when missing. |
images |
Calls images.yml with the tag: checks the tag is vX.Y.Z on main, then builds ghcr.io/tomoda-labs/tomoda-{backend,frontend}:vX.Y.Z on GitHub-hosted runners. |
An image already in GHCR is skipped (docker buildx imagetools inspect), so only the missing one rebuilds. |
bump-prod |
After images: pins newTag in k8s/apps/tomoda/overlays/prod/kustomization.yaml and the prod seed-load overlay of tomoda-labs/devops on branch release/tomoda-vX.Y.Z, and opens the deploy(prod): tomoda vX.Y.Z PR. The PR link lands in the run summary. |
The branch is rebuilt from devops main and force-pushed; an open PR for it is updated, never duplicated. If prod already pins the version, it does nothing. |
ota-gate |
After bump-prod: reads devops main and checks that it pins prod to this tag. If so it passes pinned=true; if not, it shows the deploy PR link (notice and run summary) and passes pinned=false. |
It only reads. |
ota-prod |
Only when ota-gate saw the pin: calls ota.yml with the tag, which checks the pin again, runs the frontend tests, then publishes the tag's JS bundle to ota.tomoda.life under both platforms' prod runtimeVersions. |
Publishing the same bundle again is harmless; the newest publish is what binaries fetch. |
- Policy dates: a policy change merges with
**Last updated: unreleased**and an undated manifest entry;cutstamps both with the release day (UTC) so the tag ships dated documents, and fails if any entry stays unreleased. See Auth → Policy acceptance.
bump-prod lives in release.yml rather than images.yml so the image build stays a plain build with no devops credentials, and the deploy PR is a job of its own in the run.
Re-run and repair¶
- A job failed: fix the cause and press Re-run failed jobs on the same run. Every job is idempotent, so the run picks up where it stopped and a re-run never cuts a second version.
- The run is gone or you want a fresh run for a tag: run Release with
versionset to the latest tag.cutsees the tag exists, checks it is onmain, makes sure the GitHub Release exists and skips the cut;images,bump-prod,ota-gateandota-prodrun again for that tag. Only the latest tag is accepted; to go back to an older version, revert its deploy PR. - Prod OTA did not publish after the merge: re-run the
OTA gate vX.Y.Zjob in the Release run. It passes now that devopsmainpins the tag, andota-prodruns. If that run can no longer be re-run, use Actions → OTA → Run workflow with the tag (below). - Force an image rebuild for a tag: delete that version from the GHCR package, then repair.
Release runs only from main; on another ref every job is skipped. cut, bump-prod and ota-gate hold the release App key, so they run on the trusted in-cluster pool tomoda-arc-trusted (no PR jobs, no shared node cache; see ARC runners), as do the prod-pin job of ota.yml and its prod publish run, which reads the prod upload key. The image builds stay on GitHub-hosted runners (they need Docker). Runs share the concurrency group release: a second run waits for the first, and a third queued behind it replaces the second, which is cancelled and has to be started again.
images.yml can also be run on its own (Actions → Images → Run workflow with tag) to build a tag's images without opening a deploy PR.
Deploy and roll back¶
Merging the deploy(prod): tomoda vX.Y.Z PR in devops is the prod deploy. Reverting it rolls prod back to the previous version, and also republishes that version's JS to prod OTA (below). See DevOps → Rollback for the Argo CD UI path and the last-resort kubectl path.
Prod OTA¶
Prod OTA reaches installed apps at once, so it waits for the deploy: it publishes only a tag that devops main pins prod to. Merging the deploy PR triggers it:
- In devops,
notify-prod-deploy.ymlsees the prod pin change onmainand sends aprod-deployedrepository dispatch with the tag to this repo (release App token). - Here,
ota-after-deploy.ymlfinds the Release run onmainwhose job is namedOTA gate vX.Y.Z, waits (checking once a minute, up to 90 minutes) for that run to finish, and re-runs the job. - The gate passes, so
ota-prodpublishes in the same Release run, typically a few minutes after the merge, around when Argo CD finishes the rollout.
If no recent Release run has a gate for the tag, or GitHub refuses the re-run (runs older than 30 days can't be re-run), ota-after-deploy.yml runs ota.yml for the tag directly. Reverting a deploy PR goes through the same path with the older tag, so prod OTA follows the rollback.
To publish by hand, re-run OTA gate vX.Y.Z in the Release run, or use Actions → OTA → Run workflow on main with tag. Both publish only when devops main pins prod to that tag; ota.yml's prod-pin job fails otherwise.
ota-after-deploy.yml runs one at a time per tag. A second dispatch for the same tag while one is queued replaces the queued one; if a gate re-run never happened, re-run the gate by hand. Every ota job is a no-op while the repo variable OTA_ENABLED is not true.
This repo is private on the GitHub Free plan, where environment protection rules (required reviewers, wait timers and custom deployment protection rules) only apply to public repositories. That is why the gate is a job that checks the devops pin and not an environment approval.
Native binaries¶
Native binaries ship separately from this pipeline, and only when native code changes. Store binaries are prod builds made from the release tag after its deploy PR is merged, built and submitted from a developer Mac with fastlane; see Native → Shipping to the stores.
Migrations: expand and contract¶
A release's migrations run while the previous release still serves traffic, and again after a rollback. Every migration keeps the previous release working: split a breaking change into an expand release and a later contract release. task db:migrate:check (also run in CI on every PR) fails on drops, renames, type changes and new NOT NULL columns without a default. See Database → Backward-compatible migrations.
Cutting from a laptop¶
scripts/release/release.sh cuts the tag locally (interactive, or with RELEASE_VERSION / RELEASE_BUMP) and then offers to dispatch the Release workflow for it, which runs in repair mode. Nothing builds on a tag push, so a local cut without that run ships nothing. See Scripts.
One-time setup¶
- Release GitHub App: installed on
tomodaanddevopswith Contents and Pull requests Read and write. Its ID is the variableRELEASE_APP_IDin both repos. Its private key is the Secret Manager secrettomoda-release-app-private-key, whichrelease.yml,ota.ymland devopsnotify-prod-deploy.ymlread onmainas thetomoda-releaseservice account through WIF (variablesGCP_WIF_PROVIDER,GCP_RELEASE_SA); branch, tag and PR runs get no token. Ifmainis protected in either repo, add the App to the bypass list. - GHCR packages: each package (
tomoda-backend,tomoda-frontend) lists this repo under Package settings → Manage Actions access with Admin (Write is enough to push; image retention needs Admin to delete old versions). - OTA: the secret
XAVIA_DEV_UPLOAD_KEY, the variableOTA_ENABLED=true, and the prod upload keytomoda-ota-prod-upload-keyread from Secret Manager the same way (see Native → Enabling OTA).
Where the manifests live¶
The shape of the application in the cluster is defined entirely in the DevOps repo under k8s/apps/tomoda/:
base/— Deployments (backend + frontend), Ingress (Traefik, path-routed/api,/ws→ backend), Service, NetworkPolicy, Kustomizationoverlays/dev/— namespacetomoda-dev, image tag tracked by Image Updateroverlays/prod/— namespacetomoda-prod, image tag pinned in git, PodDisruptionBudget, HPAs
See DevOps → Tomoda K8s app for the manifests, resource limits, replicas, env vars, hostnames, probes, and the canonical health-check definition. Read those manifests, not this page, for current values.
Frontend deploy¶
The frontend Pod (Nginx serving the static Expo web export) is deployed via the same flow. Because EXPO_PUBLIC_* env vars are baked in at build time, any change to public API URLs / OAuth client IDs requires a rebuild, not a config change (see Container images).