Skip to content

Deployment

Backend and web ship together as one version; the native mobile app versions separately. A merge to main auto-deploys backend + web to dev; prod deploys when the deploy PR in devops that a Release run opens is merged. Everything is GitOps, no human kubectl.

End-to-end flow

merge to main
↓ auto (no approval)
dev · automatic
images.yml (backend + web)
↓ image :dev-sha → GHCR
Argo Image Updater → Argo CD
↓
k3s tomoda-dev
prod · Release run is the gate
Release run → tag vX.Y.Z
↓ same run
images → devops deploy PR → merge → Argo
↓
k3s tomoda-prod
mobile app
JS merge → OTA dev (Xavia)
↓ Release run
OTA prod
↓ native change only
build from the tag → TestFlight / Play
Three lanes. Backend/web deploy to dev on merge and to prod together from one Release run. The mobile app ships JS over-the-air (dev server on merge, prod server from the Release run); native binaries are built from the release tag only when native code changes.

Step by step:

  1. Merge to main → images.yml builds the changed images automatically (no approval) and pushes :dev-<sha> + :dev to GHCR. Argo CD Image Updater rolls tomoda-dev forward, and ota.yml publishes the JS bundle to ota-dev.
  2. Run Release (below). One run cuts vX.Y.Z, builds its images and opens the devops deploy PR.
  3. Merge the deploy PR. That is the prod deploy: Argo CD rolls tomoda-prod (readiness-gated, PodDisruptionBudget minAvailable: 1), and the same Release run then publishes the prod OTA bundle.
  4. Smoke test. After every sync Argo CD runs a PostSync Job (health, web, auth middleware, and in prod a real login with the synthetic probe account). A failure marks the sync Failed and alerts Discord.
  5. Rollback. Revert the deploy PR in devops. Migrations are expand/contract, so the previous release works against the current schema (Migrations).

Releasing

Run Release

Actions → Release → Run workflow, on main:

Input Default Meaning
version blank vX.Y.Z to cut. Blank bumps the highest existing vX.Y.Z tag by bump. The latest existing tag runs the release jobs again without cutting (repair).
bump patch patch, minor or major, used when version is blank.

The run is green when the images are in GHCR and the devops deploy PR is open (or updated). It does not wait for the PR to merge; that is the deploy, and it stays a human decision. Until the merge, ota-gate shows a notice linking the PR and ota-prod is skipped. Merging re-runs ota-gate in the same run, which then publishes prod OTA (Prod OTA).

What each job does

Job Does Safe to re-run because
cut Runs scripts/release/release.sh: computes the version, stamps unreleased policy versions (scripts/release/stamp-legal-release.sh, committed as legal: stamp policy release date), commits chore: release vX.Y.Z (bumps backend/VERSION + frontend/VERSION) to main, tags it and creates the GitHub Release. Pushes with the release GitHub App token. It reuses a tag this run already pushed, skips a commit main already has, and creates the GitHub Release only when missing.
images Calls images.yml with the tag: checks the tag is vX.Y.Z on main, then builds ghcr.io/tomoda-labs/tomoda-{backend,frontend}:vX.Y.Z on GitHub-hosted runners. An image already in GHCR is skipped (docker buildx imagetools inspect), so only the missing one rebuilds.
bump-prod After images: pins newTag in k8s/apps/tomoda/overlays/prod/kustomization.yaml and the prod seed-load overlay of tomoda-labs/devops on branch release/tomoda-vX.Y.Z, and opens the deploy(prod): tomoda vX.Y.Z PR. The PR link lands in the run summary. The branch is rebuilt from devops main and force-pushed; an open PR for it is updated, never duplicated. If prod already pins the version, it does nothing.
ota-gate After bump-prod: reads devops main and checks that it pins prod to this tag. If so it passes pinned=true; if not, it shows the deploy PR link (notice and run summary) and passes pinned=false. It only reads.
ota-prod Only when ota-gate saw the pin: calls ota.yml with the tag, which checks the pin again, runs the frontend tests, then publishes the tag's JS bundle to ota.tomoda.life under both platforms' prod runtimeVersions. Publishing the same bundle again is harmless; the newest publish is what binaries fetch.
  • Policy dates: a policy change merges with **Last updated: unreleased** and an undated manifest entry; cut stamps both with the release day (UTC) so the tag ships dated documents, and fails if any entry stays unreleased. See Auth → Policy acceptance.

bump-prod lives in release.yml rather than images.yml so the image build stays a plain build with no devops credentials, and the deploy PR is a job of its own in the run.

Re-run and repair

  • A job failed: fix the cause and press Re-run failed jobs on the same run. Every job is idempotent, so the run picks up where it stopped and a re-run never cuts a second version.
  • The run is gone or you want a fresh run for a tag: run Release with version set to the latest tag. cut sees the tag exists, checks it is on main, makes sure the GitHub Release exists and skips the cut; images, bump-prod, ota-gate and ota-prod run again for that tag. Only the latest tag is accepted; to go back to an older version, revert its deploy PR.
  • Prod OTA did not publish after the merge: re-run the OTA gate vX.Y.Z job in the Release run. It passes now that devops main pins the tag, and ota-prod runs. If that run can no longer be re-run, use Actions → OTA → Run workflow with the tag (below).
  • Force an image rebuild for a tag: delete that version from the GHCR package, then repair.

Release runs only from main; on another ref every job is skipped. cut, bump-prod and ota-gate hold the release App key, so they run on the trusted in-cluster pool tomoda-arc-trusted (no PR jobs, no shared node cache; see ARC runners), as do the prod-pin job of ota.yml and its prod publish run, which reads the prod upload key. The image builds stay on GitHub-hosted runners (they need Docker). Runs share the concurrency group release: a second run waits for the first, and a third queued behind it replaces the second, which is cancelled and has to be started again.

images.yml can also be run on its own (Actions → Images → Run workflow with tag) to build a tag's images without opening a deploy PR.

Deploy and roll back

Merging the deploy(prod): tomoda vX.Y.Z PR in devops is the prod deploy. Reverting it rolls prod back to the previous version, and also republishes that version's JS to prod OTA (below). See DevOps → Rollback for the Argo CD UI path and the last-resort kubectl path.

Prod OTA

Prod OTA reaches installed apps at once, so it waits for the deploy: it publishes only a tag that devops main pins prod to. Merging the deploy PR triggers it:

  1. In devops, notify-prod-deploy.yml sees the prod pin change on main and sends a prod-deployed repository dispatch with the tag to this repo (release App token).
  2. Here, ota-after-deploy.yml finds the Release run on main whose job is named OTA gate vX.Y.Z, waits (checking once a minute, up to 90 minutes) for that run to finish, and re-runs the job.
  3. The gate passes, so ota-prod publishes in the same Release run, typically a few minutes after the merge, around when Argo CD finishes the rollout.

If no recent Release run has a gate for the tag, or GitHub refuses the re-run (runs older than 30 days can't be re-run), ota-after-deploy.yml runs ota.yml for the tag directly. Reverting a deploy PR goes through the same path with the older tag, so prod OTA follows the rollback.

To publish by hand, re-run OTA gate vX.Y.Z in the Release run, or use Actions → OTA → Run workflow on main with tag. Both publish only when devops main pins prod to that tag; ota.yml's prod-pin job fails otherwise.

ota-after-deploy.yml runs one at a time per tag. A second dispatch for the same tag while one is queued replaces the queued one; if a gate re-run never happened, re-run the gate by hand. Every ota job is a no-op while the repo variable OTA_ENABLED is not true.

This repo is private on the GitHub Free plan, where environment protection rules (required reviewers, wait timers and custom deployment protection rules) only apply to public repositories. That is why the gate is a job that checks the devops pin and not an environment approval.

Native binaries

Native binaries ship separately from this pipeline, and only when native code changes. Store binaries are prod builds made from the release tag after its deploy PR is merged, built and submitted from a developer Mac with fastlane; see Native → Shipping to the stores.

Migrations: expand and contract

A release's migrations run while the previous release still serves traffic, and again after a rollback. Every migration keeps the previous release working: split a breaking change into an expand release and a later contract release. task db:migrate:check (also run in CI on every PR) fails on drops, renames, type changes and new NOT NULL columns without a default. See Database → Backward-compatible migrations.

Cutting from a laptop

scripts/release/release.sh cuts the tag locally (interactive, or with RELEASE_VERSION / RELEASE_BUMP) and then offers to dispatch the Release workflow for it, which runs in repair mode. Nothing builds on a tag push, so a local cut without that run ships nothing. See Scripts.

One-time setup

  • Release GitHub App: installed on tomoda and devops with Contents and Pull requests Read and write. Its ID is the variable RELEASE_APP_ID in both repos. Its private key is the Secret Manager secret tomoda-release-app-private-key, which release.yml, ota.yml and devops notify-prod-deploy.yml read on main as the tomoda-release service account through WIF (variables GCP_WIF_PROVIDER, GCP_RELEASE_SA); branch, tag and PR runs get no token. If main is protected in either repo, add the App to the bypass list.
  • GHCR packages: each package (tomoda-backend, tomoda-frontend) lists this repo under Package settings → Manage Actions access with Admin (Write is enough to push; image retention needs Admin to delete old versions).
  • OTA: the secret XAVIA_DEV_UPLOAD_KEY, the variable OTA_ENABLED=true, and the prod upload key tomoda-ota-prod-upload-key read from Secret Manager the same way (see Native → Enabling OTA).

Where the manifests live

The shape of the application in the cluster is defined entirely in the DevOps repo under k8s/apps/tomoda/:

  • base/ — Deployments (backend + frontend), Ingress (Traefik, path-routed /api, /ws → backend), Service, NetworkPolicy, Kustomization
  • overlays/dev/ — namespace tomoda-dev, image tag tracked by Image Updater
  • overlays/prod/ — namespace tomoda-prod, image tag pinned in git, PodDisruptionBudget, HPAs

See DevOps → Tomoda K8s app for the manifests, resource limits, replicas, env vars, hostnames, probes, and the canonical health-check definition. Read those manifests, not this page, for current values.

Frontend deploy

The frontend Pod (Nginx serving the static Expo web export) is deployed via the same flow. Because EXPO_PUBLIC_* env vars are baked in at build time, any change to public API URLs / OAuth client IDs requires a rebuild, not a config change (see Container images).