Skip to content

Rollback

When a deploy breaks prod, pick the option that gets you back to known-good fastest, then clean up the GitOps state.

Option 1: Revert the prod bump (fastest)

Prod's version is pinned in k8s/apps/tomoda/overlays/prod/kustomization.yaml. Revert the deploy(prod): tomoda vX.Y.Z PR in GitHub (or open a PR setting newTag back to the last good version) and merge it. Argo CD rolls within a minute or two, and the post-deploy smoke test runs again. The merge also republishes that version's JS to the prod OTA server: notify-prod-deploy.yml tells the tomoda repo, which re-runs that version's Release run OTA gate, or runs its ota.yml for the tag when the run is too old to re-run (over 30 days). To do it by hand: tomoda Actions → OTA → Run workflow on main with the tag, after the revert is merged.

gh pr list --repo tomoda-labs/devops --search "deploy(prod)" --state merged --limit 5
gh pr view <number> --repo tomoda-labs/devops   # then "Revert" in the UI, merge the revert PR
kubectl rollout status deployment/tomoda-api -n tomoda-prod

Argo CD's History and Rollback also works for an immediate rollback, since nothing else moves the prod tag. Follow it with the git revert so main matches what's running; otherwise self-heal returns to the bad version on the next sync.

Rolling back the app is safe because migrations are expand/contract: the previous release works against the current schema. If a release broke that rule, see Option 4.

Option 2: Fix forward

Revert the offending commit in the app repo (tomoda), then run the Release workflow and merge the bump PR it opens.

cd tomoda
git revert <bad-commit-sha>
git push origin main
# Actions → Release, then merge the deploy(prod) PR in devops

Use this when the fix is small and you'd rather ship forward than pin an old version. Total wall-clock: 10-15 min (build + PR + Argo CD sync).

Option 3: Manual scale-down (emergency only)

If the app is actively causing data corruption or runaway costs and you need it off right now:

# Stop the API pool: kills inbound traffic immediately.
kubectl scale deployment/tomoda-api -n tomoda-prod --replicas=0

# Optionally stop async too if the incident involves runaway jobs.
kubectl scale deployment/tomoda-async -n tomoda-prod --replicas=0

This stops new traffic immediately. Argo CD will try to scale it back up because selfHeal: true is configured. To prevent that during the incident:

# Temporarily disable auto-sync on the Argo CD app
argocd app set tomoda-prod --sync-policy none

Last resort

A scaled-down app leaves the environment in a non-GitOps state. Re-enable sync (argocd app set tomoda-prod --sync-policy automated) as soon as the incident is contained, and follow up with Option 1 or 2 to get the manifest aligned with reality.

Option 4: Database rollback

App rollbacks are cheap; DB rollbacks are not. If the bad deploy ran a destructive migration or wrote corrupt data, you need point-in-time recovery.

See Disaster Recovery for the full procedure. Summary:

./scripts/disaster-recovery.sh --env prod \
  --mode pitr \
  --target-time "2026-04-27T09:55:00Z"

PITR creates a new CNPG cluster restored to the requested timestamp. After verification, --mode cutover points the ExternalName alias at it in git, and the merge applies it.

Verifying the rollback

# Image SHA matches the known-good
kubectl get deploy tomoda-api -n tomoda-prod \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Health endpoint
curl -fsS https://api.tomoda.life/health

# Argo CD reports Synced + Healthy
argocd app get tomoda-prod

After verification, write a brief post-mortem and add the failure pattern to Debugging so the next on-call recognises it faster.