Skip to content

Runbook

Common day-to-day operational procedures. Each section is self-contained — skim the headings and jump.

Start local dev

# Terminal 1 — backend + infra
task dev

# Terminal 2 — frontend (Metro + web; press i/a to attach an installed native dev binary)
task dev:frontend

# One-time per Simulator/emulator (and after native dep changes)
task dev:ios            # builds + installs the iOS dev binary on the Simulator
task dev:android        # builds + installs the Android dev binary on the emulator

task dev brings up Postgres, Redis, MinIO, and Photon via Docker Compose, runs migrations, regenerates Swagger docs, pulls GCP secrets (if available), and starts the backend on :8080 with Air for hot reload. The frontend runs on :8081 via the Expo dev server. Native JS edits hot-reload via Fast Refresh; only native dep or app.json plugin changes need another task dev:ios / task dev:android rebuild. See Docker Compose for the underlying stack.

Reset local DB

When migrations are in a weird state or you want to start over:

task db:reset           # docker:clean -> docker:up -> migrate
task db:test:setup      # re-seed ~20+ users, events, chats

db:reset runs task docker:clean (which is docker-compose down -v --remove-orphans) so it wipes the named volumes too — your local Postgres data is gone.

Pull GCP secrets

If your task dev is starting without real secrets (you see "GCP credentials not available — running with local defaults only" in the output), auth with gcloud:

gcloud auth login
gcloud config set project development-485000
task dev   # re-run — secrets will now be exported into the shell

The full mechanism is documented in Secrets.

Tail backend logs

Locally

The backend logs to stdout via Zap. In structured-logging mode this is JSON; in ENV=local it's the colored console encoder. Just look at the terminal where task dev is running.

To pipe through jq for readability:

task dev 2>&1 | jq -R 'fromjson? // .'

Production

Backend logs go to Cloud Logging (each Pod's stdout is collected by the GKE logging agent). Use the Logs Explorer in the GCP Console with a query like:

resource.type="k8s_container"
resource.labels.namespace_name="<dev-or-prod>"
resource.labels.container_name="backend"

For a faster CLI workflow:

gcloud logging read \
  'resource.type="k8s_container" AND resource.labels.container_name="backend"' \
  --limit 100 --format json --project development-485000

Deploy a backend hotfix

For a small, urgent fix on main:

git checkout main && git pull
# make the fix, commit, push
./scripts/release.sh    # bump VERSION, tag, push tag, optional GH release

The tag push triggers Cloud Build → image lands in Artifact Registry → ArgoCD picks it up → rolling update on GKE. See Deployment for the full chain.

If you need to ship a fix without bumping a tag (rare and discouraged), bump only the image reference in the devops/ repo and let ArgoCD reconcile.

Rollback

Two paths:

  1. ArgoCD UI — open the tomoda Application, click History and Rollback, select the previous sync, and confirm. ArgoCD re-applies the previous manifest set, including the previous image SHA. Fastest option.
  2. Git revert in the devops repo — git revert the commit that bumped the image (in devops/k8s/apps/tomoda/) and push. ArgoCD reconciles on its next sync.

Avoid kubectl rollout undo unless ArgoCD itself is broken — it'll drift from the Git source of truth.

Native app rollback

The native (iOS + Android) app is a separate beast because rollback means different things depending on the release vehicle.

Type of release Rollback options
EAS Update (OTA, JS-only) eas update --branch production --message "rollback" republishing the previous bundle, or use eas update:rollback to revert the channel pointer.
Native release (new binary submitted to App Store / Play Store) You can't unship a binary that's been approved. Submit a hotfix build with an incremented version. For Android you can halt the staged rollout in Play Console; for iOS you can pull from sale (App Store Connect → Pricing) but that's a sledgehammer.

See Native → Release for the full release flow.

Add a new env var

A new backend env var is a multi-step change:

  1. Add the field to the config struct (backend/internal/config/config.go or equivalent — see ../backend/infrastructure/config.md).
  2. Add the default / per-env value in backend/config.local.yaml and backend/config.{dev,prod}.yaml.
  3. Update backend/env.example.local so other developers see it.
  4. Add to scripts/pull-secrets.sh if the value is sensitive (so task dev picks it up).
  5. Create the secret in GCP Secret Manager (gcloud secrets create ...) for non-local environments.
  6. Add a key to the K8s backend-secrets Secret in the devops/ repo so it gets injected into the Pod.

Skipping step 6 will leave the Pod running without the value — usually a silent failure in the affected code path.

Add a new scheduled task

Scheduled jobs are not K8s CronJobs — they're handled by the in-process scheduler. See ../backend/infrastructure/async.md for how to register a new periodic task.

Investigate a slow query

task db:console       # opens psql against the local Postgres

Then:

EXPLAIN (ANALYZE, BUFFERS) SELECT ...;

For production, use Cloud SQL Insights in the GCP Console — it tracks the top queries by total time and exposes execution plans without needing a direct DB shell.

Common culprits in this codebase:

  • Missing index on a column added via a recent migration. Check backend/migrations/ and compare against pg_indexes.
  • A spatial query missing the GiST geography index (idx_*_coordinates_geog). Discovery and presence queries lean on these.
  • Sequential scan on a JSONB column. Add a GIN index or pull the field out into a column.

Investigate WebSocket flapping

Symptoms: users report repeated reconnects, real-time updates stop arriving, presence shows users as offline that are actually online.

The Hub is in-memory per pod; cross-pod fanout flows through Redis pub/sub on chat:event:*. Pod restarts drop the connections owned by that pod (clients reconnect, but it's user-visible); a Redis outage breaks cross-pod fanout specifically — same-pod chat still works.

Triage steps:

  1. Check Pod restart count. kubectl -n <ns> get pods -l app.kubernetes.io/name=tomoda-api — if RESTARTS is climbing, the Pod is OOMing or panicking. Look at logs (kubectl logs ... --previous).
  2. Check liveness probe. A slow /health (>30s with periodSeconds: 30) will get the Pod killed. The /health route is registered in backend/internal/wiring/router.go.
  3. Check ping/pong intervals. The Hub sends pings; the client must pong within the deadline. Mismatched timeouts between the proxy in front (Cloudflare / Ingress) and the Hub will look like flapping. See ../backend/infrastructure/websocket.md.
  4. Check Ingress idle timeout. GKE Ingress defaults can drop idle WS connections — verify in devops/k8s/apps/tomoda/base/ingress.yaml.
  5. Check cross-pod fanout. If users on one pod see messages but users on a sibling pod don't, the issue is the Hub subscriber. Logs include Hub subscriber loop error; reconnecting on Redis pub/sub failures; MONITOR Redis briefly to confirm chat:event:* traffic if needed.

If the Pod is healthy and connections are still flapping, the issue is client-side or network-side, not on the server.