Skip to content

Operations

Day-to-day operational guides for running the tomoda platform on GKE. Start with the Runbook for routine commands, then drill into specific topics below.

  • Runbook

    Daily ops cookbook: connect to clusters, tail logs, sync apps, restart deployments.

  • Deploy

    How a code change reaches dev and prod via Cloud Build, Artifact Registry, and Argo CD Image Updater.

  • Rollback

    Reverting bad deploys. Image rollback, Git revert, manual scale-down, and DB rollback options.

  • Scaling

    Scale backend replicas, CNPG storage, Redis, and GKE node pools.

  • Debugging

    Production debugging: kubectl, Loki, Grafana, Argo CD events, External Secrets diagnostics.

  • Disaster Recovery

    PITR, latest-backup restore, and full-rebuild recovery using scripts/disaster-recovery.sh.

  • Postgres Operations

    CNPG cluster shape, Barman WAL archiving, manual backups, restores, scaling, upgrades.

  • Photon Multilang Rollout

    Building and uploading the multilingual Photon index. Atomic in-cluster swap.

  • Access Control

    Argo CD SSO via Dex, Cloud Build approvals, GCP/AWS IAM bindings.

  • Observability

    Tempo (traces), Prometheus scrape coverage, Loki retention + Promtail pipeline.