Skip to content

Operations

Day-to-day operational guides for running the tomoda platform on the k3s cluster. Start with the Runbook for routine commands, then drill into specific topics below.

Bare metal (OVH)

The server itself: bringing it up, who can reach it, and changing the set of nodes. Hardware and k3s configuration live in OVH bare metal.

  • Bootstrap


    From nothing to a running cluster: secrets, the server, k3s, Argo CD.

  • Server Access


    Tailscale, personal logins, connecting kubectl, granting and removing access.

  • Node Lifecycle


    Adding, removing and replacing nodes, and what happens to each workload's data.

Running the platform

  • Runbook


    Daily ops cookbook: connect to clusters, tail logs, sync apps, restart deployments.

  • Deploy


    How a code change reaches dev (automatic) and prod (a merged tag bump PR), and the post-deploy smoke test.

  • Rollback


    Reverting bad deploys. Image rollback, Git revert, manual scale-down, and DB rollback options.

  • Scaling


    Scale backend replicas, CNPG storage and Redis; server capacity.

  • Debugging


    Production debugging: kubectl, Loki, Grafana, Argo CD events, External Secrets diagnostics.

  • Disaster Recovery


    The quarterly restore drill, PITR and latest-backup restore from R2 with scripts/disaster-recovery.sh, and rebuilding a lost server.

  • Postgres Operations


    CNPG cluster shape, Barman WAL archiving, manual backups, restores, scaling, upgrades.

  • Photon indexing


    Building and uploading the multilingual Photon index. Atomic in-cluster swap.

  • Access Control


    Argo CD SSO via Dex, release permissions, cluster access, GCP and Cloudflare credentials.

  • Observability


    Tempo (traces), Prometheus scrape coverage, Loki retention + Promtail pipeline.