Operations¶
Day-to-day operational guides for running the tomoda platform on the k3s cluster. Start with the Runbook for routine commands, then drill into specific topics below.
Bare metal (OVH)¶
The server itself: bringing it up, who can reach it, and changing the set of nodes. Hardware and k3s configuration live in OVH bare metal.
-
From nothing to a running cluster: secrets, the server, k3s, Argo CD.
-
Tailscale, personal logins, connecting
kubectl, granting and removing access. -
Adding, removing and replacing nodes, and what happens to each workload's data.
Running the platform¶
-
Daily ops cookbook: connect to clusters, tail logs, sync apps, restart deployments.
-
How a code change reaches dev (automatic) and prod (a merged tag bump PR), and the post-deploy smoke test.
-
Reverting bad deploys. Image rollback, Git revert, manual scale-down, and DB rollback options.
-
Scale backend replicas, CNPG storage and Redis; server capacity.
-
Production debugging: kubectl, Loki, Grafana, Argo CD events, External Secrets diagnostics.
-
The quarterly restore drill, PITR and latest-backup restore from R2 with
scripts/disaster-recovery.sh, and rebuilding a lost server. -
CNPG cluster shape, Barman WAL archiving, manual backups, restores, scaling, upgrades.
-
Building and uploading the multilingual Photon index. Atomic in-cluster swap.
-
Argo CD SSO via Dex, release permissions, cluster access, GCP and Cloudflare credentials.
-
Tempo (traces), Prometheus scrape coverage, Loki retention + Promtail pipeline.