Disaster Recovery¶
Recovery procedures by failure mode. All DB recovery flows, and the quarterly restore drill, are wrapped by scripts/disaster-recovery.sh.
Failure modes at a glance¶
| Scenario | Auto-recovers? | Action | Downtime |
|---|---|---|---|
| Pod crash | Yes (Kubernetes restarts it) | None | Seconds |
| Disk failure | Yes (RAID1 keeps serving) | Ask OVH to swap the disk | None |
| DB data corruption | No | PITR via CNPG | 10-20 min |
| Accidental table drop | No | PITR to before drop | 10-20 min |
| OTA (Xavia) DB lost or corrupt | No | pg_restore the latest daily dump |
Minutes |
| Server lost (hardware, reinstall) | No | Replacement server + bootstrap + DB restore | 1-2 h plus OVH lead time |
| Beauharnois DC outage | No | Bootstrap a server elsewhere, restore from R2 | Hours |
Backup architecture¶
PostgreSQL (CNPG) + Barman Cloud plugin sidecar
|
+-- WAL archiving (continuous, gzip) --> R2
+-- Base backup (daily 03:00 UTC) --> R2
|
R2 bucket: tomoda-db-backups (off-server, separate provider)
prod/<cluster>/ # one prefix per CNPG cluster: postgres-prod, or a recovered postgres-prod-<date>
base/ # Full backups
wals/ # WAL segments (enables PITR to any second)
ota/{dev,prod}/
xavia-<UTC timestamp>.dump # daily pg_dump of the Xavia OTA DB
k3s-etcd/ # etcd snapshots, every 6h
- WAL archiving is continuous. RPO is effectively seconds: only in-flight transactions are at risk.
- Base backups run daily at 03:00 UTC. PITR uses the nearest base backup plus replayed WALs.
- Retention is 30 days (
retentionPolicyon thepostgres-prod-backupsObjectStore, applied by the plugin toprod/postgres-prod/), with a 45-day R2 lifecycle rule as a backstop. - Bucket locks:
ota/dumps can't be deleted or overwritten for 14 days andk3s-etcd/snapshots for 3 days, even with the cluster R2 token. CNPG'sprod/is not locked (Barman rewrites keys); see Bucket locks. - Dev has no CNPG backups; its data is reseeded.
- OTA (Xavia) DBs are not CNPG; they get a daily logical dump instead. See OTA (Xavia) database.
Restore drill¶
The drill proves the backups restore without touching prod. Run it quarterly (first week of January, April, July and October) and after any change to the CNPG image, the backup config or scripts/disaster-recovery.sh.
task dr:drill # latest backup
task dr:drill -- --target-time "2026-10-01T00:00:00Z" # point in time
task dr:drill -- --keep # leave the drill cluster up to inspect
The script:
- Reads the live cluster (the one
postgres-prod-postgresqlpoints at), its Barman Cloud plugin parameters and theObjectStorethey name. Prod is only read:kubectl getandpsqlsessions withdefault_transaction_read_only=on. Nothing in prod is scaled, patched or backed up. - Checks the node has room and refuses otherwise:
- memory: at least 4 GiB of node memory still available after the drill's 2 GiB limit, and unrequested allocatable memory for its 1 GiB request
- disk: free space for 1.25x the prod database size, with 50 GiB left over (
MIN_FREE_MEM_GIB,MIN_FREE_DISK_GIBoverride the floors)
- Creates the
restore-drillnamespace with an ExternalSecret for the R2 credentials, a NetworkPolicy that admits only the CNPG operator, a copy of prod'sObjectStorewithoutretentionPolicy, and a single-instancepostgres-prod-drillcluster: 1 GiB request, 2 GiB limit, 256 MBshared_buffers, no PodMonitor and nopluginsentry. The copied store is only itsexternalClustersrecovery source (serverName: postgres-prod), so the drill neither archives WAL nor prunes backups. Backups from before and after the move to the plugin shareprod/postgres-prod/, so a drill or PITR can target any point in the 30-day window. - Waits up to 90 minutes (
READY_TIMEOUT_MIN) for the cluster to report healthy. - Prints the report, deletes the namespace (skipped with
--keep), and exits non-zero if any check fails.
| Check | Passes when |
|---|---|
| Ready | Cluster in healthy state within the timeout |
| Promoted | pg_is_in_recovery() is false |
| Extensions | Same pg_extension list as prod |
| TLS | The server cert chains to the cluster CA and carries the postgres-prod-postgresql.data.svc.cluster.local SAN the backend verifies |
| Row counts | users, friendships, moments, chat_messages, user_checkins and locations exist in both, and (for latest) are within 5% of prod |
| Freshness | Latest created_at across users, moments, chat_messages and user_checkins is within 60 minutes of prod (latest), or not past the target (PITR) |
DRILL_TABLES, DRILL_TS_TABLES, DRILL_COUNT_TOLERANCE_PCT and DRILL_MAX_LAG_MIN tune the checks. While a drill runs, the R2 cluster token is also mounted in restore-drill.
Drill log¶
Add a row after each drill.
| Date | Target | Ready after | Result |
|---|---|---|---|
| 2026-10-19 | latest | About 1 min (0.1 GiB) | Pass. Promoted, extensions match, TLS ok, row counts match prod (pre-launch: 1 user, other checked tables empty), lag 0 min (max 60) |
Recovery¶
scripts/disaster-recovery.sh restores into a new cluster beside the live one and cuts over through git. A lost server is rebuilt first with Bootstrap (Steps 5-6), then restored with latest.
| Mode | When |
|---|---|
pitr |
You know when the bad thing happened (bad migration, dropped table). Restore to just before it. |
latest |
You need the most recent state (lost server, corrupt volume). |
1. Pause Argo CD for the backend¶
The script stops writes by scaling tomoda-api and tomoda-async to 0, which self-heal would undo. Pause the parent app first, since it owns the tomoda-prod Application:
kubectl patch application prod -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":null}}}'
kubectl patch application tomoda-prod -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":null}}}'
The script refuses to run while either app still auto-syncs. postgres-prod keeps auto-sync; the script adds new resources and changes nothing it manages.
2. Restore¶
task dr:pitr -- prod "2026-04-27T09:55:00Z"
task dr:latest -- prod
The script:
- Reads the live cluster and checks the node has room for its memory request and database size.
- Asks you to type the new cluster's name (
postgres-prod-<YYYYMMDD>; pass--nameto choose another). - Scales
tomoda-apiandtomoda-asyncto 0. - Writes
k8s/envs/prod/postgres/manifests/<name>.yamland applies it:- a NetworkPolicy copied from the live cluster's, selecting the new cluster
- an
ObjectStore<name>-backups: the live store's configuration, retention and sidecar resources - the Cluster: the live image, parameters,
pg_hba, resources, storage andserverAltDNSNames(always including the alias the backend dials);bootstrap.recoveryfrom the live cluster's backups through the plugin (serverNameof the live cluster), with the owner password set frompostgres-prod-credentials; the plugin as its WAL archiver on<name>-backups, writings3://tomoda-db-backups/prod/<name>/so it never writes into the source's path - a daily ScheduledBackup that also takes a base backup right away
- Waits for the cluster, then prints row counts, the latest
created_at, the recovery state and the TLS SAN check.
3. Cut over in git¶
task dr:cutover -- prod <name>
This edits the working tree only:
| File | Change |
|---|---|
k8s/envs/prod/postgres/manifests/cluster.yaml |
The postgres-prod-postgresql alias points at <name>-rw |
k8s/envs/prod/postgres/manifests/ca-reader-rbac.yaml |
The CA reader Role can also read <name>-ca |
k8s/apps/tomoda/overlays/prod/ca-sync.yaml |
The backend mirrors <name>-ca, so verify-full trusts the new server |
k8s/envs/platform/manifests/postgres-alerts.yaml, alerting-rules.yaml |
Postgres alert selectors (job, pod) follow the new cluster |
It also lists other files that still name the old cluster. Commit them together with <name>.yaml, open a PR and merge; Argo CD syncs the alias and adopts the new cluster.
4. Resume and check¶
kubectl patch application prod -n argocd --type merge \
-p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'
The parent restores tomoda-prod's auto-sync from git, which mirrors the new CA and scales the backend back up. A backend pod that starts before the CA mirror refreshes fails its DB connection; it recovers on restart.
curl -fsS https://api.tomoda.life/healthreturns OK.kubectl get backup -n datashows the new cluster's first base backup, andContinuousArchivingisTruefor it (see Verifying backup health).
Roll back or retire¶
- Before the cutover merges:
kubectl delete -f k8s/envs/prod/postgres/manifests/<name>.yaml, delete the file, then resume as in step 4. - After the cutover: keep the old cluster until the new one has a base backup. Then remove the old Cluster, its ObjectStore, ScheduledBackup and NetworkPolicy from git in a follow-up PR. Keep the ExternalSecrets and the alias Service; the new cluster uses them.
Later drills and recoveries follow the alias, so they restore from whichever cluster is live.
Lost server¶
Before starting, the prod secrets (*-prod, including tomoda-asset-signing-key-prod and tomoda-r2-app-prod-credentials) must be in GCP Secret Manager (Secrets management); they survive a lost server.
- Get a replacement server from OVH (or order one), then run Bootstrap Steps 5-6. Argo CD recreates every workload;
postgres-prodcomes up empty. - Restore with
latestand cut over, as in Recovery. Argo CD recreates the live cluster empty, with its plugin archiver on the path that holds the backups you need (the empty-archive check stops its WAL archiving there): until the cutover merges, setspec.suspend: trueon its ScheduledBackup in git (backup.yaml), so a base backup of the empty cluster can't become the newest onelatestrestores from. With no backup to restore (a brand-new prod), load the curated catalog instead: Location seed load: Load prod. - Photon re-downloads its index from R2 on first start (an hour or two).
- Restore both OTA DBs from their latest dumps, as in OTA (Xavia) database.
- The prod overlay pins a
vX.Y.Zthat already exists in GHCR, so prod pods start as soon as Argo CD syncs; newer releases follow Deploy.
Check:
kubectl get applications -n argocd: prod appsSynced/Healthy.kubectl get cluster -n data postgres-prod: 1 instance Ready, andkubectl get backup -n datashows a new base backup after the first ScheduledBackup run.https://api.tomoda.life/healthreturns OK through the proxied, origin-locked ingress.- A signed media URL loads from
https://assets.tomoda.life.
OTA (Xavia) database¶
Each Xavia instance's single-instance Postgres (xavia-ota-db-{dev,prod}, release metadata only; bundles live in the tomoda-ota-* buckets) is backed up by the CronJob xavia-ota-db-backup-{dev,prod} in platform:
| Setting | Value |
|---|---|
| Schedule | prod 03:30 UTC, dev 03:45 UTC |
| Format | pg_dump -Fc, validated with pg_restore --list before upload |
| Destination | s3://tomoda-db-backups/ota/{dev,prod}/xavia-<UTC timestamp>.dump |
| Retention | Job deletes dumps older than 30 days after each upload; the bucket's 45-day lifecycle rule is the backstop. A 14-day bucket lock blocks earlier deletes |
| Credentials | S3_* keys already in xavia-ota-{env}-secrets (tomoda-r2-cluster-credentials) |
| Alerting | KubeJobFailed on a failed run; OTABackupStale when no run has succeeded in 26 h |
RPO is up to 24 h: releases published after the last dump are lost and must be republished from CI.
Restore¶
pg_restore --clean --single-transaction replaces the schema atomically, so Xavia keeps serving the old data until the restore commits. On a lost server, let Argo CD bring the instance up first (its init container creates an empty schema), then restore.
ENV=prod # or dev
# 1. Pick and fetch a dump (rclone remote "r2")
rclone lsl r2:tomoda-db-backups/ota/$ENV/ | sort -k2,3 | tail -3
rclone copy r2:tomoda-db-backups/ota/$ENV/xavia-<timestamp>.dump .
# 2. Reach the DB
kubectl port-forward -n platform svc/xavia-ota-db-$ENV 15432:5432 &
export PGPASSWORD=$(gcloud secrets versions access latest --secret=tomoda-ota-$ENV-db-password)
# 3. Restore
pg_restore -h localhost -p 15432 -U xavia -d xavia \
--clean --if-exists --no-owner --single-transaction xavia-<timestamp>.dump
# 4. Verify
psql -h localhost -p 15432 -U xavia -d xavia \
-c 'SELECT runtime_version, max(timestamp) FROM releases GROUP BY 1;'
Use a local pg_restore of major version 16 or newer.
No-action scenarios¶
Single pod crash¶
Kubernetes restarts the pod; Argo CD selfHeal restores any drifted manifest.
kubectl get pods -n tomoda-prod
kubectl get pods -n data
Disk failure¶
The NVMe pair is RAID1, so the server keeps running. Check cat /proc/mdstat on the host and open an OVH support ticket for a disk swap; the array rebuilds online.
Verifying backup health¶
The backup alerts (Data alert runbook) cover this continuously. To check by hand:
# Last successful base backup timestamp
kubectl get cluster postgres-prod -n data \
-o jsonpath='{.status.lastSuccessfulBackup}'
# All Backup CRs
kubectl get backup -n data
# Recent base backups in R2 (rclone remote "r2")
rclone lsd r2:tomoda-db-backups/prod/postgres-prod/base/ | tail -5
# Latest OTA dumps
rclone lsl r2:tomoda-db-backups/ota/prod/ | sort -k2,3 | tail -3
kubectl get cronjob -n platform xavia-ota-db-backup-prod xavia-ota-db-backup-dev \
-o custom-columns=NAME:.metadata.name,LAST_SUCCESS:.status.lastSuccessfulTime
# Continuous archiving status
kubectl get cluster -n data \
-o jsonpath='{range .items[*]}{.metadata.name}: {.status.conditions[?(@.type=="ContinuousArchiving")].status}{"\n"}{end}'
# Archiver errors (plugin sidecar)
kubectl logs postgres-prod-1 -n data -c plugin-barman-cloud --tail=50
ContinuousArchiving should be True for the live prod cluster at all times. These checks show backups exist; the restore drill shows they restore.
RTO / RPO¶
| Metric | Target | Current capability |
|---|---|---|
| RPO (max data loss) | Seconds | Continuous WAL archiving |
| RTO (recovery time) | <30 min for data | PITR 10-20 min; lost server 1-2 h plus hardware lead time |
File reference¶
| File | Purpose |
|---|---|
scripts/disaster-recovery.sh |
DR script: --drill, pitr, latest, cutover |
k8s/envs/prod/postgres/manifests/<name>.yaml |
A recovered cluster, written by the script and committed at cutover |
k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml |
CNPG Cluster, plugin config and the prod ObjectStore |
k8s/envs/platform/cnpg/application.yaml |
CNPG operator and Barman Cloud plugin |
k8s/envs/prod/postgres/manifests/backup.yaml |
ScheduledBackup CR |
k8s/envs/platform/xavia-ota-{dev,prod}/manifests.yaml |
OTA DB backup CronJobs |
infrastructure/cloudflare/r2.tf |
tomoda-db-backups bucket, lifecycle + locks |