Skip to content

Disaster Recovery

Recovery procedures by failure mode. All DB recovery flows, and the quarterly restore drill, are wrapped by scripts/disaster-recovery.sh.

Failure modes at a glance

Scenario Auto-recovers? Action Downtime
Pod crash Yes (Kubernetes restarts it) None Seconds
Disk failure Yes (RAID1 keeps serving) Ask OVH to swap the disk None
DB data corruption No PITR via CNPG 10-20 min
Accidental table drop No PITR to before drop 10-20 min
OTA (Xavia) DB lost or corrupt No pg_restore the latest daily dump Minutes
Server lost (hardware, reinstall) No Replacement server + bootstrap + DB restore 1-2 h plus OVH lead time
Beauharnois DC outage No Bootstrap a server elsewhere, restore from R2 Hours

Backup architecture

PostgreSQL (CNPG) + Barman Cloud plugin sidecar
  |
  +-- WAL archiving (continuous, gzip) --> R2
  +-- Base backup (daily 03:00 UTC)   --> R2
  |
  R2 bucket: tomoda-db-backups (off-server, separate provider)
    prod/<cluster>/    # one prefix per CNPG cluster: postgres-prod, or a recovered postgres-prod-<date>
      base/    # Full backups
      wals/    # WAL segments (enables PITR to any second)
    ota/{dev,prod}/
      xavia-<UTC timestamp>.dump   # daily pg_dump of the Xavia OTA DB
    k3s-etcd/                      # etcd snapshots, every 6h
  • WAL archiving is continuous. RPO is effectively seconds: only in-flight transactions are at risk.
  • Base backups run daily at 03:00 UTC. PITR uses the nearest base backup plus replayed WALs.
  • Retention is 30 days (retentionPolicy on the postgres-prod-backups ObjectStore, applied by the plugin to prod/postgres-prod/), with a 45-day R2 lifecycle rule as a backstop.
  • Bucket locks: ota/ dumps can't be deleted or overwritten for 14 days and k3s-etcd/ snapshots for 3 days, even with the cluster R2 token. CNPG's prod/ is not locked (Barman rewrites keys); see Bucket locks.
  • Dev has no CNPG backups; its data is reseeded.
  • OTA (Xavia) DBs are not CNPG; they get a daily logical dump instead. See OTA (Xavia) database.

Restore drill

The drill proves the backups restore without touching prod. Run it quarterly (first week of January, April, July and October) and after any change to the CNPG image, the backup config or scripts/disaster-recovery.sh.

task dr:drill                                          # latest backup
task dr:drill -- --target-time "2026-10-01T00:00:00Z"  # point in time
task dr:drill -- --keep                                # leave the drill cluster up to inspect

The script:

  1. Reads the live cluster (the one postgres-prod-postgresql points at), its Barman Cloud plugin parameters and the ObjectStore they name. Prod is only read: kubectl get and psql sessions with default_transaction_read_only=on. Nothing in prod is scaled, patched or backed up.
  2. Checks the node has room and refuses otherwise:
    • memory: at least 4 GiB of node memory still available after the drill's 2 GiB limit, and unrequested allocatable memory for its 1 GiB request
    • disk: free space for 1.25x the prod database size, with 50 GiB left over (MIN_FREE_MEM_GIB, MIN_FREE_DISK_GIB override the floors)
  3. Creates the restore-drill namespace with an ExternalSecret for the R2 credentials, a NetworkPolicy that admits only the CNPG operator, a copy of prod's ObjectStore without retentionPolicy, and a single-instance postgres-prod-drill cluster: 1 GiB request, 2 GiB limit, 256 MB shared_buffers, no PodMonitor and no plugins entry. The copied store is only its externalClusters recovery source (serverName: postgres-prod), so the drill neither archives WAL nor prunes backups. Backups from before and after the move to the plugin share prod/postgres-prod/, so a drill or PITR can target any point in the 30-day window.
  4. Waits up to 90 minutes (READY_TIMEOUT_MIN) for the cluster to report healthy.
  5. Prints the report, deletes the namespace (skipped with --keep), and exits non-zero if any check fails.
Check Passes when
Ready Cluster in healthy state within the timeout
Promoted pg_is_in_recovery() is false
Extensions Same pg_extension list as prod
TLS The server cert chains to the cluster CA and carries the postgres-prod-postgresql.data.svc.cluster.local SAN the backend verifies
Row counts users, friendships, moments, chat_messages, user_checkins and locations exist in both, and (for latest) are within 5% of prod
Freshness Latest created_at across users, moments, chat_messages and user_checkins is within 60 minutes of prod (latest), or not past the target (PITR)

DRILL_TABLES, DRILL_TS_TABLES, DRILL_COUNT_TOLERANCE_PCT and DRILL_MAX_LAG_MIN tune the checks. While a drill runs, the R2 cluster token is also mounted in restore-drill.

Drill log

Add a row after each drill.

Date Target Ready after Result
2026-10-19 latest About 1 min (0.1 GiB) Pass. Promoted, extensions match, TLS ok, row counts match prod (pre-launch: 1 user, other checked tables empty), lag 0 min (max 60)

Recovery

scripts/disaster-recovery.sh restores into a new cluster beside the live one and cuts over through git. A lost server is rebuilt first with Bootstrap (Steps 5-6), then restored with latest.

Mode When
pitr You know when the bad thing happened (bad migration, dropped table). Restore to just before it.
latest You need the most recent state (lost server, corrupt volume).

1. Pause Argo CD for the backend

The script stops writes by scaling tomoda-api and tomoda-async to 0, which self-heal would undo. Pause the parent app first, since it owns the tomoda-prod Application:

kubectl patch application prod -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":null}}}'
kubectl patch application tomoda-prod -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":null}}}'

The script refuses to run while either app still auto-syncs. postgres-prod keeps auto-sync; the script adds new resources and changes nothing it manages.

2. Restore

task dr:pitr -- prod "2026-04-27T09:55:00Z"
task dr:latest -- prod

The script:

  1. Reads the live cluster and checks the node has room for its memory request and database size.
  2. Asks you to type the new cluster's name (postgres-prod-<YYYYMMDD>; pass --name to choose another).
  3. Scales tomoda-api and tomoda-async to 0.
  4. Writes k8s/envs/prod/postgres/manifests/<name>.yaml and applies it:
    • a NetworkPolicy copied from the live cluster's, selecting the new cluster
    • an ObjectStore <name>-backups: the live store's configuration, retention and sidecar resources
    • the Cluster: the live image, parameters, pg_hba, resources, storage and serverAltDNSNames (always including the alias the backend dials); bootstrap.recovery from the live cluster's backups through the plugin (serverName of the live cluster), with the owner password set from postgres-prod-credentials; the plugin as its WAL archiver on <name>-backups, writing s3://tomoda-db-backups/prod/<name>/ so it never writes into the source's path
    • a daily ScheduledBackup that also takes a base backup right away
  5. Waits for the cluster, then prints row counts, the latest created_at, the recovery state and the TLS SAN check.

3. Cut over in git

task dr:cutover -- prod <name>

This edits the working tree only:

File Change
k8s/envs/prod/postgres/manifests/cluster.yaml The postgres-prod-postgresql alias points at <name>-rw
k8s/envs/prod/postgres/manifests/ca-reader-rbac.yaml The CA reader Role can also read <name>-ca
k8s/apps/tomoda/overlays/prod/ca-sync.yaml The backend mirrors <name>-ca, so verify-full trusts the new server
k8s/envs/platform/manifests/postgres-alerts.yaml, alerting-rules.yaml Postgres alert selectors (job, pod) follow the new cluster

It also lists other files that still name the old cluster. Commit them together with <name>.yaml, open a PR and merge; Argo CD syncs the alias and adopts the new cluster.

4. Resume and check

kubectl patch application prod -n argocd --type merge \
  -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'

The parent restores tomoda-prod's auto-sync from git, which mirrors the new CA and scales the backend back up. A backend pod that starts before the CA mirror refreshes fails its DB connection; it recovers on restart.

  • curl -fsS https://api.tomoda.life/health returns OK.
  • kubectl get backup -n data shows the new cluster's first base backup, and ContinuousArchiving is True for it (see Verifying backup health).

Roll back or retire

  • Before the cutover merges: kubectl delete -f k8s/envs/prod/postgres/manifests/<name>.yaml, delete the file, then resume as in step 4.
  • After the cutover: keep the old cluster until the new one has a base backup. Then remove the old Cluster, its ObjectStore, ScheduledBackup and NetworkPolicy from git in a follow-up PR. Keep the ExternalSecrets and the alias Service; the new cluster uses them.

Later drills and recoveries follow the alias, so they restore from whichever cluster is live.

Lost server

Before starting, the prod secrets (*-prod, including tomoda-asset-signing-key-prod and tomoda-r2-app-prod-credentials) must be in GCP Secret Manager (Secrets management); they survive a lost server.

  1. Get a replacement server from OVH (or order one), then run Bootstrap Steps 5-6. Argo CD recreates every workload; postgres-prod comes up empty.
  2. Restore with latest and cut over, as in Recovery. Argo CD recreates the live cluster empty, with its plugin archiver on the path that holds the backups you need (the empty-archive check stops its WAL archiving there): until the cutover merges, set spec.suspend: true on its ScheduledBackup in git (backup.yaml), so a base backup of the empty cluster can't become the newest one latest restores from. With no backup to restore (a brand-new prod), load the curated catalog instead: Location seed load: Load prod.
  3. Photon re-downloads its index from R2 on first start (an hour or two).
  4. Restore both OTA DBs from their latest dumps, as in OTA (Xavia) database.
  5. The prod overlay pins a vX.Y.Z that already exists in GHCR, so prod pods start as soon as Argo CD syncs; newer releases follow Deploy.

Check:

  • kubectl get applications -n argocd: prod apps Synced / Healthy.
  • kubectl get cluster -n data postgres-prod: 1 instance Ready, and kubectl get backup -n data shows a new base backup after the first ScheduledBackup run.
  • https://api.tomoda.life/health returns OK through the proxied, origin-locked ingress.
  • A signed media URL loads from https://assets.tomoda.life.

OTA (Xavia) database

Each Xavia instance's single-instance Postgres (xavia-ota-db-{dev,prod}, release metadata only; bundles live in the tomoda-ota-* buckets) is backed up by the CronJob xavia-ota-db-backup-{dev,prod} in platform:

Setting Value
Schedule prod 03:30 UTC, dev 03:45 UTC
Format pg_dump -Fc, validated with pg_restore --list before upload
Destination s3://tomoda-db-backups/ota/{dev,prod}/xavia-<UTC timestamp>.dump
Retention Job deletes dumps older than 30 days after each upload; the bucket's 45-day lifecycle rule is the backstop. A 14-day bucket lock blocks earlier deletes
Credentials S3_* keys already in xavia-ota-{env}-secrets (tomoda-r2-cluster-credentials)
Alerting KubeJobFailed on a failed run; OTABackupStale when no run has succeeded in 26 h

RPO is up to 24 h: releases published after the last dump are lost and must be republished from CI.

Restore

pg_restore --clean --single-transaction replaces the schema atomically, so Xavia keeps serving the old data until the restore commits. On a lost server, let Argo CD bring the instance up first (its init container creates an empty schema), then restore.

ENV=prod   # or dev

# 1. Pick and fetch a dump (rclone remote "r2")
rclone lsl r2:tomoda-db-backups/ota/$ENV/ | sort -k2,3 | tail -3
rclone copy r2:tomoda-db-backups/ota/$ENV/xavia-<timestamp>.dump .

# 2. Reach the DB
kubectl port-forward -n platform svc/xavia-ota-db-$ENV 15432:5432 &
export PGPASSWORD=$(gcloud secrets versions access latest --secret=tomoda-ota-$ENV-db-password)

# 3. Restore
pg_restore -h localhost -p 15432 -U xavia -d xavia \
  --clean --if-exists --no-owner --single-transaction xavia-<timestamp>.dump

# 4. Verify
psql -h localhost -p 15432 -U xavia -d xavia \
  -c 'SELECT runtime_version, max(timestamp) FROM releases GROUP BY 1;'

Use a local pg_restore of major version 16 or newer.

No-action scenarios

Single pod crash

Kubernetes restarts the pod; Argo CD selfHeal restores any drifted manifest.

kubectl get pods -n tomoda-prod
kubectl get pods -n data

Disk failure

The NVMe pair is RAID1, so the server keeps running. Check cat /proc/mdstat on the host and open an OVH support ticket for a disk swap; the array rebuilds online.

Verifying backup health

The backup alerts (Data alert runbook) cover this continuously. To check by hand:

# Last successful base backup timestamp
kubectl get cluster postgres-prod -n data \
  -o jsonpath='{.status.lastSuccessfulBackup}'

# All Backup CRs
kubectl get backup -n data

# Recent base backups in R2 (rclone remote "r2")
rclone lsd r2:tomoda-db-backups/prod/postgres-prod/base/ | tail -5

# Latest OTA dumps
rclone lsl r2:tomoda-db-backups/ota/prod/ | sort -k2,3 | tail -3
kubectl get cronjob -n platform xavia-ota-db-backup-prod xavia-ota-db-backup-dev \
  -o custom-columns=NAME:.metadata.name,LAST_SUCCESS:.status.lastSuccessfulTime

# Continuous archiving status
kubectl get cluster -n data \
  -o jsonpath='{range .items[*]}{.metadata.name}: {.status.conditions[?(@.type=="ContinuousArchiving")].status}{"\n"}{end}'

# Archiver errors (plugin sidecar)
kubectl logs postgres-prod-1 -n data -c plugin-barman-cloud --tail=50

ContinuousArchiving should be True for the live prod cluster at all times. These checks show backups exist; the restore drill shows they restore.

RTO / RPO

Metric Target Current capability
RPO (max data loss) Seconds Continuous WAL archiving
RTO (recovery time) <30 min for data PITR 10-20 min; lost server 1-2 h plus hardware lead time

File reference

File Purpose
scripts/disaster-recovery.sh DR script: --drill, pitr, latest, cutover
k8s/envs/prod/postgres/manifests/<name>.yaml A recovered cluster, written by the script and committed at cutover
k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml CNPG Cluster, plugin config and the prod ObjectStore
k8s/envs/platform/cnpg/application.yaml CNPG operator and Barman Cloud plugin
k8s/envs/prod/postgres/manifests/backup.yaml ScheduledBackup CR
k8s/envs/platform/xavia-ota-{dev,prod}/manifests.yaml OTA DB backup CronJobs
infrastructure/cloudflare/r2.tf tomoda-db-backups bucket, lifecycle + locks