Skip to content

Data alert runbook

What to do when a Postgres or Redis alert fires in #monitoring. Rules live in k8s/envs/platform/manifests/postgres-alerts.yaml, redis-alerts.yaml and the tomoda.data.containers group of alerting-rules.yaml; each rule's runbook_url links its section here. Everything else: Alert response runbook.

Every alert carries an env label, shown in the Discord message. Pairs are critical for prod and warning for dev at the same threshold; a heading without "dev" is prod only. Selectors use the scrape job (data/postgres-<env>, redis-<env>-metrics), not a cluster label: CNPG sets cluster only on cnpg_collector_up.


Database

PostgresDown (critical prod / warning dev)

Trigger: cnpg_collector_up == 0, or no cnpg_collector_up series at all (pod gone, scrape failing), for 1 minute. Suppresses every severity=warning alert with the same env, backend warnings included.

Debug path:

kubectl get cluster -n data                         # "Cluster in healthy state"? failover in progress: wait 60s
kubectl get pods -n data -l cnpg.io/cluster=postgres-prod
kubectl describe pod -n data <pod-name> | grep -A20 Events:
kubectl logs -n data <pod-name> -c postgres --tail=200
#   "the database system is starting up"  → startup, wait
#   "out of memory" / OOMKilled          → see OOMKilled
#   "No space left on device"            → see Disk full
#   "could not open file pg_wal/..."     → WAL corruption: escalate, don't restart blindly

Pod crashed, restarting: CNPG restarts it. If it crashes again, read kubectl logs -n data <pod> -c postgres --previous.

Disk full: local-path PVCs are directories on the server's root filesystem; spec.storage.size is a budget, not a quota. "Full" means the NVMe is full, and NodeFilesystem* alerts fire too. Free space on the server (sudo du -sh /var/lib/rancher/k3s/storage/*, old images, Photon tarballs), then trim data. See the disk budget.

OOMKilled: find the trigger first.

task pg:console -- prod
SELECT pid, now() - query_start AS duration, state, query
  FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;
SELECT pg_cancel_backend(<pid>);     -- or pg_terminate_backend

Then raise spec.resources.limits.memory in k8s/envs/prod/postgres/manifests/cluster.yaml through Git.

Connection storm (too many clients already): see PostgresHighConnections.

Escalation:

  • Down 5 minutes with no clear cause: @here
  • Data unrecoverable (failed disk, corruption): Disaster Recovery, point-in-time restore from R2
  • WAL corruption in logs: talk to the backend team before restarting

SLO impact: burns api-availability, login-success and async-task-success at once.

PostgresNoPrimary (critical prod / warning dev)

Trigger: every instance of the cluster reports pg_is_in_recovery() for 5 minutes: nothing accepts writes. Usually a restore or a replica-mode cluster that never got promoted.

kubectl get cluster -n data postgres-prod -o jsonpath='{.status.currentPrimary} {.status.phase}{"\n"}'
kubectl logs -n cnpg-system deploy/cnpg-operator-cloudnative-pg --tail=200 | grep -i -E "promot|switchover|failover"

A recovered cluster still bootstrapping from R2 is expected; wait for "Cluster in healthy state". Otherwise check spec.replica.enabled in the cluster manifest and promote through Git.

PostgresReplicationLag (critical prod / warning dev)

Trigger: replica lag > 10s for 5 minutes. Dormant while both clusters run one instance.

kubectl exec -n data postgres-prod-1 -c postgres -- psql -c "SELECT * FROM pg_stat_replication;" shows which replica and how far behind. Usually heavy write load and transient.

PostgresHighConnections (critical prod / warning dev)

Trigger: backends (cnpg_backends_total) > 80% of max_connections (200 prod, 100 dev) for 5 minutes. PostgresConnectionsNearMax is the 70% prod early warning.

kubectl exec -n data postgres-prod-1 -c postgres -- psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
Cause Recipe
Connection leak in the backend Find the code path; deploy a fix
Long-running queries holding connections List non-idle queries by query_start; pg_cancel_backend(pid)
Pods × DB_MAX_CONNS near the cap after a scale-up Lower DB_MAX_CONNS or add the pooler (Scaling)

PostgresBackupStale / PostgresWALArchivingFailing (critical)

Trigger: PostgresBackupStale: the last archived WAL segment is > 1h old (dev too, as a warning). PostgresWALArchivingFailing: the latest archive attempt failed and none has succeeded since, for 10 minutes (prod). Unarchived WAL stays on disk, so PostgresDiskUsageHigh follows if this lasts.

Check kubectl logs -n data postgres-prod-1 -c plugin-barman-cloud --tail=50 (the archiver runs in that sidecar; no such container means the plugin was not injected, see kubectl get pods -n cnpg-system -l app.kubernetes.io/name=plugin-barman-cloud) and rclone lsd r2:tomoda-db-backups/prod/postgres-prod/.

Cause Recipe
R2 credentials wrong or revoked kubectl get externalsecret r2-backup-credentials -n data should be SecretSynced; AccessDenied / InvalidAccessKeyId in logs means the token in tomoda-r2-cluster-credentials is wrong
R2 unreachable kubectl exec -n data postgres-prod-1 -c postgres -- curl -sI https://8d3b225f38b3bcd337b9973d128c0eb6.r2.cloudflarestorage.com should return an HTTP status

Escalation: broken > 24h is a durability risk. Hold deploys until fixed.

PostgresBaseBackupFailed / PostgresBaseBackupStale (critical)

Trigger: the latest base backup of postgres-prod failed, or none completed in 26h (daily at 03:00 UTC).

kubectl get backups -n data --sort-by=.metadata.creationTimestamp | tail -5
kubectl describe backup -n data <latest>
kubectl logs -n cnpg-system deploy/cnpg-operator-cloudnative-pg --tail=200 | grep -i backup
kubectl logs -n data postgres-prod-1 -c plugin-barman-cloud --tail=100   # barman-cloud-backup output

The backup metrics (barman_cloud_cloudnative_pg_io_*) come from the plugin sidecar, which refreshes them from the R2 backup catalog when it starts and every 30 minutes. A sidecar OOM kill (kubectl describe pod postgres-prod-1 -n data) means instanceSidecarConfiguration.resources on the postgres-prod-backups ObjectStore is too small.

Same credentials as WAL archiving, so PostgresBackupStale usually fires too.

OTABackupStale (critical prod / warning dev)

Trigger: CronJob xavia-ota-db-backup-{env} in platform hasn't completed in 26h (daily: prod 03:30 UTC, dev 03:45 UTC). Also fires for a CronJob that has never succeeded.

kubectl get cronjob,jobs -n platform | grep xavia-ota-db-backup
kubectl logs -n platform job/<latest> -c dump
kubectl logs -n platform job/<latest> -c upload

Usual causes: the Xavia Postgres is down (dump can't connect), or the R2 token can't write tomoda-db-backups. Restore: Disaster Recovery.

PostgresLongRunningTransaction / PostgresLockWaits / PostgresDeadlocks (warning)

Trigger: a transaction open for more than 10 minutes, more than 5 backends waiting on a lock for 5 minutes, or any deadlock in 15 minutes (prod).

task pg:console -- prod
SELECT pid, now() - xact_start AS age, state, wait_event_type, left(query, 80)
  FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY age DESC LIMIT 10;
SELECT pid, pg_blocking_pids(pid) AS blocked_by, left(query, 80)
  FROM pg_stat_activity WHERE cardinality(pg_blocking_pids(pid)) > 0;
SELECT pg_cancel_backend(<pid>);     -- or pg_terminate_backend
Cause Recipe
idle in transaction session (a psql left open, a code path missing commit) Terminate it; fix the code path
Migration or seed load holding a table lock Let it finish if it is progressing; schedule big ones off-peak
Deadlocks The Postgres log names both statements (kubectl logs -n data postgres-prod-1 -c postgres \| grep -A5 deadlock); make the code take locks in one order

PostgresXIDAgeHigh / PostgresXIDWraparoundImminent (warning / critical)

Trigger: a database's transaction ID age passes 500M for 30 minutes (warning, prod and dev) or 1.2B for 5 minutes (critical, prod). autovacuum_freeze_max_age is 200M, so 500M means anti-wraparound vacuum is not keeping up; Postgres stops accepting writes near 2.1B.

SELECT relname, age(relfrozenxid) FROM pg_class WHERE relkind = 'r' ORDER BY 2 DESC LIMIT 10;
SELECT pid, now() - xact_start, state FROM pg_stat_activity WHERE backend_xmin IS NOT NULL ORDER BY 2 DESC;

A long-open transaction or an abandoned replication slot pins the horizon: end it first. Then VACUUM (FREEZE, VERBOSE) <table>; the oldest tables.

PostgresDiskUsageHigh / PostgresDiskUsageCritical / PostgresDiskFillingUp

Trigger: databases plus WAL pass 80% (warning, prod and dev) or 90% (critical, prod) of the PVC request, or the 6h trend crosses it within 24h (warning, prod and dev). local-path volumes report no usage, so the PVC size (150 GiB prod, 3 GiB dev) is a budget, not a quota: the real limit is the root filesystem (disk budget).

If WAL is the part growing, PostgresWALArchivingFailing is the cause. Otherwise find what grew (SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10; via task pg:console -- prod), then trim, raise spec.storage.size if the root filesystem has room, or plan more disk.

PostgresCacheHitRatioLow (warning)

Trigger: under 90% of block reads come from shared buffers for 30 minutes while reads exceed 100 blocks/s (prod). The working set outgrew shared_buffers (4 GiB prod), or a query is scanning a large table.

Look for sequential scans on big tables (SELECT relname, seq_scan, seq_tup_read FROM pg_stat_user_tables ORDER BY seq_tup_read DESC LIMIT 10;) and add the missing index. If the data simply grew, raise shared_buffers with the memory limit in k8s/envs/prod/postgres/manifests/cluster.yaml.

Redis

RedisDown (critical prod / warning dev)

Trigger: redis_up == 0, or no redis_up series (exporter gone), for 1 minute. Suppresses the other Redis warnings for the same env.

kubectl get pods -n data -l app.kubernetes.io/name=redis
kubectl logs -n data <redis-pod> --tail=100

Usual causes: pod crash (auto-restarts), volume mount errors (kubectl describe pod), or a bad values change (revert the commit).

Impact: Asynq job dispatch stops and backend pods fail readiness; async-task-success starts burning.

RedisHighMemory (critical prod / warning dev)

Trigger: used memory > 90% of maxmemory for 5 minutes. RedisMemoryNearMax is the prod 80% early warning (15 minutes). maxmemory sits below the container limit with noeviction (values in Redis): at the cap Redis rejects writes (enqueues fail) instead of evicting jobs or being OOMKilled.

kubectl exec -n data <redis-pod> -- redis-cli INFO memory
kubectl exec -n data <redis-pod> -- redis-cli --bigkeys
Cause Recipe
Keys without a TTL --bigkeys finds them; add a TTL in the backend
Asynq queues not draining tomoda_async_task_total{status="failure"} by task type and the async pod logs
Load outgrew the cap Raise --maxmemory and master.resources.limits.memory together in k8s/envs/{dev,prod}/redis/values.yaml

RedisMaxmemoryUnset (warning)

Trigger: Redis reports maxmemory 0 for 15 minutes. The memory alerts skip an unbounded instance, and without the cap the container limit OOMKills Redis instead of rejecting writes. Restore --maxmemory in k8s/envs/{dev,prod}/redis/values.yaml (kubectl exec -n data <redis-pod> -- redis-cli CONFIG GET maxmemory confirms).

RedisRejectingWrites (critical prod / warning dev)

Trigger: any command failed with OOM in 5 minutes: Redis is at maxmemory and, with noeviction, rejects writes. Asynq enqueues fail now. Follow RedisHighMemory; the fastest relief is draining or deleting a stuck queue, then raising the cap.

RedisClientsNearMax / RedisRejectedConnections

Trigger: connected clients above 80% of maxclients for 5 minutes (warning, prod), or any connection rejected at maxclients in 5 minutes (critical prod / warning dev).

kubectl exec -n data <redis-pod> -- redis-cli CLIENT LIST | awk '{print $2}' | cut -d= -f2 | cut -d: -f1 | sort | uniq -c | sort -rn | head

Many connections from one pod IP is a client leak in that workload; pods × pool size near the cap after a scale-up means lowering the pool size.

RedisPersistenceFailing (critical prod / warning dev)

Trigger: the last RDB save, AOF write or AOF rewrite failed, for 5 minutes. A restart loses writes since the last good save.

kubectl exec -n data <redis-pod> -- redis-cli INFO persistence
kubectl logs -n data <redis-pod> -c redis --tail=100 | grep -i -E "aof|rdb|fork|disk"

Usual causes: the root filesystem is full (see the NodeFilesystem* alerts), or a fork failing under memory pressure.

RedisDiskUsageHigh (warning)

Trigger: the append-only file passes 80% of the PVC request (10 GiB prod, 1 GiB dev) for 30 minutes. Like Postgres, the PVC is a budget on the root filesystem. Trigger a rewrite with redis-cli BGREWRITEAOF; if it stays large, the dataset itself grew (see RedisHighMemory).

Data store containers

DataStoreMemoryNearLimit / DataStoreRestarted (warning)

Trigger: a Postgres or Redis container's working set above 90% of its memory limit (15 minutes prod, 30 minutes dev), or a restart in the last 15 minutes (a container restart, or CNPG restarting Postgres inside the pod).

kubectl get pod -n data <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'   # OOMKilled?
kubectl logs -n data <pod> -c <postgres|redis> --previous --tail=100

Postgres working set includes page cache the kernel can reclaim, so a value near the limit is not always an OOM risk; an OOMKilled reason is. Raise resources.limits.memory in the env's cluster manifest or Redis values (Redis: keep maxmemory below the limit). Postgres OOM triage is under PostgresDown.