Data alert runbook¶
What to do when a Postgres or Redis alert fires in #monitoring. Rules live in k8s/envs/platform/manifests/postgres-alerts.yaml, redis-alerts.yaml and the tomoda.data.containers group of alerting-rules.yaml; each rule's runbook_url links its section here. Everything else: Alert response runbook.
Every alert carries an env label, shown in the Discord message. Pairs are critical for prod and warning for dev at the same threshold; a heading without "dev" is prod only. Selectors use the scrape job (data/postgres-<env>, redis-<env>-metrics), not a cluster label: CNPG sets cluster only on cnpg_collector_up.
Database¶
PostgresDown (critical prod / warning dev)¶
Trigger: cnpg_collector_up == 0, or no cnpg_collector_up series at all (pod gone, scrape failing), for 1 minute. Suppresses every severity=warning alert with the same env, backend warnings included.
Debug path:
kubectl get cluster -n data # "Cluster in healthy state"? failover in progress: wait 60s
kubectl get pods -n data -l cnpg.io/cluster=postgres-prod
kubectl describe pod -n data <pod-name> | grep -A20 Events:
kubectl logs -n data <pod-name> -c postgres --tail=200
# "the database system is starting up" → startup, wait
# "out of memory" / OOMKilled → see OOMKilled
# "No space left on device" → see Disk full
# "could not open file pg_wal/..." → WAL corruption: escalate, don't restart blindly
Pod crashed, restarting: CNPG restarts it. If it crashes again, read kubectl logs -n data <pod> -c postgres --previous.
Disk full: local-path PVCs are directories on the server's root filesystem; spec.storage.size is a budget, not a quota. "Full" means the NVMe is full, and NodeFilesystem* alerts fire too. Free space on the server (sudo du -sh /var/lib/rancher/k3s/storage/*, old images, Photon tarballs), then trim data. See the disk budget.
OOMKilled: find the trigger first.
task pg:console -- prod
SELECT pid, now() - query_start AS duration, state, query
FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;
SELECT pg_cancel_backend(<pid>); -- or pg_terminate_backend
Then raise spec.resources.limits.memory in k8s/envs/prod/postgres/manifests/cluster.yaml through Git.
Connection storm (too many clients already): see PostgresHighConnections.
Escalation:
- Down 5 minutes with no clear cause:
@here - Data unrecoverable (failed disk, corruption): Disaster Recovery, point-in-time restore from R2
- WAL corruption in logs: talk to the backend team before restarting
SLO impact: burns api-availability, login-success and async-task-success at once.
PostgresNoPrimary (critical prod / warning dev)¶
Trigger: every instance of the cluster reports pg_is_in_recovery() for 5 minutes: nothing accepts writes. Usually a restore or a replica-mode cluster that never got promoted.
kubectl get cluster -n data postgres-prod -o jsonpath='{.status.currentPrimary} {.status.phase}{"\n"}'
kubectl logs -n cnpg-system deploy/cnpg-operator-cloudnative-pg --tail=200 | grep -i -E "promot|switchover|failover"
A recovered cluster still bootstrapping from R2 is expected; wait for "Cluster in healthy state". Otherwise check spec.replica.enabled in the cluster manifest and promote through Git.
PostgresReplicationLag (critical prod / warning dev)¶
Trigger: replica lag > 10s for 5 minutes. Dormant while both clusters run one instance.
kubectl exec -n data postgres-prod-1 -c postgres -- psql -c "SELECT * FROM pg_stat_replication;" shows which replica and how far behind. Usually heavy write load and transient.
PostgresHighConnections (critical prod / warning dev)¶
Trigger: backends (cnpg_backends_total) > 80% of max_connections (200 prod, 100 dev) for 5 minutes. PostgresConnectionsNearMax is the 70% prod early warning.
kubectl exec -n data postgres-prod-1 -c postgres -- psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
| Cause | Recipe |
|---|---|
| Connection leak in the backend | Find the code path; deploy a fix |
| Long-running queries holding connections | List non-idle queries by query_start; pg_cancel_backend(pid) |
Pods × DB_MAX_CONNS near the cap after a scale-up |
Lower DB_MAX_CONNS or add the pooler (Scaling) |
PostgresBackupStale / PostgresWALArchivingFailing (critical)¶
Trigger: PostgresBackupStale: the last archived WAL segment is > 1h old (dev too, as a warning). PostgresWALArchivingFailing: the latest archive attempt failed and none has succeeded since, for 10 minutes (prod). Unarchived WAL stays on disk, so PostgresDiskUsageHigh follows if this lasts.
Check kubectl logs -n data postgres-prod-1 -c plugin-barman-cloud --tail=50 (the archiver runs in that sidecar; no such container means the plugin was not injected, see kubectl get pods -n cnpg-system -l app.kubernetes.io/name=plugin-barman-cloud) and rclone lsd r2:tomoda-db-backups/prod/postgres-prod/.
| Cause | Recipe |
|---|---|
| R2 credentials wrong or revoked | kubectl get externalsecret r2-backup-credentials -n data should be SecretSynced; AccessDenied / InvalidAccessKeyId in logs means the token in tomoda-r2-cluster-credentials is wrong |
| R2 unreachable | kubectl exec -n data postgres-prod-1 -c postgres -- curl -sI https://8d3b225f38b3bcd337b9973d128c0eb6.r2.cloudflarestorage.com should return an HTTP status |
Escalation: broken > 24h is a durability risk. Hold deploys until fixed.
PostgresBaseBackupFailed / PostgresBaseBackupStale (critical)¶
Trigger: the latest base backup of postgres-prod failed, or none completed in 26h (daily at 03:00 UTC).
kubectl get backups -n data --sort-by=.metadata.creationTimestamp | tail -5
kubectl describe backup -n data <latest>
kubectl logs -n cnpg-system deploy/cnpg-operator-cloudnative-pg --tail=200 | grep -i backup
kubectl logs -n data postgres-prod-1 -c plugin-barman-cloud --tail=100 # barman-cloud-backup output
The backup metrics (barman_cloud_cloudnative_pg_io_*) come from the plugin sidecar, which refreshes them from the R2 backup catalog when it starts and every 30 minutes. A sidecar OOM kill (kubectl describe pod postgres-prod-1 -n data) means instanceSidecarConfiguration.resources on the postgres-prod-backups ObjectStore is too small.
Same credentials as WAL archiving, so PostgresBackupStale usually fires too.
OTABackupStale (critical prod / warning dev)¶
Trigger: CronJob xavia-ota-db-backup-{env} in platform hasn't completed in 26h (daily: prod 03:30 UTC, dev 03:45 UTC). Also fires for a CronJob that has never succeeded.
kubectl get cronjob,jobs -n platform | grep xavia-ota-db-backup
kubectl logs -n platform job/<latest> -c dump
kubectl logs -n platform job/<latest> -c upload
Usual causes: the Xavia Postgres is down (dump can't connect), or the R2 token can't write tomoda-db-backups. Restore: Disaster Recovery.
PostgresLongRunningTransaction / PostgresLockWaits / PostgresDeadlocks (warning)¶
Trigger: a transaction open for more than 10 minutes, more than 5 backends waiting on a lock for 5 minutes, or any deadlock in 15 minutes (prod).
task pg:console -- prod
SELECT pid, now() - xact_start AS age, state, wait_event_type, left(query, 80)
FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY age DESC LIMIT 10;
SELECT pid, pg_blocking_pids(pid) AS blocked_by, left(query, 80)
FROM pg_stat_activity WHERE cardinality(pg_blocking_pids(pid)) > 0;
SELECT pg_cancel_backend(<pid>); -- or pg_terminate_backend
| Cause | Recipe |
|---|---|
idle in transaction session (a psql left open, a code path missing commit) |
Terminate it; fix the code path |
| Migration or seed load holding a table lock | Let it finish if it is progressing; schedule big ones off-peak |
| Deadlocks | The Postgres log names both statements (kubectl logs -n data postgres-prod-1 -c postgres \| grep -A5 deadlock); make the code take locks in one order |
PostgresXIDAgeHigh / PostgresXIDWraparoundImminent (warning / critical)¶
Trigger: a database's transaction ID age passes 500M for 30 minutes (warning, prod and dev) or 1.2B for 5 minutes (critical, prod). autovacuum_freeze_max_age is 200M, so 500M means anti-wraparound vacuum is not keeping up; Postgres stops accepting writes near 2.1B.
SELECT relname, age(relfrozenxid) FROM pg_class WHERE relkind = 'r' ORDER BY 2 DESC LIMIT 10;
SELECT pid, now() - xact_start, state FROM pg_stat_activity WHERE backend_xmin IS NOT NULL ORDER BY 2 DESC;
A long-open transaction or an abandoned replication slot pins the horizon: end it first. Then VACUUM (FREEZE, VERBOSE) <table>; the oldest tables.
PostgresDiskUsageHigh / PostgresDiskUsageCritical / PostgresDiskFillingUp¶
Trigger: databases plus WAL pass 80% (warning, prod and dev) or 90% (critical, prod) of the PVC request, or the 6h trend crosses it within 24h (warning, prod and dev). local-path volumes report no usage, so the PVC size (150 GiB prod, 3 GiB dev) is a budget, not a quota: the real limit is the root filesystem (disk budget).
If WAL is the part growing, PostgresWALArchivingFailing is the cause. Otherwise find what grew (SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10; via task pg:console -- prod), then trim, raise spec.storage.size if the root filesystem has room, or plan more disk.
PostgresCacheHitRatioLow (warning)¶
Trigger: under 90% of block reads come from shared buffers for 30 minutes while reads exceed 100 blocks/s (prod). The working set outgrew shared_buffers (4 GiB prod), or a query is scanning a large table.
Look for sequential scans on big tables (SELECT relname, seq_scan, seq_tup_read FROM pg_stat_user_tables ORDER BY seq_tup_read DESC LIMIT 10;) and add the missing index. If the data simply grew, raise shared_buffers with the memory limit in k8s/envs/prod/postgres/manifests/cluster.yaml.
Redis¶
RedisDown (critical prod / warning dev)¶
Trigger: redis_up == 0, or no redis_up series (exporter gone), for 1 minute. Suppresses the other Redis warnings for the same env.
kubectl get pods -n data -l app.kubernetes.io/name=redis
kubectl logs -n data <redis-pod> --tail=100
Usual causes: pod crash (auto-restarts), volume mount errors (kubectl describe pod), or a bad values change (revert the commit).
Impact: Asynq job dispatch stops and backend pods fail readiness; async-task-success starts burning.
RedisHighMemory (critical prod / warning dev)¶
Trigger: used memory > 90% of maxmemory for 5 minutes. RedisMemoryNearMax is the prod 80% early warning (15 minutes). maxmemory sits below the container limit with noeviction (values in Redis): at the cap Redis rejects writes (enqueues fail) instead of evicting jobs or being OOMKilled.
kubectl exec -n data <redis-pod> -- redis-cli INFO memory
kubectl exec -n data <redis-pod> -- redis-cli --bigkeys
| Cause | Recipe |
|---|---|
| Keys without a TTL | --bigkeys finds them; add a TTL in the backend |
| Asynq queues not draining | tomoda_async_task_total{status="failure"} by task type and the async pod logs |
| Load outgrew the cap | Raise --maxmemory and master.resources.limits.memory together in k8s/envs/{dev,prod}/redis/values.yaml |
RedisMaxmemoryUnset (warning)¶
Trigger: Redis reports maxmemory 0 for 15 minutes. The memory alerts skip an unbounded instance, and without the cap the container limit OOMKills Redis instead of rejecting writes. Restore --maxmemory in k8s/envs/{dev,prod}/redis/values.yaml (kubectl exec -n data <redis-pod> -- redis-cli CONFIG GET maxmemory confirms).
RedisRejectingWrites (critical prod / warning dev)¶
Trigger: any command failed with OOM in 5 minutes: Redis is at maxmemory and, with noeviction, rejects writes. Asynq enqueues fail now. Follow RedisHighMemory; the fastest relief is draining or deleting a stuck queue, then raising the cap.
RedisClientsNearMax / RedisRejectedConnections¶
Trigger: connected clients above 80% of maxclients for 5 minutes (warning, prod), or any connection rejected at maxclients in 5 minutes (critical prod / warning dev).
kubectl exec -n data <redis-pod> -- redis-cli CLIENT LIST | awk '{print $2}' | cut -d= -f2 | cut -d: -f1 | sort | uniq -c | sort -rn | head
Many connections from one pod IP is a client leak in that workload; pods × pool size near the cap after a scale-up means lowering the pool size.
RedisPersistenceFailing (critical prod / warning dev)¶
Trigger: the last RDB save, AOF write or AOF rewrite failed, for 5 minutes. A restart loses writes since the last good save.
kubectl exec -n data <redis-pod> -- redis-cli INFO persistence
kubectl logs -n data <redis-pod> -c redis --tail=100 | grep -i -E "aof|rdb|fork|disk"
Usual causes: the root filesystem is full (see the NodeFilesystem* alerts), or a fork failing under memory pressure.
RedisDiskUsageHigh (warning)¶
Trigger: the append-only file passes 80% of the PVC request (10 GiB prod, 1 GiB dev) for 30 minutes. Like Postgres, the PVC is a budget on the root filesystem. Trigger a rewrite with redis-cli BGREWRITEAOF; if it stays large, the dataset itself grew (see RedisHighMemory).
Data store containers¶
DataStoreMemoryNearLimit / DataStoreRestarted (warning)¶
Trigger: a Postgres or Redis container's working set above 90% of its memory limit (15 minutes prod, 30 minutes dev), or a restart in the last 15 minutes (a container restart, or CNPG restarting Postgres inside the pod).
kubectl get pod -n data <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}' # OOMKilled?
kubectl logs -n data <pod> -c <postgres|redis> --previous --tail=100
Postgres working set includes page cache the kernel can reclaim, so a value near the limit is not always an OOM risk; an OOMKilled reason is. Raise resources.limits.memory in the env's cluster manifest or Redis values (Redis: keep maxmemory below the limit). Postgres OOM triage is under PostgresDown.
Related¶
- Postgres operations and Redis
- Disaster recovery: restoring Postgres from R2
- Dashboards: the Postgres and Redis dashboards chart every signal these alerts read