Alert response runbook¶

What to do when an alert fires in #monitoring. Indexed by alert name — Ctrl+F the alert from the Discord message to find the right section.

Looking for daily operational commands? That's the sister page: Daily ops runbook. It covers connect-to-cluster, tail-logs, deploy/rollback, port-forward, psql/redis-cli — the things you reach for routinely. This page is for "an alert just fired, what now."

Alert sources covered: - Prometheus rules in k8s/envs/platform/manifests/alerting-rules.yaml (see Alerting) - Pyrra burn-rate alerts auto-generated from ServiceLevelObjective CRs (see SLOs). Same playbook for all SLOs — generic section at the bottom - Synthetic Worker probes — direct Discord messages from the Cloudflare Worker (see Synthetics)

Every section follows: Trigger → Debug path → Causes & recipes → Escalation.

Application¶

BackendDown (critical)¶

Trigger: up{job="backend-service"} == 0 for 2 minutes (Prometheus scrape consistently fails).

Debug path:

# 1. Pod state — first thing to look at
kubectl get pods -n prod -l app=tomoda-api -o wide
kubectl get pods -n tomoda -l app=tomoda-api -o wide   # dev for comparison

# 2. If any pod is not Running, describe it for the failure reason:
kubectl describe pod -n prod <pod-name> | tail -40
# Look for: Events section at the bottom — most-recent reason for crash

# 3. Logs from the latest crash (or current Running pod if not crashing):
kubectl logs -n prod -l app=tomoda-api --tail=200 --since=10m
kubectl logs -n prod -l app=tomoda-api --tail=200 --previous   # logs from before last restart

# 4. Crash count + restart history
kubectl get pod -n prod <pod-name> -o jsonpath='{.status.containerStatuses[*].restartCount}'
kubectl get pod -n prod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
# OOMKilled / Error / Completed — each maps to a different root cause below

Look for these specific patterns:

What you see	What it means	Recipe
`CrashLoopBackOff` + `panic:` in logs	Code-level crash on startup	Revert latest deploy. `kubectl -n prod logs <pod> --previous` to find the panic. Open a tomoda issue, link the panic
`ImagePullBackOff`	Wrong image tag or pull secret missing	`kubectl describe pod` shows the rejected image. Check `overlays/prod/kustomization.yaml` for the tag. Verify Artifact Registry has the image: `gcloud artifacts docker images list us-central1-docker.pkg.dev/development-485000/tomoda-prod-repo/tomoda-backend`
`Pending` indefinitely + `FailedScheduling` in events	Node pool exhausted	`kubectl describe pod <name>` shows the scheduling reason. Bump node pool max: `gcloud container clusters update gke-tomoda --zone us-central1-a --enable-autoscaling --node-pool=<pool> --min-nodes=1 --max-nodes=8`. Or check cluster autoscaler logs: `kubectl logs -n kube-system deploy/cluster-autoscaler --tail=200 \\| grep tomoda`
`lastState.terminated.reason: OOMKilled`	Memory limit hit	API RED dashboard → check `tomoda_http_requests_in_flight` for traffic spike. If real spike: temporary HPA scale-up + later, bump `resources.limits.memory` in overlays/{dev,prod}/. If no spike: memory leak — capture heap profile via `kubectl exec <pod> -- curl -s localhost:8080/debug/pprof/heap > heap.out`, share with backend team
Pod logs show `dial tcp ...:5432: connect: connection refused`	Backend can't reach Postgres	Stop here, run PostgresDown playbook below. This isn't really a BackendDown — it's a downstream symptom
Pod logs show `redis: connection pool timeout`	Redis is down	Run RedisDown playbook
All pods `Running`, but Prometheus says `up == 0`	Scrape problem, not real outage	Check the ServiceMonitor + NetworkPolicy: `kubectl get servicemonitor -n monitoring tomoda-backend -o yaml`. NetworkPolicy in `k8s/apps/tomoda/base/network-policy.yaml` must allow ingress from `monitoring` namespace

Rollback recipe (when a recent deploy caused it):

# 1. Find the last good image tag from git history
git -C ~/workspace/tomoda log --oneline main -20 --grep="backend"

# 2. For dev: revert the bad commit; Image Updater rolls back automatically (~2 min)
git -C ~/workspace/tomoda revert <bad-sha>
git push origin main

# 3. For prod: it's tag-driven, edit the kustomization manually
#    overlays/prod/kustomization.yaml has the image tag pinned;
#    update it back to a previous v* tag from Artifact Registry, commit, push.

Escalation: down for > 10 minutes despite the above is a full prod outage. Discord @here in #monitoring + check whether to file a status-page update.

BackendDown SLO impact¶

This alert burns the api-availability SLO budget aggressively (every minute of total downtime = ~6% of the 30-day budget). Expect a corresponding ErrorBudgetBurn alert if the outage lasts > 10 min. Don't treat both as separate incidents.

BackendHighErrorRate (warning)¶

Trigger: 5xx rate > 5% over 5 minutes.

First checks:

# Grafana → "Tomoda — API RED" dashboard → "5xx error rate by route"
# Identify which route is the offender, then:
kubectl logs -n <ns> -l app=tomoda-api --tail=200 | grep -E "ERROR|panic|5[0-9][0-9]"

Cross-check Sentry for new issues (Sentry → Issues → filter: is:unresolved age:-1h environment:prod).

Likely causes & fixes:

Cause	Recipe
Recent deploy introduced a bug	Check Sentry for new issues correlated with deploy time. Revert if confirmed
Downstream dependency failing (Photon, Stripe, OAuth provider)	Check that probe in synthetic monitor + that service's status page
DB connection pool exhausted	See PostgresHighConnections — backend errors with `pq: too many connections`
One specific route broke (e.g. Stripe webhook signature change)	The dashboard breakdown by route narrows it; targeted hotfix

BackendHighLatency (warning)¶

Trigger: p95 > 2s over 5 minutes.

First checks:

# Grafana → "Tomoda — API RED" → "Latency (p50 / p95 / p99) by route"
# Tempo: jump from Grafana into a slow trace for the offending route

Likely causes & fixes:

Cause	Recipe
Slow DB query (missing index after schema change)	Trace in Tempo shows long GORM span → `EXPLAIN ANALYZE` the query in psql → add an index migration
Connection pool saturation	Symptom: queries wait on connection acquire. See PostgresHighConnections
External service slow (Photon, Stripe)	Trace shows long span on a `fetch` to that service. Page their status page
Cold-start after autoscale	Brief, self-resolves. If chronic, raise `minReplicas` in the HPA

Database¶

PostgresDown (critical)¶

Trigger: cnpg_collector_up == 0 for 1 minute.

Suppresses backend warnings (BackendHighErrorRate, BackendHighLatency) via Alertmanager inhibition — they'll fire as a cascade and would clutter #monitoring otherwise. So if PostgresDown is firing alone, it's authoritative — the DB is the root cause, not a downstream symptom.

Debug path:

# 1. CNPG Cluster status
kubectl get cluster -n data
# Expected: Status=Cluster in healthy state with X instances, primary on Y
# Look for: Cluster in healthy state. If "primary upgrading" or "failover in
# progress", give it 60s — CNPG self-heals.

# 2. Pod state
kubectl get pods -n data -l cnpg.io/cluster=postgres-prod
kubectl describe pod -n data <pod-name> | grep -A 20 "Events:"

# 3. Postgres logs from the pod's `postgres` container
kubectl logs -n data <pod-name> -c postgres --tail=200

# Key patterns to grep for:
#   - "FATAL: the database system is starting up" → genuine startup, wait 30s
#   - "out of memory" / "OOMKilled" → OOM, see recipe below
#   - "could not write to file" / "No space left on device" → disk full
#   - "could not open file pg_wal/..." → WAL corruption (RARE, escalate)
#   - "stuck spinning on a sysconf lock" → barman backup stuck

# 4. Disk usage
kubectl exec -n data <pod-name> -c postgres -- df -h /var/lib/postgresql/data
# Watermarks: 80% → warning, 90% → tight, 95%+ → THE PROBLEM

# 5. Recent backup status (separate worry — see PostgresBackupStale)
kubectl get scheduledbackup -n data
kubectl get backup -n data --sort-by=.metadata.creationTimestamp | tail -5

Causes + recipes (most → least common):

Pod crashed + autorestart in progress¶

Symptom: pod was Running 2min ago, now Pending or Init. Fix: wait 60s. CNPG operator auto-restarts. If it crashes back into the same state, look at previous logs:

kubectl logs -n data <pod-name> -c postgres --previous | tail -100

Disk full¶

Symptom: logs full of No space left on device, df shows > 95%.

# IMMEDIATE: free space if possible to get the DB back online
# Easy wins: VACUUM FULL temp tables, drop unused indexes, kill stuck queries

# THEN expand the PVC:
# 1. Edit the CNPG Cluster resource — change spec.storage.size
kubectl edit cluster -n data postgres-prod
# Change: spec.storage.size: 10Gi  →  20Gi (or whatever)

# 2. The Cluster reconciles + expands the PVC. Watch:
kubectl get pvc -n data -w
# Expected: pvc Bound, capacity updates within a few minutes

# 3. Postgres auto-recognizes the new space; no restart needed for ext4-backed PVCs.

# Note: StorageClass MUST have allowVolumeExpansion: true. Check with:
kubectl get sc <name> -o jsonpath='{.allowVolumeExpansion}'
# If false, can't expand online. Need to:
#   - Take a backup
#   - Provision a new larger PVC
#   - Restore — disaster-recovery procedure

OOMKilled¶

Symptom: lastState.terminated.reason: OOMKilled on the postgres container.

# 1. Identify the trigger
# - Recent traffic spike? Check API RED dashboard for tomoda_http_requests_in_flight
# - Slow query running uncapped memory? Check pg_stat_activity:
task pg:console -- prod
SELECT pid, now() - query_start AS duration, state, query
  FROM pg_stat_activity
  WHERE state != 'idle'
  ORDER BY duration DESC LIMIT 10;
# Kill any obvious offender:
SELECT pg_cancel_backend(<pid>);   # gentle
SELECT pg_terminate_backend(<pid>); # firm

# 2. Bump memory in the Cluster resource (permanent fix)
kubectl edit cluster -n data postgres-prod
# spec.resources.limits.memory: 1Gi → 2Gi (or whatever current+50%)

Backup job stuck holding a lock¶

Symptom: pg_locks_count high, query state shows WaitEventType: Lock.

# Find the stuck process
kubectl exec -n data <pod> -c postgres -- bash -c "ps aux | grep barman | head"

# Kill it — CNPG starts a fresh backup on the next schedule
kubectl exec -n data <pod> -c postgres -- kill -9 <pid>

# Or restart the whole pod (CNPG safely handles primary swap if needed):
kubectl delete pod -n data <pod-name>

Connection storm (every backend pod retries)¶

Symptom: Postgres logs show FATAL: sorry, too many clients already. This is downstream from BackendHighConnections — scale that playbook.

Escalation thresholds:

5min down + no clear cause → page everyone
PVC is unrecoverable (failed disk, corrupted data) → disaster-recovery.md — point-in-time restore from GCS WAL archive
WAL corruption errors in logs → DO NOT restart blindly; talk to backend team first to avoid making it worse

PostgresDown SLO impact¶

This burns multiple SLOs simultaneously: api-availability (backend errors as DB calls fail), login-success (logins fail), async-task-success (workers can't process). Don't be surprised to see 3+ burn-rate alerts firing alongside.

PostgresReplicationLag (warning)¶

Trigger: Replica lag > 10s for 5 minutes.

First checks:

kubectl exec -n data postgres-prod-1 -- psql -c "SELECT * FROM pg_stat_replication;"

lag_bytes and state columns tell you which replica + how far behind.

Likely causes & fixes: heavy write load, slow network between primary and replica, replica disk slower than primary. Usually transient. If chronic, the replica needs more resources or a co-located placement.

PostgresHighConnections (warning)¶

Trigger: Connections > 80% of max_connections for 5 minutes.

First checks:

kubectl exec -n data postgres-prod-1 -- psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

Likely causes & fixes:

Cause	Recipe
Backend connection leak (forgot to defer close)	Find the leaky code path in tomoda; deploy fix
Long-running queries blocking pool turnover	`SELECT pid, query_start, state, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start;` → identify + kill with `pg_cancel_backend(pid)`
Connection pool sized too small	Bump `DB_MAX_CONNECTIONS` config on the backend

PostgresBackupStale (warning)¶

Trigger: Last WAL archive > 1h ago.

First checks: kubectl logs -n data <pod> -c bootstrap-controller for barman errors. Check the GCS backup bucket via gcloud storage ls gs://tomoda-db-backups-development-485000/.

Likely causes & fixes:

Cause	Recipe
GCP backup SA lost permissions	`gcloud projects get-iam-policy development-485000 \| grep cnpg-backup-sa` — should have `roles/storage.objectAdmin` on the backup bucket
Workload Identity binding broke (rare, after cluster restructure)	Re-bind: `gcloud iam service-accounts add-iam-policy-binding cnpg-backup-sa@... --role=roles/iam.workloadIdentityUser --member="serviceAccount:development-485000.svc.id.goog[data/postgres-prod-cluster]"`
Backup bucket is full or has lifecycle issues	`gcloud storage du gs://tomoda-db-backups-development-485000/ --readable-sizes`

Escalation: If backups have been broken for > 24h, this is a data-durability risk. Block deploys until fixed.

PostgresDiskUsageHigh (warning)¶

Trigger: DB size > 15 GB.

Not urgent. Plan for PVC expansion + index cleanup over a maintenance window. gcloud storage lifecycle policies do not apply (Postgres is in-cluster PVCs, not GCS).

Redis¶

RedisDown (critical)¶

Trigger: redis_up == 0 for 1 minute. Suppresses RedisHighMemory.

First checks:

kubectl get pods -n data -l app.kubernetes.io/name=redis
kubectl logs -n data <redis-pod> --tail=100

Likely causes & fixes:

Cause	Recipe
Pod crash	Auto-restarts. If repeated, check memory limit (OOM kills)
PVC issue	`kubectl describe pod <redis-pod>` → look for volume mount errors
Config error after a Helm upgrade	Roll back: `helm rollback redis -n data` (or git revert)

Impact: Most reads gracefully degrade (Redis caches DB reads). But background-job dispatch via Asynq breaks immediately — queue depth on AsyncQueueBacklog will then fire.

RedisHighMemory (warning)¶

Trigger: Memory > 90% of max for 5 minutes.

First checks:

kubectl exec -n data <redis-pod> -- redis-cli INFO memory
kubectl exec -n data <redis-pod> -- redis-cli MEMORY STATS

Likely causes & fixes:

Cause	Recipe
Cache hot keys never expire (TTL missing)	`redis-cli --bigkeys` finds the offenders; add TTL in the backend code
Asynq queue growth (jobs not draining)	See AsyncQueueBacklog
Memory limit set too low for current load	Bump `master.resources.limits.memory` in the chart values

Kubernetes resources¶

PodCrashLooping (warning)¶

Trigger: Restart rate > 0 in 15 minutes on a pod in tomoda / prod / data.

First checks:

kubectl get pods -A | grep -v "Running\|Completed"
kubectl describe pod -n <ns> <pod-name>
kubectl logs -n <ns> <pod-name> --previous   # logs from before the latest crash

Likely causes: all over the map. Could be liveness probe too aggressive, OOM, panic on startup, config typo. Logs are the entry point.

PVCUsageHigh (warning)¶

Trigger: PVC > 85% full for 10 minutes.

First checks:

kubectl get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CAP:.status.capacity.storage,USED:.status.allocatedResources

Likely causes & fixes:

Cause	Recipe
Postgres data growth	See PostgresDiskUsageHigh; plan expansion
Loki retention not enforcing	Should be capped at 7d; check Loki compactor logs. If broken, `kubectl rollout restart deploy loki -n monitoring`
Tempo storage local PVC fills	Tempo lifecycle pushes to GCS but stale local data accumulates if compactor stalls. Restart Tempo

NodeMemoryHigh (warning)¶

Trigger: Node memory > 90% for 10 minutes.

First checks: Grafana → "Kubernetes cluster overview" → Node memory by pod. Find the noisy neighbour.

Recipe: If pod is legit (real traffic) and not OOM-killing yet, autoscaler usually brings up another node. Verify cluster-autoscaler is running. If a single pod is grossly over-consuming, that's a memory leak — investigate.

Business metrics¶

AsyncQueueBacklog (warning)¶

Trigger: Asynq pending count > 500 for 10 minutes.

First checks: Grafana → "Tomoda — Async Worker" → task rate by type. If a specific task type is failing repeatedly, it'll show on the failure-rate chart.

Likely causes & fixes:

Cause	Recipe
Worker pod is down	See pod status; restart
One task type is consistently failing → retries pile up	Check `tomoda_async_task_total{status="failed"}` by type; logs reveal the failure; fix the handler
Real traffic spike	Increase `tomoda-async` replica count (HPA usually handles this in prod)
Redis down	RedisDown should also be firing

WSHubConnectionDrop (warning)¶

Trigger: Hub dropping > 1 conn/sec for 5 minutes.

First checks:

kubectl logs -n <ns> -l app=tomoda-api,role=ws-hub --tail=200 | grep -i "close\|disconnect"
# Grafana → "Tomoda — WebSocket Hub" → broadcast latency + in-flight messages

Likely causes & fixes:

Cause	Recipe
Recent deploy rolled WS pods, clients reconnect en masse	Self-resolves within 1-2 min
Redis pub/sub broken (cross-pod fanout fails)	See RedisDown; cross-check `tomoda_ws_messages_total{direction="fanout"}` flatlining
Network blip / GKE node preemption	Spot nodes get evicted; usually self-heals via the autoscaler
Memory pressure on hub pods causing forced shutdowns	Check `tomoda_ws_connections_active` per pod; if one pod is way out of balance + showing OOMKills, bump memory

Certificates¶

CertificateExpiringSoon (warning)¶

Trigger: Cert expires in < 14d.

First checks:

kubectl get certificate -A
kubectl describe certificate <name> -n <ns>

Likely causes & fixes: cert-manager should auto-renew at the 30d-remaining mark. If it hasn't by 14d, look at the cert-manager logs:

kubectl logs -n cert-manager deploy/cert-manager --tail=200 | grep <cert-name>

Common: ACME challenge failing because external-dns hasn't pushed the TXT record yet (Cloudflare API rate limit?), or the DNS-01 vs HTTP-01 issuer config flipped. Manually trigger renewal: kubectl annotate cert <name> -n <ns> cert-manager.io/issue-temporary-certificate=true --overwrite.

CertificateRenewalFailed (critical)¶

Trigger: cert-manager reports Ready=False for > 1h.

First checks: Same as above. This is the louder version of CertificateExpiringSoon — the auto-renewal has actively failed, not just "hasn't happened yet."

Recipe: Often the underlying issuer is misconfigured or rate-limited. Check Let's Encrypt rate-limit status: https://crt.sh/?Identity=<your-domain>. If hit, switch to the LE staging issuer temporarily.

Escalation: If TLS will expire in < 24h, this is critical — page someone.

Synthetic monitor¶

Discord 🚨 messages directly from the Cloudflare Worker — not a PrometheusRule, no Alertmanager involvement. See Synthetics for architecture.

STILL-DOWN reminders: if a probe stays down for > 1h, the Worker re-fires a 🚨 STILL DOWN each hour. Resolving the underlying issue + the next cron tick (within 1 min) triggers ✅ RECOVERED.

`api` / `api-dev` DOWN¶

What it means: /health returned non-200 or timed out. The backend is unreachable from Cloudflare's edge.

Debug path:

# 1. Cross-check with BackendDown — usually fires in parallel
#    Discord #monitoring should already have a 🚨 BackendDown if it's a pod issue.

# 2. Test the endpoint from your laptop (outside the cluster):
curl -v https://api.tomoda.life/health
# - 200 = fine, transient blip; wait for ✅ RECOVERED
# - 5xx = cluster reachable but backend errored — see BackendDown above
# - Cert error = run CertificateRenewalFailed playbook
# - DNS NXDOMAIN = check external-dns:
kubectl logs -n external-dns deploy/external-dns --tail=50 | grep tomoda.life
# - Connection refused = ingress / Traefik issue:
kubectl get svc -n traefik-system traefik
kubectl logs -n traefik-system deploy/traefik --tail=100 | grep ERROR

# 3. Test from inside the cluster (rules out Cloudflare):
kubectl run -it --rm -n tomoda netshoot --image=nicolaka/netshoot --restart=Never -- \
  curl -v http://tomoda-api/health
# 200 in-cluster + non-200 from outside = the ingress/DNS/cert layer is broken,
#   NOT the backend

Most likely causes (in order):

Backend pods down → run BackendDown playbook
Ingress / Traefik issue → kubectl rollout restart deploy/traefik -n traefik-system
DNS propagation lag (e.g. just after a CNAME change) → wait 1-2 min, check dig api.tomoda.life resolves
Cert expired → CertificateRenewalFailed playbook

`app` / `app-dev` DOWN¶

Debug path:

# 1. Pod status
kubectl get pods -n <ns> -l app=tomoda-frontend
kubectl logs -n <ns> -l app=tomoda-frontend --tail=200

# 2. DNS + cert
curl -vk https://app.tomoda.life 2>&1 | grep -E "HTTP/|expire|verify"

# 3. Look for static-asset fetch failures in the frontend HTML response
curl -s https://app.tomoda.life | grep -oE 'src="[^"]*"' | head -5
# If src paths are broken (e.g. pointing at the old assets bucket), recent
# frontend deploy probably mis-built the asset URLs.

Most likely causes: frontend pod crash (similar recipe to BackendDown but namespace-scoped to the frontend Deployment), or a regression in build-time asset URL generation.

`assets` DOWN¶

Debug path:

# 1. CloudFront status
aws cloudfront list-distributions \
  --query 'DistributionList.Items[?Aliases.Items[?contains(@,`assets`)]].{Id:Id,Status:Status,Enabled:Enabled}'
# Status MUST be "Deployed", Enabled MUST be true.

# 2. Origin (S3) responding
aws s3 ls s3://tomoda-assets-prod/ | head -3   # or tomoda-assets-dev

# 3. ACM cert valid
aws acm list-certificates --region us-east-1 \
  --query 'CertificateSummaryList[?DomainName==`assets.tomoda.life`].Status'
# MUST be "ISSUED". Note: CloudFront ACM certs live in us-east-1 specifically.

Most likely causes:

CloudFront distribution accidentally disabled
S3 bucket policy got tightened in a way that breaks public reads
ACM cert expired (rare — ACM auto-renews)

Recipe: Re-enable the distribution via console or aws cloudfront update-distribution --id <id> --distribution-config <config> with Enabled=true. Wait ~15 min for propagation.

`argocd` DOWN¶

Important context: loss of Argo CD doesn't break running workloads — it just halts GitOps. Apps already deployed keep running. This is less urgent than a backend outage.

Debug path:

kubectl get pods -n argocd
kubectl logs -n argocd deploy/argocd-server --tail=100
kubectl logs -n argocd deploy/argocd-repo-server --tail=100

Most likely causes: repo-server OOMKilled on a large Helm chart, Dex (OAuth) misconfig after a Google OAuth client rotation, Argo CD self-update broken.

Recipe: kubectl rollout restart on the offending Deployment usually fixes it. If repo-server keeps crashing, bump its memory limit in infrastructure/gcp/argocd.tf and terraform apply.

`docs` DOWN¶

Debug path:

1. Cloudflare dashboard → Workers & Pages → tomoda-docs → Deployments
   - Latest deployment "Success"? If not, click into it for the build log.
   - Stuck "Building" for > 5 min? Cancel + retry.

2. From your laptop:
curl -vk https://docs.tomoda.life
# Should be 302 to Cloudflare Access. If 404/500, last build failed.

3. If build failed, the typical culprit is:
   - Submodule clone (PAT expired? Pages-app access revoked?)
   - mkdocs --strict failing on a broken link in a recently-merged doc PR

Recipe: If a doc PR broke the build, revert it OR push a fix; Pages auto-rebuilds. If the build script itself is broken, fix in docs/operations/docs-site.md's Build command (which is documented but Cloudflare-side configured).

`login` DOWN (positive-path login probe)¶

What it means: real-credentials login returned non-200. The auth critical path is broken.

Debug path:

# 1. Cross-check with BackendDown — if backend is down everywhere, that's the root cause.

# 2. Try the login flow with curl using the same credentials the Worker uses:
PASSWORD=$(gcloud secrets versions access latest --secret=tomoda-synthetic-probe-password)
curl -v -X POST https://api.tomoda.life/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d "{\"email\":\"synthetic+probe@tomoda.life\",\"password\":\"$PASSWORD\"}"

# Status codes:
# - 200 + JWT = login works. Worker may be using a stale password — wait for next cron.
# - 401 = password in GCP SM doesn't match the bcrypt hash in DB. Causes:
#   a) Password rotated in GCP SM but backend not redeployed since.
#   b) Synthetic user got deleted from DB.
# - 5xx = backend handler broken or DB unreachable. Run BackendHighErrorRate playbook.

# 3. Verify the synthetic user exists + check account_type:
task pg:console -- prod
SELECT id, email, account_type, is_active, created_at, updated_at
  FROM users WHERE email = 'synthetic+probe@tomoda.life';
# Expected: account_type='synthetic', is_active=true

Recipes by cause:

Cause	Fix
GCP SM password rotated, backend not redeployed	`kubectl rollout restart deploy/tomoda-api -n prod` → backend boots → seed re-runs idempotently → DB password hash updates → next cron returns 200
Synthetic user deleted	Same fix (restart triggers seed recreate). If that doesn't restore the user, inspect seed.go logs for errors
Password files diverged between GCP SM and the Worker's Terraform-bound value	Re-run `terraform apply` in `infrastructure/cloudflare/` with the latest `TF_VAR_synthetic_probe_password` from GCP SM

`login-bad-password` DOWN (🔴 SECURITY)¶

What it means: the Worker sent a bogus password and got 200 back (or 5xx). 401 was expected. A 200 means any password is being accepted.

Severity: treat as a security incident until proven otherwise. The synthetic user could be used as a backdoor.

Immediate actions (in order):

# 1. Reproduce manually to confirm:
curl -v -X POST https://api.tomoda.life/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"synthetic+probe@tomoda.life","password":"definitely-wrong-password-12345"}'
# - 401 = false alarm (the Worker had a transient issue); wait for next cron
# - 200 with JWT = CONFIRMED security regression. Continue with mitigation.
# - 5xx = backend bug, not a security issue but still urgent

# 2. If confirmed 200 from arbitrary passwords:
#    a) Identify the latest deploy
kubectl get deploy tomoda-api -n prod -o jsonpath='{.spec.template.spec.containers[0].image}'
git log --oneline main -10   # in tomoda repo — what's been deployed lately

#    b) Rollback to the previous known-good image
#       (edit overlays/prod/kustomization.yaml back to previous tag, push, Argo syncs)

#    c) Disable the synthetic user as a safety measure while debugging:
task pg:console -- prod
UPDATE users SET is_active=false WHERE email='synthetic+probe@tomoda.life';

# 3. Investigate root cause:
#    - Find the auth handler diff between the broken commit and last good
#    - Common culprits: bcrypt comparison removed/short-circuited, dev-mode
#      "accept any password" flag accidentally enabled in prod config,
#      middleware order changed

Escalation: notify all engineers immediately. This is a critical-severity security alert.

`auth-middleware` DOWN (🔴 SECURITY)¶

What it means: an unauthenticated GET to a protected endpoint (/api/v1/users/me) returned 200 instead of 401. The auth middleware is letting anonymous traffic through.

Severity: treat as a security incident. Protected user data may be exposed.

Immediate actions (in order):

# 1. Reproduce manually:
curl -v https://api.tomoda.life/api/v1/users/me
# - 401 = false alarm
# - 200 = CONFIRMED middleware bypass. Continue.
# - 404 = the probe URL doesn't match the protected route layout anymore;
#         not a security issue, but update worker.js to a current route

# 2. Test a few more known-protected endpoints to scope the damage:
curl -v https://api.tomoda.life/api/v1/events
curl -v https://api.tomoda.life/api/v1/me
# Confirm whether the bypass is scoped to one route or all of them.

# 3. Rollback to the previous known-good image (same procedure as login-bad-password)

# 4. If the route layout changed legitimately (refactor), update the probe URL in
#    synthetic/worker.js, push, terraform apply in infrastructure/cloudflare/.

Common causes:

Middleware ordering changed — auth middleware ran AFTER the route handler instead of before
A new route forgot to apply the auth middleware decorator
A wildcard "skip auth on /api/v1/_" was added too broadly
Auth middleware itself was removed by mistake

Escalation: same as login-bad-password — critical. Even if the affected route layer is small, "the middleware doesn't work" is a load-bearing security invariant.

SLO burn-rate alerts (generic playbook)¶

Pyrra emits 3 alerts per SLO — see SLOs for the multi-window burn-rate scheme. Alert names follow the pattern ErrorBudgetBurn_<slo-name>_<window> (e.g. ErrorBudgetBurn_api_availability_1h).

Trigger interpretation:

Window	Threshold	Meaning
1h	14.4× burn rate	2% of monthly error budget consumed in last hour
6h	6× burn rate	5% consumed in last 6 hours
3d	1× burn rate	10% drift over last 3 days — chronic, not acute

First checks per SLO:

Identify the indicator — e.g. for api-availability, it's tomoda_http_requests_total ratio. Grafana → SLO dashboard → see which route is bleeding the budget.
Check whether a rule-based alert is also firing for the same area (e.g. BackendHighErrorRate for the availability SLO). If yes, treat it as the same incident; run the rule's runbook above.
Decide: acute outage (1h burn rate firing) or chronic drift (3d window only)?
- Acute: run the corresponding rule-based playbook above; the SLO alert will silence once the root cause is fixed.
- Chronic: this is a "we've slowly degraded over weeks" alert. Often no single recent event caused it — it's a signal to invest in reliability work for that area.

Tightening or loosening targets: if an SLO is consistently green (95%+ budget remaining) → target is too lax, tighten. If consistently red and the system is healthy → target is unrealistic; loosen or invest in the underlying service. Don't chase "always green" — SLOs lose meaning.

When to escalate to human comms¶

Some incidents need a Discord @here ping or status-page update, not just a runbook fix:

BackendDown / PostgresDown / RedisDown lasting > 10 minutes
CertificateRenewalFailed with cert expiring in < 24h
Multiple critical alerts firing simultaneously (likely cluster-wide issue)
Any signal that user data is at risk (PostgresBackupStale + active writes)

The Discord webhook doesn't @-mention by default — operator's call to escalate.

Adding a new alert¶

When you add a new rule to alerting-rules.yaml (or a new SLO), add a corresponding runbook section here in the same PR. The "alert fires → developer reads runbook → developer doesn't know what to do" path is the failure mode this doc exists to prevent.

Section template:

### AlertName (severity)

**Trigger:** the PromQL expression in plain English.

**First checks:** kubectl / curl / Grafana commands.

**Likely causes & fixes:** table of cause → recipe.

**Escalation:** when to wake people up.

Alerting — the routing + throttling layer all PrometheusRule alerts pass through
SLOs — the Pyrra SLO catalogue + how burn-rate alerts are generated
Synthetics — the Cloudflare Worker that emits probe DOWN/RECOVERED messages
Disaster recovery — for catastrophic database failure scenarios

Alert response runbook¶

Application¶

BackendDown (critical)¶

BackendDown SLO impact¶

BackendHighErrorRate (warning)¶

BackendHighLatency (warning)¶

Database¶

PostgresDown (critical)¶

Pod crashed + autorestart in progress¶

Disk full¶

OOMKilled¶

Backup job stuck holding a lock¶

Connection storm (every backend pod retries)¶

PostgresDown SLO impact¶

PostgresReplicationLag (warning)¶

PostgresHighConnections (warning)¶

PostgresBackupStale (warning)¶

PostgresDiskUsageHigh (warning)¶

Redis¶

RedisDown (critical)¶

RedisHighMemory (warning)¶

Kubernetes resources¶

PodCrashLooping (warning)¶

PVCUsageHigh (warning)¶

NodeMemoryHigh (warning)¶

Business metrics¶

AsyncQueueBacklog (warning)¶

WSHubConnectionDrop (warning)¶

Certificates¶

CertificateExpiringSoon (warning)¶

CertificateRenewalFailed (critical)¶

Synthetic monitor¶

api / api-dev DOWN¶

app / app-dev DOWN¶

assets DOWN¶

argocd DOWN¶

docs DOWN¶

login DOWN (positive-path login probe)¶

login-bad-password DOWN (🔴 SECURITY)¶

auth-middleware DOWN (🔴 SECURITY)¶

SLO burn-rate alerts (generic playbook)¶

When to escalate to human comms¶

Adding a new alert¶

Related¶

`api` / `api-dev` DOWN¶

`app` / `app-dev` DOWN¶

`assets` DOWN¶

`argocd` DOWN¶

`docs` DOWN¶

`login` DOWN (positive-path login probe)¶

`login-bad-password` DOWN (🔴 SECURITY)¶

`auth-middleware` DOWN (🔴 SECURITY)¶