Alert response runbook¶
What to do when an alert fires in #monitoring. Search this page for the alert name in the Discord message. For routine commands (connect, logs, psql, redis-cli) see the Daily ops runbook.
Alert sources:
- Prometheus rules in
k8s/envs/platform/manifests/alerting-rules.yamlandcost-alerts.yaml(Alerting); Postgres and Redis rules inpostgres-alerts.yamlandredis-alerts.yamlhave their own Data alert runbook;location-harvest-alerts.yamllinks the app's harvesting page - Pyrra burn-rate alerts generated from
ServiceLevelObjectiveCRs (SLOs); one generic playbook at the bottom - Synthetic Worker probes: Discord messages sent directly by the Cloudflare Worker (Synthetics)
Most rules come in pairs: critical for prod, warning for dev, same threshold. Headings note where that applies.
Application¶
BackendDown (critical prod / warning dev)¶
Trigger: tomoda-api in tomoda-prod (or tomoda-dev) has 0 available replicas for 2 minutes. Suppresses BackendHighErrorRate and BackendHighLatency for the same env.
Debug path:
kubectl get pods -n tomoda-prod -l app=tomoda-api -o wide
kubectl describe pod -n tomoda-prod <pod-name> | tail -40 # Events: most recent failure reason
kubectl logs -n tomoda-prod -l app=tomoda-api --tail=200 --since=10m
kubectl logs -n tomoda-prod <pod-name> --previous --tail=200 # before the last restart
kubectl get pod -n tomoda-prod <pod-name> \
-o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}' # OOMKilled / Error
| What you see | Meaning | Recipe |
|---|---|---|
CrashLoopBackOff + panic: in logs |
Crash on startup | Roll back (Rollback); open a tomoda issue with the panic |
| Readiness failing, logs show Postgres or Redis errors | The readiness probe checks Postgres and Redis, so a data-tier outage takes every pod out of service | Run PostgresDown or RedisDown instead |
ImagePullBackOff |
Wrong tag or GHCR credentials | Check the tag in k8s/apps/tomoda/overlays/prod/kustomization.yaml exists (task images:list -- tomoda-backend) and the node's /etc/rancher/k3s/registries.yaml credentials are valid |
Pending + FailedScheduling |
Server out of allocatable CPU/memory | kubectl describe node; trim requests or scale down dev/CI (Scaling) |
OOMKilled |
Memory limit hit | Workloads dashboard for memory vs limit, Tomoda API for a traffic spike. Spike: raise resources.limits.memory in the overlay. No spike: likely a leak; capture kubectl exec <pod> -- curl -s localhost:8080/debug/pprof/heap > heap.out for the backend team |
Escalation: prod down > 10 minutes is a full outage. @here in #monitoring.
SLO impact: burns api-availability fast; expect an ErrorBudgetBurn alert after ~10 minutes. Treat both as one incident.
BackendHighErrorRate (warning)¶
Trigger: prod 5xx rate > 5% over 5 minutes.
Grafana Tomoda API dashboard → error routes, then kubectl logs -n tomoda-prod -l app=tomoda-api --tail=200 | grep -E "ERROR|panic". Check Sentry for new issues in environment:prod.
| Cause | Recipe |
|---|---|
| Recent deploy introduced a bug | Correlate new Sentry issues with deploy time; roll back if confirmed |
| Downstream dependency failing (Photon, Stripe, OAuth provider) | Check the matching synthetic probe and the provider's status page |
| DB connection pool exhausted | See PostgresHighConnections |
| One route broke | The per-route breakdown narrows it; targeted hotfix |
BackendHighLatency (warning)¶
Trigger: prod p95 > 2s over 5 minutes.
Grafana Tomoda API → latency by route, then jump into a slow trace in Tempo.
| Cause | Recipe |
|---|---|
| Slow query (missing index) | Long DB span in the trace → EXPLAIN ANALYZE in psql → index migration |
| Connection pool saturation | Spans wait on connection acquire. See PostgresHighConnections |
| Slow external service | Long outbound span to that service; check its status page |
Database and Redis¶
Postgres, Redis and their containers: Data alert runbook.
Node and platform¶
Pod crash loops, node memory, disk and RAID come from the kube-prometheus-stack default rules (KubePodCrashLooping, NodeMemoryHighUtilization, NodeFilesystemSpaceFillingUp, NodeFilesystemAlmostOutOfSpace, NodeRAIDDegraded, NodeRAIDDiskFailure), each with an upstream runbook_url. Server-specific notes:
- Disk space (
NodeFilesystem*): every volume is a directory on the root filesystem, so this is the only disk signal.sudo du -sh /var/lib/rancher/k3s/storage/*shows which volume grew. Budget: disk budget. - RAID (
NodeRAIDDegraded,NodeRAIDDiskFailure): one NVMe left the mirror. Checkcat /proc/mdstat(task server:status), then open an OVH support ticket. Don't reboot until the disk is replaced. - Memory (
NodeMemoryHighUtilization):kubectl top pods -A --sort-by=memory; scale down dev or CI runners first, plan a second server if it's real traffic (Node Lifecycle).
NodeRootFilesystemLow (warning)¶
Trigger: under 15% of / free for 30 minutes, earlier than the default NodeFilesystemAlmostOutOfSpace (5%). Same recipe as Disk space above.
IngressDown (critical)¶
Trigger: Traefik in traefik-system has 0 available replicas for 2 minutes. Every public hostname is down.
kubectl get pods -n traefik-system
kubectl describe pod -n traefik-system -l app.kubernetes.io/name=traefik
kubectl get svc -n traefik-system traefik # EXTERNAL-IP must be the node IP (ServiceLB)
Usual causes: a bad Traefik values change (revert it), or the svclb-traefik-* pod in kube-system not running.
ExternalSecretSyncFailed (warning)¶
Trigger: an ExternalSecret Ready=False for 30 minutes.
kubectl describe externalsecret -n <ns> <name> # the event names the missing key
task secrets:empty # Secret Manager entries with no value
Usual causes: the secret has no enabled version, or the eso-reader key in external-secrets/gcpsm-credentials was revoked (Secrets management).
ArgoCDAppDegraded (warning)¶
Trigger: an Argo CD app Degraded or Missing for 15 minutes.
task argo:health
kubectl get application -n argocd <name> -o jsonpath='{.status.resources}' | jq '.[] | select(.health.status!="Healthy")'
ArgoCDSyncFailed (critical)¶
Trigger: the last sync of tomoda-dev or tomoda-prod failed, almost always the post-deploy smoke test. Read kubectl logs job/smoke-test -n tomoda-prod, then roll back per Rollback.
PhotonDown / EmbeddingsDown (warning)¶
Photon tolerates 2h of cold start (index download and unpack); Embeddings fires after 5 minutes. Check kubectl logs -n platform deploy/photon (or deploy/embeddings) and the index in R2 (Photon indexing).
BlackboxProbeFailed (warning)¶
A /ready probe for Loki, Tempo or Photon failed for 2 minutes: the pod isn't serving even if Argo reports it Healthy. The service label names which one; check its logs and restart it.
Realtime¶
WSHubConnectionDrop (warning)¶
Trigger: the WebSocket hub drops more than 1 message/sec for 5 minutes (tomoda_ws_dropped_total): slow or vanished clients.
| Cause | Recipe |
|---|---|
| A deploy rolled the api pods | Self-resolves in 1-2 minutes as clients reconnect |
NATS backplane slow (NATSSlowConsumers) |
See NATS |
| One api pod overloaded | Compare tomoda_ws_connections_active per pod; check memory and HPAPinnedAtMax |
NATSDown, NATSSlowConsumers, NATSHighMemory, RealtimeBroadcastLatencyHigh: see NATS.
Scaling, semantic and cost alerts¶
PostgresCPUHigh,PostgresReplicaLagNearBudget,PostgresConnectionsNearMax,HPAPinnedAtMax,PodMemoryNearLimit: prod early warnings, each mapped to a step in Scaling.SemanticHighErrorRate,SemanticAuthFailing,SemanticRateLimited,SemanticP95Slow,SemanticDailySpendHigh,SemanticCacheHitLow,SearchEnrichmentHighErrorRate,SearchEnrichmentAuthFailing,SearchEnrichmentRateLimited: the LLM resolver and search enrichment in the async pods. Auth failures mean the API key in Secret Manager is wrong or revoked; rate limits and errors fall back to the deterministic path, so they degrade quality, not availability.DeepSeekBalanceLow,DeepSeekBalanceCheckStale,SerperCreditsLow: prepaid balances. Top up per Cost monitoring → Top-up workflow. Cloudflare billing-usage alerts also land in#monitoring(Cost monitoring → Cloudflare).
Certificates¶
CertificateExpiringSoon (warning)¶
Trigger: a certificate expires in < 14 days.
kubectl get certificate -A
kubectl describe certificate <name> -n <ns>
kubectl logs -n cert-manager deploy/cert-manager --tail=200 | grep <cert-name>
cert-manager renews at 30 days remaining, so this means renewal is stuck. Common: the ACME challenge failing (DNS record not created, Cloudflare API rate limit) or an issuer config change.
CertificateRenewalFailed (critical)¶
Trigger: a certificate Ready=False for > 1 hour. Same checks as above. If the issuer is rate-limited (https://crt.sh/?Identity=<domain>), switch to the staging issuer temporarily.
Escalation: expiry in < 24h: @here.
Synthetic monitor¶
Messages sent directly by the Cloudflare Worker, not through Alertmanager (Synthetics). DOWN posts after two consecutive failed runs; a probe down for > 1h re-posts STILL DOWN hourly; the next passing run (within 1 minute) posts RECOVERED.
api / api-dev DOWN¶
/health returned non-200 or timed out from Cloudflare's edge.
curl -v https://api.tomoda.life/health
# 5xx → backend errored: see BackendDown
# cert error → CertificateRenewalFailed
# NXDOMAIN → kubectl logs -n external-dns deploy/external-dns --tail=50
# refused/timeout → kubectl logs -n traefik-system deploy/traefik --tail=100 | grep ERROR
# In-cluster check (rules out Cloudflare):
kubectl run -it --rm -n tomoda-prod netshoot --image=nicolaka/netshoot --restart=Never -- \
curl -v http://tomoda-api/health
# 200 in-cluster but failing outside → ingress, DNS or cert layer, not the backend
app / app-dev DOWN¶
kubectl get pods -n tomoda-prod -l app=frontend
kubectl logs -n tomoda-prod -l app=frontend --tail=200
curl -vk https://tomoda.life 2>&1 | grep -E "HTTP/|expire|verify|location"
Usually a frontend pod crash, or a build with broken asset URLs (check the src= paths in the HTML). A 302 from the apex to app.tomoda.life is a redirect loop: app. 301s back to the apex, so the OG Worker (og-preview/worker.js) must pass prod human traffic through on the apex.
assets / assets-dev DOWN¶
curl -sI https://assets.tomoda.life/email-logo-black.png # public object, expect 200
curl -sI "<signed url from the app>" # 403: bad signature, 404: object missing
Most likely: tomoda-asset-signing-key-{env} rotated in Secret Manager without a cloudflare Terraform apply (the Worker has the old key) or without a backend restart; or the Worker's custom domain or deploy failed. Re-run the cloudflare apply (gh workflow run terraform-apply.yml), then kubectl rollout restart deployment/tomoda-api -n tomoda-prod.
argocd DOWN¶
Losing Argo CD halts GitOps but running workloads keep serving; less urgent than a backend outage.
kubectl get pods -n argocd
kubectl logs -n argocd deploy/argocd-server --tail=100
kubectl logs -n argocd deploy/argocd-repo-server --tail=100
Usual causes: repo-server OOMKilled on a large chart, or Dex misconfigured after an OAuth client rotation. kubectl rollout restart the offending Deployment; for a recurring OOM raise its memory in k8s/envs/platform/argocd/values.yaml.
docs DOWN¶
https://docs.tomoda.life should return 302 to Cloudflare Access. Check the tomoda-docs Pages project's latest deployment in the Cloudflare dashboard. A failed build is usually mkdocs build --strict failing on a broken link from a recent docs PR (revert or fix; Pages rebuilds), or the submodule clone failing. The build command is in infrastructure/cloudflare/pages.tf.
login DOWN¶
A real-credentials login returned non-200: the auth path is broken.
PASSWORD=$(gcloud secrets versions access latest --secret=tomoda-synthetic-probe-password)
curl -v -X POST https://api.tomoda.life/api/v1/auth/login \
-H "Content-Type: application/json" \
-d "{\"email\":\"synthetic+probe@tomoda.life\",\"password\":\"$PASSWORD\"}"
# 200 → transient; wait for the next run
# 401 → password mismatch or the synthetic user is missing
# 429 → the probe user is locked out (5 failed logins in 15 minutes)
# 5xx → backend or DB problem: BackendHighErrorRate
| Cause | Fix |
|---|---|
| Password rotated in Secret Manager, backend not restarted | kubectl rollout restart deploy/tomoda-api -n tomoda-prod; the boot seed updates the user's hash |
| Synthetic user deleted | Same restart recreates it; if not, check the seed errors in the backend logs |
| Worker holds a stale password | Re-run the cloudflare Terraform apply |
| 429: probe user locked out | Locked attempts count as failures, so the login probe keeps the lock alive. Delete the user's failed rows from the last 15 minutes: DELETE FROM login_histories WHERE success = false AND created_at > now() - interval '15 minutes' AND user_id = (SELECT id FROM users WHERE email = 'synthetic+probe@tomoda.life'); via task pg:console -- prod. Then find what is failing: a stale password, or login-bad-password running more often than every 5 minutes |
login-bad-password DOWN (security)¶
A wrong password returned something other than 401. A 200 means any password is accepted: treat as a security incident.
curl -v -X POST https://api.tomoda.life/api/v1/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"synthetic+probe@tomoda.life","password":"definitely-wrong-password-12345"}'
# 401 → false alarm 5xx → backend bug, urgent but not a bypass 200 → confirmed
# 429 → probe user locked out: see `login` DOWN
Each manual wrong-password attempt counts toward the probe user's lockout, so try it once, not in a loop.
If confirmed:
- Roll back prod to the previous tag (Rollback).
- Disable the probe user while debugging:
UPDATE users SET is_active=false WHERE email='synthetic+probe@tomoda.life';viatask pg:console -- prod. - Diff the auth handler between the bad and last good release (password check short-circuited, a dev-only flag enabled in prod, middleware order).
- Notify all engineers.
auth-middleware DOWN (security)¶
An unauthenticated GET /api/v1/auth/profile returned something other than 401. A 200 means protected data is exposed.
curl -v https://api.tomoda.life/api/v1/auth/profile
# 401 → false alarm 404 → route moved: update synthetic/worker.js 200 → confirmed
If confirmed, test a few other protected routes to scope it, roll back as for login-bad-password, and notify all engineers. Common causes: middleware order changed, a new route missing the auth middleware, an overly broad skip-auth pattern.
SLO burn-rate alerts¶
Pyrra emits ErrorBudgetBurn alerts per SLO (api-availability, api-latency-p95, login-success, async-task-success, ws-broadcast-latency-p99) on the multi-window scheme in SLOs.
- Find the indicator on the SLO dashboard (which route or task type is burning).
- If a rule-based alert for the same area is also firing, it's one incident; follow that playbook.
- Acute (1h or 6h window): fix the root cause and the burn alert clears. Chronic (3d window only): no single event; schedule reliability work for that area.
Escalation¶
@here in #monitoring (the webhook never mentions anyone) when:
- BackendDown, PostgresDown or RedisDown in prod lasts > 10 minutes
- CertificateRenewalFailed with expiry < 24h
- Several critical alerts fire at once (likely server-wide)
- User data is at risk (backups stale while writes continue), or a synthetic security probe confirms
Adding an alert¶
Add a section in the same PR as the rule, here or, for Postgres and Redis, in the Data alert runbook, and set the rule's runbook_url to it:
### AlertName (severity)
**Trigger:** the expression in plain English.
First checks (commands), then a cause → recipe table, then escalation.
Related¶
- Alerting: routing, inhibition and throttling for every Prometheus alert
- SLOs: the SLO catalogue and burn-rate generation
- Synthetics: the Worker behind DOWN/RECOVERED messages
- Disaster recovery: catastrophic database failure