Skip to content

Alert response runbook

What to do when an alert fires in #monitoring. Search this page for the alert name in the Discord message. For routine commands (connect, logs, psql, redis-cli) see the Daily ops runbook.

Alert sources:

  • Prometheus rules in k8s/envs/platform/manifests/alerting-rules.yaml and cost-alerts.yaml (Alerting); Postgres and Redis rules in postgres-alerts.yaml and redis-alerts.yaml have their own Data alert runbook; location-harvest-alerts.yaml links the app's harvesting page
  • Pyrra burn-rate alerts generated from ServiceLevelObjective CRs (SLOs); one generic playbook at the bottom
  • Synthetic Worker probes: Discord messages sent directly by the Cloudflare Worker (Synthetics)

Most rules come in pairs: critical for prod, warning for dev, same threshold. Headings note where that applies.


Application

BackendDown (critical prod / warning dev)

Trigger: tomoda-api in tomoda-prod (or tomoda-dev) has 0 available replicas for 2 minutes. Suppresses BackendHighErrorRate and BackendHighLatency for the same env.

Debug path:

kubectl get pods -n tomoda-prod -l app=tomoda-api -o wide
kubectl describe pod -n tomoda-prod <pod-name> | tail -40        # Events: most recent failure reason
kubectl logs -n tomoda-prod -l app=tomoda-api --tail=200 --since=10m
kubectl logs -n tomoda-prod <pod-name> --previous --tail=200     # before the last restart
kubectl get pod -n tomoda-prod <pod-name> \
  -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'   # OOMKilled / Error
What you see Meaning Recipe
CrashLoopBackOff + panic: in logs Crash on startup Roll back (Rollback); open a tomoda issue with the panic
Readiness failing, logs show Postgres or Redis errors The readiness probe checks Postgres and Redis, so a data-tier outage takes every pod out of service Run PostgresDown or RedisDown instead
ImagePullBackOff Wrong tag or GHCR credentials Check the tag in k8s/apps/tomoda/overlays/prod/kustomization.yaml exists (task images:list -- tomoda-backend) and the node's /etc/rancher/k3s/registries.yaml credentials are valid
Pending + FailedScheduling Server out of allocatable CPU/memory kubectl describe node; trim requests or scale down dev/CI (Scaling)
OOMKilled Memory limit hit Workloads dashboard for memory vs limit, Tomoda API for a traffic spike. Spike: raise resources.limits.memory in the overlay. No spike: likely a leak; capture kubectl exec <pod> -- curl -s localhost:8080/debug/pprof/heap > heap.out for the backend team

Escalation: prod down > 10 minutes is a full outage. @here in #monitoring.

SLO impact: burns api-availability fast; expect an ErrorBudgetBurn alert after ~10 minutes. Treat both as one incident.

BackendHighErrorRate (warning)

Trigger: prod 5xx rate > 5% over 5 minutes.

Grafana Tomoda API dashboard → error routes, then kubectl logs -n tomoda-prod -l app=tomoda-api --tail=200 | grep -E "ERROR|panic". Check Sentry for new issues in environment:prod.

Cause Recipe
Recent deploy introduced a bug Correlate new Sentry issues with deploy time; roll back if confirmed
Downstream dependency failing (Photon, Stripe, OAuth provider) Check the matching synthetic probe and the provider's status page
DB connection pool exhausted See PostgresHighConnections
One route broke The per-route breakdown narrows it; targeted hotfix

BackendHighLatency (warning)

Trigger: prod p95 > 2s over 5 minutes.

Grafana Tomoda API → latency by route, then jump into a slow trace in Tempo.

Cause Recipe
Slow query (missing index) Long DB span in the trace → EXPLAIN ANALYZE in psql → index migration
Connection pool saturation Spans wait on connection acquire. See PostgresHighConnections
Slow external service Long outbound span to that service; check its status page

Database and Redis

Postgres, Redis and their containers: Data alert runbook.


Node and platform

Pod crash loops, node memory, disk and RAID come from the kube-prometheus-stack default rules (KubePodCrashLooping, NodeMemoryHighUtilization, NodeFilesystemSpaceFillingUp, NodeFilesystemAlmostOutOfSpace, NodeRAIDDegraded, NodeRAIDDiskFailure), each with an upstream runbook_url. Server-specific notes:

  • Disk space (NodeFilesystem*): every volume is a directory on the root filesystem, so this is the only disk signal. sudo du -sh /var/lib/rancher/k3s/storage/* shows which volume grew. Budget: disk budget.
  • RAID (NodeRAIDDegraded, NodeRAIDDiskFailure): one NVMe left the mirror. Check cat /proc/mdstat (task server:status), then open an OVH support ticket. Don't reboot until the disk is replaced.
  • Memory (NodeMemoryHighUtilization): kubectl top pods -A --sort-by=memory; scale down dev or CI runners first, plan a second server if it's real traffic (Node Lifecycle).

NodeRootFilesystemLow (warning)

Trigger: under 15% of / free for 30 minutes, earlier than the default NodeFilesystemAlmostOutOfSpace (5%). Same recipe as Disk space above.

IngressDown (critical)

Trigger: Traefik in traefik-system has 0 available replicas for 2 minutes. Every public hostname is down.

kubectl get pods -n traefik-system
kubectl describe pod -n traefik-system -l app.kubernetes.io/name=traefik
kubectl get svc -n traefik-system traefik    # EXTERNAL-IP must be the node IP (ServiceLB)

Usual causes: a bad Traefik values change (revert it), or the svclb-traefik-* pod in kube-system not running.

ExternalSecretSyncFailed (warning)

Trigger: an ExternalSecret Ready=False for 30 minutes.

kubectl describe externalsecret -n <ns> <name>     # the event names the missing key
task secrets:empty                                  # Secret Manager entries with no value

Usual causes: the secret has no enabled version, or the eso-reader key in external-secrets/gcpsm-credentials was revoked (Secrets management).

ArgoCDAppDegraded (warning)

Trigger: an Argo CD app Degraded or Missing for 15 minutes.

task argo:health
kubectl get application -n argocd <name> -o jsonpath='{.status.resources}' | jq '.[] | select(.health.status!="Healthy")'

ArgoCDSyncFailed (critical)

Trigger: the last sync of tomoda-dev or tomoda-prod failed, almost always the post-deploy smoke test. Read kubectl logs job/smoke-test -n tomoda-prod, then roll back per Rollback.

PhotonDown / EmbeddingsDown (warning)

Photon tolerates 2h of cold start (index download and unpack); Embeddings fires after 5 minutes. Check kubectl logs -n platform deploy/photon (or deploy/embeddings) and the index in R2 (Photon indexing).

BlackboxProbeFailed (warning)

A /ready probe for Loki, Tempo or Photon failed for 2 minutes: the pod isn't serving even if Argo reports it Healthy. The service label names which one; check its logs and restart it.


Realtime

WSHubConnectionDrop (warning)

Trigger: the WebSocket hub drops more than 1 message/sec for 5 minutes (tomoda_ws_dropped_total): slow or vanished clients.

Cause Recipe
A deploy rolled the api pods Self-resolves in 1-2 minutes as clients reconnect
NATS backplane slow (NATSSlowConsumers) See NATS
One api pod overloaded Compare tomoda_ws_connections_active per pod; check memory and HPAPinnedAtMax

NATSDown, NATSSlowConsumers, NATSHighMemory, RealtimeBroadcastLatencyHigh: see NATS.

Scaling, semantic and cost alerts

  • PostgresCPUHigh, PostgresReplicaLagNearBudget, PostgresConnectionsNearMax, HPAPinnedAtMax, PodMemoryNearLimit: prod early warnings, each mapped to a step in Scaling.
  • SemanticHighErrorRate, SemanticAuthFailing, SemanticRateLimited, SemanticP95Slow, SemanticDailySpendHigh, SemanticCacheHitLow, SearchEnrichmentHighErrorRate, SearchEnrichmentAuthFailing, SearchEnrichmentRateLimited: the LLM resolver and search enrichment in the async pods. Auth failures mean the API key in Secret Manager is wrong or revoked; rate limits and errors fall back to the deterministic path, so they degrade quality, not availability.
  • DeepSeekBalanceLow, DeepSeekBalanceCheckStale, SerperCreditsLow: prepaid balances. Top up per Cost monitoring → Top-up workflow. Cloudflare billing-usage alerts also land in #monitoring (Cost monitoring → Cloudflare).

Certificates

CertificateExpiringSoon (warning)

Trigger: a certificate expires in < 14 days.

kubectl get certificate -A
kubectl describe certificate <name> -n <ns>
kubectl logs -n cert-manager deploy/cert-manager --tail=200 | grep <cert-name>

cert-manager renews at 30 days remaining, so this means renewal is stuck. Common: the ACME challenge failing (DNS record not created, Cloudflare API rate limit) or an issuer config change.

CertificateRenewalFailed (critical)

Trigger: a certificate Ready=False for > 1 hour. Same checks as above. If the issuer is rate-limited (https://crt.sh/?Identity=<domain>), switch to the staging issuer temporarily.

Escalation: expiry in < 24h: @here.


Synthetic monitor

Messages sent directly by the Cloudflare Worker, not through Alertmanager (Synthetics). DOWN posts after two consecutive failed runs; a probe down for > 1h re-posts STILL DOWN hourly; the next passing run (within 1 minute) posts RECOVERED.

api / api-dev DOWN

/health returned non-200 or timed out from Cloudflare's edge.

curl -v https://api.tomoda.life/health
#   5xx              → backend errored: see BackendDown
#   cert error       → CertificateRenewalFailed
#   NXDOMAIN         → kubectl logs -n external-dns deploy/external-dns --tail=50
#   refused/timeout  → kubectl logs -n traefik-system deploy/traefik --tail=100 | grep ERROR
# In-cluster check (rules out Cloudflare):
kubectl run -it --rm -n tomoda-prod netshoot --image=nicolaka/netshoot --restart=Never -- \
  curl -v http://tomoda-api/health
# 200 in-cluster but failing outside → ingress, DNS or cert layer, not the backend

app / app-dev DOWN

kubectl get pods -n tomoda-prod -l app=frontend
kubectl logs -n tomoda-prod -l app=frontend --tail=200
curl -vk https://tomoda.life 2>&1 | grep -E "HTTP/|expire|verify|location"

Usually a frontend pod crash, or a build with broken asset URLs (check the src= paths in the HTML). A 302 from the apex to app.tomoda.life is a redirect loop: app. 301s back to the apex, so the OG Worker (og-preview/worker.js) must pass prod human traffic through on the apex.

assets / assets-dev DOWN

curl -sI https://assets.tomoda.life/email-logo-black.png    # public object, expect 200
curl -sI "<signed url from the app>"                         # 403: bad signature, 404: object missing

Most likely: tomoda-asset-signing-key-{env} rotated in Secret Manager without a cloudflare Terraform apply (the Worker has the old key) or without a backend restart; or the Worker's custom domain or deploy failed. Re-run the cloudflare apply (gh workflow run terraform-apply.yml), then kubectl rollout restart deployment/tomoda-api -n tomoda-prod.

argocd DOWN

Losing Argo CD halts GitOps but running workloads keep serving; less urgent than a backend outage.

kubectl get pods -n argocd
kubectl logs -n argocd deploy/argocd-server --tail=100
kubectl logs -n argocd deploy/argocd-repo-server --tail=100

Usual causes: repo-server OOMKilled on a large chart, or Dex misconfigured after an OAuth client rotation. kubectl rollout restart the offending Deployment; for a recurring OOM raise its memory in k8s/envs/platform/argocd/values.yaml.

docs DOWN

https://docs.tomoda.life should return 302 to Cloudflare Access. Check the tomoda-docs Pages project's latest deployment in the Cloudflare dashboard. A failed build is usually mkdocs build --strict failing on a broken link from a recent docs PR (revert or fix; Pages rebuilds), or the submodule clone failing. The build command is in infrastructure/cloudflare/pages.tf.

login DOWN

A real-credentials login returned non-200: the auth path is broken.

PASSWORD=$(gcloud secrets versions access latest --secret=tomoda-synthetic-probe-password)
curl -v -X POST https://api.tomoda.life/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d "{\"email\":\"synthetic+probe@tomoda.life\",\"password\":\"$PASSWORD\"}"
#   200 → transient; wait for the next run
#   401 → password mismatch or the synthetic user is missing
#   429 → the probe user is locked out (5 failed logins in 15 minutes)
#   5xx → backend or DB problem: BackendHighErrorRate
Cause Fix
Password rotated in Secret Manager, backend not restarted kubectl rollout restart deploy/tomoda-api -n tomoda-prod; the boot seed updates the user's hash
Synthetic user deleted Same restart recreates it; if not, check the seed errors in the backend logs
Worker holds a stale password Re-run the cloudflare Terraform apply
429: probe user locked out Locked attempts count as failures, so the login probe keeps the lock alive. Delete the user's failed rows from the last 15 minutes: DELETE FROM login_histories WHERE success = false AND created_at > now() - interval '15 minutes' AND user_id = (SELECT id FROM users WHERE email = 'synthetic+probe@tomoda.life'); via task pg:console -- prod. Then find what is failing: a stale password, or login-bad-password running more often than every 5 minutes

login-bad-password DOWN (security)

A wrong password returned something other than 401. A 200 means any password is accepted: treat as a security incident.

curl -v -X POST https://api.tomoda.life/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"synthetic+probe@tomoda.life","password":"definitely-wrong-password-12345"}'
#   401 → false alarm   5xx → backend bug, urgent but not a bypass   200 → confirmed
#   429 → probe user locked out: see `login` DOWN

Each manual wrong-password attempt counts toward the probe user's lockout, so try it once, not in a loop.

If confirmed:

  1. Roll back prod to the previous tag (Rollback).
  2. Disable the probe user while debugging: UPDATE users SET is_active=false WHERE email='synthetic+probe@tomoda.life'; via task pg:console -- prod.
  3. Diff the auth handler between the bad and last good release (password check short-circuited, a dev-only flag enabled in prod, middleware order).
  4. Notify all engineers.

auth-middleware DOWN (security)

An unauthenticated GET /api/v1/auth/profile returned something other than 401. A 200 means protected data is exposed.

curl -v https://api.tomoda.life/api/v1/auth/profile
#   401 → false alarm   404 → route moved: update synthetic/worker.js   200 → confirmed

If confirmed, test a few other protected routes to scope it, roll back as for login-bad-password, and notify all engineers. Common causes: middleware order changed, a new route missing the auth middleware, an overly broad skip-auth pattern.


SLO burn-rate alerts

Pyrra emits ErrorBudgetBurn alerts per SLO (api-availability, api-latency-p95, login-success, async-task-success, ws-broadcast-latency-p99) on the multi-window scheme in SLOs.

  1. Find the indicator on the SLO dashboard (which route or task type is burning).
  2. If a rule-based alert for the same area is also firing, it's one incident; follow that playbook.
  3. Acute (1h or 6h window): fix the root cause and the burn alert clears. Chronic (3d window only): no single event; schedule reliability work for that area.

Escalation

@here in #monitoring (the webhook never mentions anyone) when:

  • BackendDown, PostgresDown or RedisDown in prod lasts > 10 minutes
  • CertificateRenewalFailed with expiry < 24h
  • Several critical alerts fire at once (likely server-wide)
  • User data is at risk (backups stale while writes continue), or a synthetic security probe confirms

Adding an alert

Add a section in the same PR as the rule, here or, for Postgres and Redis, in the Data alert runbook, and set the rule's runbook_url to it:

### AlertName (severity)

**Trigger:** the expression in plain English.

First checks (commands), then a cause → recipe table, then escalation.
  • Alerting: routing, inhibition and throttling for every Prometheus alert
  • SLOs: the SLO catalogue and burn-rate generation
  • Synthetics: the Worker behind DOWN/RECOVERED messages
  • Disaster recovery: catastrophic database failure