Skip to content

Alerting

Tomodachi, the Discord webhook avatar that posts alerts to #monitoring

Prometheus Alertmanager → Discord #monitoring channel. One unified incident pipe for backend / database / cluster / cert / business-metric alerts. Messages are posted by the Tomodachi webhook. Sentry's frontend alerts route separately to #alerts-frontend through Sentry's OAuth Discord integration (Sentry data source).

Pipeline

Prometheus ──► fires rule (PrometheusRule CRD)
   │
   ▼
Alertmanager ──► group_by [alertname, env, cluster, service]
   │            ──► group_wait 30s, group_interval 5m
   │            ──► inhibition rules (e.g. PostgresDown silences DB warnings)
   ▼
Discord webhook (Cloudflare-fronted) ──► #monitoring channel

Webhook URL is sourced from GCP SM tomoda-alert-webhook-primary via ESO → K8s Secret alertmanager-discord-credentials in monitoring/. The Discord receiver's apiURL reads the webhook-url key of that secret directly, so the URL is never inlined in git.

AlertmanagerConfig CRD, not raw config

Routing + the Discord receiver live in an AlertmanagerConfig CR (k8s/envs/platform/manifests/alertmanager-config.yaml), not inline in the chart values. The prometheus-operator's raw-config validator rejects discord webhook_url_file (#7159); the CRD's discordConfigs.apiURL reads the webhook from a secret key instead.

Routing + throttling defaults

These live in the AlertmanagerConfig CR (k8s/envs/platform/manifests/alertmanager-config.yaml) under spec.route:

Setting Default Effect
group_by [alertname, env, cluster, service] Related alerts batched into one Discord message; prod and dev never share one
group_wait 30s Wait 30s after the first alert so correlated alerts land in the first batch
group_interval 5m Within a group, batched update at most every 5min
repeat_interval 4h (warning), 1h (critical) Don't re-fire the same alert for N hours
severity = critical route override group_wait: 10s, repeat_interval: 1h Critical paging incidents stay loud during long outages
alertname = Watchdog route repeat_interval: 24h Daily heartbeat message proving the pipeline delivers
InfoInhibitor, severity = info null receiver Inhibition plumbing, never posted

Inhibition rules

When a root-cause alert fires, downstream-symptom warnings are suppressed:

Source (firing) Suppresses Matched on
PostgresDown All severity=warning alerts for the same env env label
BackendDown (critical) BackendHighErrorRate, BackendHighLatency for the same env env label
RedisDown Redis* warnings for the same env env label

Match on a label both sides carry. Alertmanager treats an equal label missing on both alerts as equal, so a label only the source has (such as cluster) would suppress unrelated alerts in every env. BackendDown comes from kube-state-metrics, whose service label every default Kube* alert shares, so its rule matches by alert name and env. The Discord message shows env next to the severity.

Result: a 6-hour incident with 50 correlated warnings fires ~12-15 Discord messages instead of thousands.

Alert rule inventory

Rules live in PrometheusRule resources in monitoring/: tomoda-alerts (k8s/envs/platform/manifests/alerting-rules.yaml), tomoda-postgres-alerts (postgres-alerts.yaml), tomoda-redis-alerts (redis-alerts.yaml) and tomoda-location-harvest-alerts (location-harvest-alerts.yaml). Postgres and Redis alerts link their section of the Data alert runbook through runbook_url; harvest alerts link the app's capture-picker harvesting page. scripts/check-alert-rules.sh runs promtool check rules on every file and the unit tests in k8s/envs/platform/alert-tests/; CI runs it on every k8s PR.

Group Rule Severity Fires when
application ArgoCDSyncFailed critical Argo CD sync of tomoda-dev or tomoda-prod failed (usually the smoke test)
BackendDown critical (prod) / warning (dev) Backend is down
BackendHighErrorRate warning Backend error rate above 5%
BackendHighLatency warning Backend p95 latency above 2s
database PostgresDown critical (prod) / warning (dev) CNPG collector reports Postgres down, or the instance isn't scraped, for 1m
PostgresNoPrimary critical (prod) / warning (dev) Every instance is in recovery for 5m
PostgresReplicationLag critical (prod) / warning (dev) PostgreSQL replication lag above 10s
PostgresHighConnections critical (prod) / warning (dev) PostgreSQL connections above 80% of max
PostgresBackupStale critical (prod) / warning (dev) Last WAL archive over 1h ago
PostgresWALArchivingFailing critical (prod) Latest WAL archive attempt failed, none succeeded since, for 10m
PostgresBaseBackupFailed critical (prod) PostgreSQL base backup failed
PostgresBaseBackupStale critical (prod) No PostgreSQL base backup in 26h
OTABackupStale critical (prod) / warning (dev) No successful Xavia OTA DB backup in 26h
PostgresLongRunningTransaction warning (prod) A transaction open for over 10m
PostgresLockWaits warning (prod) More than 5 backends waiting on locks for 5m
PostgresDeadlocks warning (prod) Any deadlock in 15m
PostgresXIDAgeHigh warning (prod, dev) Transaction ID age above 500M for 30m
PostgresXIDWraparoundImminent critical (prod) Transaction ID age above 1.2B
PostgresDiskUsageHigh warning (prod, dev) Databases plus WAL above 80% of the PVC request for 30m
PostgresDiskUsageCritical critical (prod) Databases plus WAL above 90% of the PVC request for 15m
PostgresDiskFillingUp warning (prod, dev) 6h trend fills the PVC request within 24h
PostgresCacheHitRatioLow warning (prod) Buffer cache hit ratio under 90% under real read load for 30m
redis RedisDown critical (prod) / warning (dev) Redis is down or its exporter isn't scraped
RedisHighMemory critical (prod) / warning (dev) Memory above 90% of maxmemory
RedisMemoryNearMax warning (prod) Memory above 80% of maxmemory for 15m
RedisMaxmemoryUnset warning (prod, dev) maxmemory is 0, so the memory alerts can't fire
RedisRejectingWrites critical (prod) / warning (dev) Writes failing with OOM at maxmemory
RedisClientsNearMax warning (prod) Clients above 80% of maxclients
RedisRejectedConnections critical (prod) / warning (dev) Connections rejected at maxclients
RedisPersistenceFailing critical (prod) / warning (dev) Last RDB save, AOF write or AOF rewrite failed
RedisDiskUsageHigh warning (prod, dev) AOF above 80% of the PVC request
data.containers DataStoreMemoryNearLimit warning (prod, dev) Postgres or Redis container above 90% of its memory limit
DataStoreRestarted warning (prod, dev) Postgres or Redis restarted in the last 15m
data.node NodeRootFilesystemLow warning Under 15% of / free for 30m
scaling PostgresCPUHigh warning Postgres primary above 4 CPU cores
PostgresReplicaLagNearBudget warning Postgres replica lag above 5s
PostgresConnectionsNearMax warning Postgres connections above 70% of max
HPAPinnedAtMax warning A prod HPA sits at maxReplicas for 15m
PodMemoryNearLimit warning A prod container is above 85% of its memory limit for 15m
location-harvest HarvestPanic critical Any recovered panic in harvest goroutines in 10m
HarvestSpendLogFailing critical Any paid provider call in 10m with no spend row written
HarvestHandoffArchived critical Any location:burst_seed task archived after exhausting retries in 10m
HarvestProviderCreditsEstimated warning Provider responses omit credits (spend charged at the estimate)
HarvestBreakerOpen critical Provider circuit breaker open for 5m
HarvestBudgetNearCeiling warning A trigger's daily budget is above 80% (below 100%)
HarvestBudgetExhausted critical A trigger's daily budget is at or above 100%
HarvestOutboxBacklog warning Stored paid answers pending commit for 15m
business WSHubConnectionDrop warning WebSocket hub dropping > 1 message/sec
semantic SemanticHighErrorRate warning LLM error rate > 10% for 10m
SemanticAuthFailing critical DeepSeek auth failures (401/403)
SemanticRateLimited warning DeepSeek 429 rate-limit sustained for 15m
SemanticP95Slow warning LLM p95 latency > 10s for 10m
SemanticDailySpendHigh warning LLM 24h spend > $25 (off-peak-rate estimate)
SemanticCacheHitLow warning Prompt-cache hit ratio < 30% for 30m
SearchEnrichmentHighErrorRate warning Search enrichment error rate > 10% for 10m
SearchEnrichmentAuthFailing critical Search-enrichment auth failures (401/403)
SearchEnrichmentRateLimited warning Search-enrichment 429 rate-limit sustained for 15m
certificates CertificateExpiringSoon warning A certificate expires in under 14 days
CertificateRenewalFailed critical A certificate is not Ready for 1h
platform EmbeddingsDown warning Embeddings service is down
PhotonDown warning Photon geocoder is down
IngressDown critical Traefik has no available replicas
ExternalSecretSyncFailed warning An ExternalSecret has failed to sync for 30m
ArgoCDAppDegraded warning An Argo CD app is Degraded or Missing for 15m
observability BlackboxProbeFailed warning A blackbox /ready probe (Loki, Tempo, Photon) fails for 2m
messaging NATSDown critical NATS backplane is down
NATSSlowConsumers warning NATS slow consumers detected
NATSHighMemory warning NATS memory above 800MB
RealtimeBroadcastLatencyHigh warning WebSocket broadcast p99 latency above 1s

Prod rules select env="prod": the ServiceMonitor (api) and PodMonitor (async) copy each overlay's env label onto every scraped series.

The kube-prometheus-stack default rules cover the platform and the node: pod crash loops (KubePodCrashLooping), pods not ready (KubePodNotReady), node memory (NodeMemoryHighUtilization, KubeNodePressure), disk I/O (NodeDiskIOSaturation), root filesystem space (NodeFilesystemSpaceFillingUp, NodeFilesystemAlmostOutOfSpace), software RAID (NodeRAIDDegraded, NodeRAIDDiskFailure), clock sync, kubelet, API server and Prometheus itself. KubeCPUOvercommit and KubeMemoryOvercommit are disabled: on one node they compare requests against zero spare capacity and would fire permanently. k3s local-path volumes report no per-volume usage (KubePersistentVolumeFillingUp never has data), so disk space is watched at the filesystem level, and the Postgres and Redis disk alerts compare data size against the PVC request. No data pod has a CPU limit, so CPUThrottlingHigh stays quiet for them.

Cost alerts (DeepSeekBalanceLow, DeepSeekBalanceCheckStale, SerperCreditsLow) live in a separate cost-alerts PrometheusRule and route through this same pipe. See Cost monitoring.

Verification

# 1. ESO has projected the webhook secret
kubectl get secret alertmanager-discord-credentials -n monitoring
#   NAME                                  TYPE     DATA   AGE
#   alertmanager-discord-credentials      Opaque   1      ...

# 2. Alertmanager config loaded
kubectl exec -n monitoring statefulset/alertmanager-monitoring -- \
  amtool config show | head
#   route: ...
#   receivers: ...

# 3. Send a test alert
amtool alert add alertname=AlertmanagerTestPing severity=warning \
  --alertmanager.url=http://localhost:9093 \
  --annotations=summary="Test ping from amtool"
# (port-forward first: kubectl port-forward -n monitoring svc/alertmanager-operated 9093)

A real Discord message should appear in #monitoring within ~30s of the test ping.

Rotation

tomoda-alert-webhook-primary rotates independently:

# 1. Generate a new Discord webhook on the same #monitoring channel
#    (Discord Channel Settings → Integrations → Webhooks → New Webhook)
# 2. Copy the webhook URL, then push it to GCP SM as a new version
pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-alert-webhook-primary \
  --project=development-485000 --data-file=-
pbcopy </dev/null
# 3. ESO syncs within 1h; force-restart Alertmanager to verify immediately
kubectl rollout restart statefulset/alertmanager-monitoring -n monitoring
# 4. Delete the old webhook in Discord (otherwise both fire)

Provider swap

The secret name is provider-neutral. To move to another chat or paging provider, put its webhook URL in tomoda-alert-webhook-primary and swap the discordConfigs receiver in k8s/envs/platform/manifests/alertmanager-config.yaml for the matching receiver type (and its message template). No Secret renames.