Alerting¶
![]()
Prometheus Alertmanager → Discord #monitoring channel. One unified incident pipe for backend / database / cluster / cert / business-metric alerts. Messages are posted by the Tomodachi webhook. Sentry's frontend alerts route separately to #alerts-frontend through Sentry's OAuth Discord integration (Sentry data source).
Pipeline¶
Prometheus ──► fires rule (PrometheusRule CRD)
│
▼
Alertmanager ──► group_by [alertname, env, cluster, service]
│ ──► group_wait 30s, group_interval 5m
│ ──► inhibition rules (e.g. PostgresDown silences DB warnings)
▼
Discord webhook (Cloudflare-fronted) ──► #monitoring channel
Webhook URL is sourced from GCP SM tomoda-alert-webhook-primary via ESO → K8s Secret alertmanager-discord-credentials in monitoring/. The Discord receiver's apiURL reads the webhook-url key of that secret directly, so the URL is never inlined in git.
AlertmanagerConfig CRD, not raw config
Routing + the Discord receiver live in an AlertmanagerConfig CR (k8s/envs/platform/manifests/alertmanager-config.yaml), not inline in the chart values. The prometheus-operator's raw-config validator rejects discord webhook_url_file (#7159); the CRD's discordConfigs.apiURL reads the webhook from a secret key instead.
Routing + throttling defaults¶
These live in the AlertmanagerConfig CR (k8s/envs/platform/manifests/alertmanager-config.yaml) under spec.route:
| Setting | Default | Effect |
|---|---|---|
group_by |
[alertname, env, cluster, service] |
Related alerts batched into one Discord message; prod and dev never share one |
group_wait |
30s |
Wait 30s after the first alert so correlated alerts land in the first batch |
group_interval |
5m |
Within a group, batched update at most every 5min |
repeat_interval |
4h (warning), 1h (critical) |
Don't re-fire the same alert for N hours |
severity = critical route override |
group_wait: 10s, repeat_interval: 1h |
Critical paging incidents stay loud during long outages |
alertname = Watchdog route |
repeat_interval: 24h |
Daily heartbeat message proving the pipeline delivers |
InfoInhibitor, severity = info |
null receiver |
Inhibition plumbing, never posted |
Inhibition rules¶
When a root-cause alert fires, downstream-symptom warnings are suppressed:
| Source (firing) | Suppresses | Matched on |
|---|---|---|
PostgresDown |
All severity=warning alerts for the same env |
env label |
BackendDown (critical) |
BackendHighErrorRate, BackendHighLatency for the same env |
env label |
RedisDown |
Redis* warnings for the same env |
env label |
Match on a label both sides carry. Alertmanager treats an equal label missing on both alerts as equal, so a label only the source has (such as cluster) would suppress unrelated alerts in every env. BackendDown comes from kube-state-metrics, whose service label every default Kube* alert shares, so its rule matches by alert name and env. The Discord message shows env next to the severity.
Result: a 6-hour incident with 50 correlated warnings fires ~12-15 Discord messages instead of thousands.
Alert rule inventory¶
Rules live in PrometheusRule resources in monitoring/: tomoda-alerts (k8s/envs/platform/manifests/alerting-rules.yaml), tomoda-postgres-alerts (postgres-alerts.yaml), tomoda-redis-alerts (redis-alerts.yaml) and tomoda-location-harvest-alerts (location-harvest-alerts.yaml). Postgres and Redis alerts link their section of the Data alert runbook through runbook_url; harvest alerts link the app's capture-picker harvesting page. scripts/check-alert-rules.sh runs promtool check rules on every file and the unit tests in k8s/envs/platform/alert-tests/; CI runs it on every k8s PR.
| Group | Rule | Severity | Fires when |
|---|---|---|---|
| application | ArgoCDSyncFailed |
critical | Argo CD sync of tomoda-dev or tomoda-prod failed (usually the smoke test) |
BackendDown |
critical (prod) / warning (dev) | Backend is down | |
BackendHighErrorRate |
warning | Backend error rate above 5% | |
BackendHighLatency |
warning | Backend p95 latency above 2s | |
| database | PostgresDown |
critical (prod) / warning (dev) | CNPG collector reports Postgres down, or the instance isn't scraped, for 1m |
PostgresNoPrimary |
critical (prod) / warning (dev) | Every instance is in recovery for 5m | |
PostgresReplicationLag |
critical (prod) / warning (dev) | PostgreSQL replication lag above 10s | |
PostgresHighConnections |
critical (prod) / warning (dev) | PostgreSQL connections above 80% of max | |
PostgresBackupStale |
critical (prod) / warning (dev) | Last WAL archive over 1h ago | |
PostgresWALArchivingFailing |
critical (prod) | Latest WAL archive attempt failed, none succeeded since, for 10m | |
PostgresBaseBackupFailed |
critical (prod) | PostgreSQL base backup failed | |
PostgresBaseBackupStale |
critical (prod) | No PostgreSQL base backup in 26h | |
OTABackupStale |
critical (prod) / warning (dev) | No successful Xavia OTA DB backup in 26h | |
PostgresLongRunningTransaction |
warning (prod) | A transaction open for over 10m | |
PostgresLockWaits |
warning (prod) | More than 5 backends waiting on locks for 5m | |
PostgresDeadlocks |
warning (prod) | Any deadlock in 15m | |
PostgresXIDAgeHigh |
warning (prod, dev) | Transaction ID age above 500M for 30m | |
PostgresXIDWraparoundImminent |
critical (prod) | Transaction ID age above 1.2B | |
PostgresDiskUsageHigh |
warning (prod, dev) | Databases plus WAL above 80% of the PVC request for 30m | |
PostgresDiskUsageCritical |
critical (prod) | Databases plus WAL above 90% of the PVC request for 15m | |
PostgresDiskFillingUp |
warning (prod, dev) | 6h trend fills the PVC request within 24h | |
PostgresCacheHitRatioLow |
warning (prod) | Buffer cache hit ratio under 90% under real read load for 30m | |
| redis | RedisDown |
critical (prod) / warning (dev) | Redis is down or its exporter isn't scraped |
RedisHighMemory |
critical (prod) / warning (dev) | Memory above 90% of maxmemory |
|
RedisMemoryNearMax |
warning (prod) | Memory above 80% of maxmemory for 15m |
|
RedisMaxmemoryUnset |
warning (prod, dev) | maxmemory is 0, so the memory alerts can't fire |
|
RedisRejectingWrites |
critical (prod) / warning (dev) | Writes failing with OOM at maxmemory |
|
RedisClientsNearMax |
warning (prod) | Clients above 80% of maxclients |
|
RedisRejectedConnections |
critical (prod) / warning (dev) | Connections rejected at maxclients |
|
RedisPersistenceFailing |
critical (prod) / warning (dev) | Last RDB save, AOF write or AOF rewrite failed | |
RedisDiskUsageHigh |
warning (prod, dev) | AOF above 80% of the PVC request | |
| data.containers | DataStoreMemoryNearLimit |
warning (prod, dev) | Postgres or Redis container above 90% of its memory limit |
DataStoreRestarted |
warning (prod, dev) | Postgres or Redis restarted in the last 15m | |
| data.node | NodeRootFilesystemLow |
warning | Under 15% of / free for 30m |
| scaling | PostgresCPUHigh |
warning | Postgres primary above 4 CPU cores |
PostgresReplicaLagNearBudget |
warning | Postgres replica lag above 5s | |
PostgresConnectionsNearMax |
warning | Postgres connections above 70% of max | |
HPAPinnedAtMax |
warning | A prod HPA sits at maxReplicas for 15m | |
PodMemoryNearLimit |
warning | A prod container is above 85% of its memory limit for 15m | |
| location-harvest | HarvestPanic |
critical | Any recovered panic in harvest goroutines in 10m |
HarvestSpendLogFailing |
critical | Any paid provider call in 10m with no spend row written | |
HarvestHandoffArchived |
critical | Any location:burst_seed task archived after exhausting retries in 10m |
|
HarvestProviderCreditsEstimated |
warning | Provider responses omit credits (spend charged at the estimate) |
|
HarvestBreakerOpen |
critical | Provider circuit breaker open for 5m | |
HarvestBudgetNearCeiling |
warning | A trigger's daily budget is above 80% (below 100%) | |
HarvestBudgetExhausted |
critical | A trigger's daily budget is at or above 100% | |
HarvestOutboxBacklog |
warning | Stored paid answers pending commit for 15m | |
| business | WSHubConnectionDrop |
warning | WebSocket hub dropping > 1 message/sec |
| semantic | SemanticHighErrorRate |
warning | LLM error rate > 10% for 10m |
SemanticAuthFailing |
critical | DeepSeek auth failures (401/403) | |
SemanticRateLimited |
warning | DeepSeek 429 rate-limit sustained for 15m | |
SemanticP95Slow |
warning | LLM p95 latency > 10s for 10m | |
SemanticDailySpendHigh |
warning | LLM 24h spend > $25 (off-peak-rate estimate) | |
SemanticCacheHitLow |
warning | Prompt-cache hit ratio < 30% for 30m | |
SearchEnrichmentHighErrorRate |
warning | Search enrichment error rate > 10% for 10m | |
SearchEnrichmentAuthFailing |
critical | Search-enrichment auth failures (401/403) | |
SearchEnrichmentRateLimited |
warning | Search-enrichment 429 rate-limit sustained for 15m | |
| certificates | CertificateExpiringSoon |
warning | A certificate expires in under 14 days |
CertificateRenewalFailed |
critical | A certificate is not Ready for 1h | |
| platform | EmbeddingsDown |
warning | Embeddings service is down |
PhotonDown |
warning | Photon geocoder is down | |
IngressDown |
critical | Traefik has no available replicas | |
ExternalSecretSyncFailed |
warning | An ExternalSecret has failed to sync for 30m | |
ArgoCDAppDegraded |
warning | An Argo CD app is Degraded or Missing for 15m | |
| observability | BlackboxProbeFailed |
warning | A blackbox /ready probe (Loki, Tempo, Photon) fails for 2m |
| messaging | NATSDown |
critical | NATS backplane is down |
NATSSlowConsumers |
warning | NATS slow consumers detected | |
NATSHighMemory |
warning | NATS memory above 800MB | |
RealtimeBroadcastLatencyHigh |
warning | WebSocket broadcast p99 latency above 1s |
Prod rules select env="prod": the ServiceMonitor (api) and PodMonitor (async) copy each overlay's env label onto every scraped series.
The kube-prometheus-stack default rules cover the platform and the node: pod crash loops (KubePodCrashLooping), pods not ready (KubePodNotReady), node memory (NodeMemoryHighUtilization, KubeNodePressure), disk I/O (NodeDiskIOSaturation), root filesystem space (NodeFilesystemSpaceFillingUp, NodeFilesystemAlmostOutOfSpace), software RAID (NodeRAIDDegraded, NodeRAIDDiskFailure), clock sync, kubelet, API server and Prometheus itself. KubeCPUOvercommit and KubeMemoryOvercommit are disabled: on one node they compare requests against zero spare capacity and would fire permanently. k3s local-path volumes report no per-volume usage (KubePersistentVolumeFillingUp never has data), so disk space is watched at the filesystem level, and the Postgres and Redis disk alerts compare data size against the PVC request. No data pod has a CPU limit, so CPUThrottlingHigh stays quiet for them.
Cost alerts (DeepSeekBalanceLow, DeepSeekBalanceCheckStale, SerperCreditsLow) live in a separate cost-alerts PrometheusRule and route through this same pipe. See Cost monitoring.
Verification¶
# 1. ESO has projected the webhook secret
kubectl get secret alertmanager-discord-credentials -n monitoring
# NAME TYPE DATA AGE
# alertmanager-discord-credentials Opaque 1 ...
# 2. Alertmanager config loaded
kubectl exec -n monitoring statefulset/alertmanager-monitoring -- \
amtool config show | head
# route: ...
# receivers: ...
# 3. Send a test alert
amtool alert add alertname=AlertmanagerTestPing severity=warning \
--alertmanager.url=http://localhost:9093 \
--annotations=summary="Test ping from amtool"
# (port-forward first: kubectl port-forward -n monitoring svc/alertmanager-operated 9093)
A real Discord message should appear in #monitoring within ~30s of the test ping.
Rotation¶
tomoda-alert-webhook-primary rotates independently:
# 1. Generate a new Discord webhook on the same #monitoring channel
# (Discord Channel Settings → Integrations → Webhooks → New Webhook)
# 2. Copy the webhook URL, then push it to GCP SM as a new version
pbpaste | tr -d '\n' | gcloud secrets versions add tomoda-alert-webhook-primary \
--project=development-485000 --data-file=-
pbcopy </dev/null
# 3. ESO syncs within 1h; force-restart Alertmanager to verify immediately
kubectl rollout restart statefulset/alertmanager-monitoring -n monitoring
# 4. Delete the old webhook in Discord (otherwise both fire)
Provider swap¶
The secret name is provider-neutral. To move to another chat or paging provider, put its webhook URL in tomoda-alert-webhook-primary and swap the discordConfigs receiver in k8s/envs/platform/manifests/alertmanager-config.yaml for the matching receiver type (and its message template). No Secret renames.
Related docs¶
- Manual setup: provisioning the webhook secret in GCP SM
- Prometheus: what's being scraped that the rules query
- Sentry data source: frontend alerts (separate channel via OAuth)