Monitoring¶
Prometheus, Alertmanager, Grafana, the operator, node-exporter and kube-state-metrics: one Helm chart, one Argo CD Application, one namespace.
Installed by k8s/envs/platform/monitoring/application.yaml, configured by k8s/envs/platform/monitoring/values.yaml. Custom alerting rules live in k8s/envs/platform/manifests/alerting-rules.yaml and are deployed by the sys-resources Application.
Chart and source¶
| Field | Value |
|---|---|
| Helm chart | kube-prometheus-stack |
| Repository | https://prometheus-community.github.io/helm-charts |
| Version | 87.17.0 (operator v0.92, Prometheus v3.13) |
| Destination namespace | monitoring (created by Argo CD) |
| Argo CD Application | monitoring |
| Server-side apply | Yes (ServerSideApply=true; the chart's CRDs exceed the client-side annotation size limit) |
What you get from this single chart:
- Prometheus: central scrape and TSDB. 15-day retention capped at 25 GB on a 30Gi PVC; scrapes
ServiceMonitorandPodMonitorresources from every namespace. - Alertmanager: alert routing. 1Gi PVC.
- Grafana: dashboards and exploration UI.
- node-exporter: per-node host metrics (CPU, memory, disk, network).
- kube-state-metrics: cluster object metrics (deployments, pods, PVCs, etc.).
- prometheus-operator: reconciles the CR types above.
Scrape configuration¶
By default the operator restricts ServiceMonitor discovery to resources labelled release: monitoring. That's disabled here:
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelector: {}
serviceMonitorNamespaceSelector: {}
podMonitorSelectorNilUsesHelmValues: false
podMonitorSelector: {}
podMonitorNamespaceSelector: {}
Empty selectors = "match everything". Any ServiceMonitor or PodMonitor in any namespace is scraped. That's how the Traefik ServiceMonitor (in traefik-system), the Loki ServiceMonitor (in monitoring), and the CNPG PodMonitor (in cnpg-system) all light up automatically.
Grafana¶
Grafana is exposed at https://grafana.tomoda.life through Traefik, gated by the same two oauth2-proxy middlewares as Argo CD (see oauth2-proxy):
ingress:
ingressClassName: traefik
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.middlewares: sys-oauth2-proxy-errors@kubernetescrd,sys-oauth2-proxy-auth@kubernetescrd
hosts:
- grafana.tomoda.life
Two layers of auth:
- oauth2-proxy stops unauthenticated traffic at Traefik.
- Grafana's
auth.proxytrusts theX-Auth-Request-Emailheader that oauth2-proxy injects, and auto-creates a Grafana user from it (auto_sign_up: true).
A separate auth.google block is configured but disabled (enabled: false); the proxy header path is the active one. Its client secret is still wired in from the google-oauth-credentials Secret, populated by External Secrets from GSM tomoda-google-client-secret.
A 2Gi PVC backs /var/lib/grafana, so dashboards and users survive restarts.
Dashboards¶
The chart's bundled dashboards are off (defaultDashboardsEnabled: false). Every dashboard is a JSON file in k8s/apps/grafana-dashboards/, packed into one sidecar ConfigMap per folder and filed by its grafana_folder annotation (sidecar.dashboards.folderAnnotation). Inventory and conventions: Dashboards.
A Loki datasource is added as an additionalDataSource (http://loki:3100), so logs and metrics share the same Grafana; see Loki. A Tempo datasource (http://tempo:3200) is also wired in with bidirectional jumps to Loki; see operations/observability/tempo and prometheus scrape coverage.
Custom alerting rules¶
k8s/envs/platform/manifests/alerting-rules.yaml defines a PrometheusRule named tomoda-alerts; postgres-alerts.yaml and redis-alerts.yaml hold the data-tier rules and location-harvest-alerts.yaml the place-harvester rules. Full inventory: Alerting.
| Group | Sample alerts |
|---|---|
tomoda.application |
ArgoCDSyncFailed, BackendDown, BackendHighErrorRate (>5% 5xx), BackendHighLatency (p95 > 2s) |
tomoda.database (postgres-alerts.yaml) |
PostgresDown, PostgresHighConnections, PostgresWALArchivingFailing, PostgresBaseBackupStale, PostgresLongRunningTransaction, PostgresXIDAgeHigh, PostgresDiskFillingUp, OTABackupStale |
tomoda.redis (redis-alerts.yaml) |
RedisDown, RedisHighMemory, RedisRejectingWrites, RedisPersistenceFailing, RedisRejectedConnections |
tomoda.data.containers, tomoda.data.node |
DataStoreMemoryNearLimit, DataStoreRestarted, NodeRootFilesystemLow |
tomoda.scaling |
PostgresCPUHigh, PostgresReplicaLagNearBudget, PostgresConnectionsNearMax, HPAPinnedAtMax, PodMemoryNearLimit |
tomoda.business |
WSHubConnectionDrop |
tomoda.platform |
IngressDown, ExternalSecretSyncFailed, ArgoCDAppDegraded, PhotonDown, EmbeddingsDown |
tomoda.semantic |
SemanticHighErrorRate, SemanticAuthFailing, SemanticRateLimited, SemanticP95Slow, SemanticDailySpendHigh, SemanticCacheHitLow, SearchEnrichmentHighErrorRate, SearchEnrichmentAuthFailing, SearchEnrichmentRateLimited |
tomoda.certificates |
CertificateExpiringSoon, CertificateRenewalFailed |
tomoda.observability |
BlackboxProbeFailed |
tomoda.messaging |
NATSDown, NATSSlowConsumers, NATSHighMemory, RealtimeBroadcastLatencyHigh |
The release: monitoring label on the rule is what the Alertmanager picks up via the kube-prometheus-stack's default ruleSelector. Routing to Discord is defined in an AlertmanagerConfig CR; see Alerting. Cost series and alerts are a separate PrometheusRule, cost-alerts; see Cost monitoring.
Operational notes¶
-
Grafana admin password. The public hostname signs users in through oauth2-proxy; the local
adminlogin only matters on a direct port-forward. Its password comes from GCP SMtomoda-grafana-admin-passwordthrough thegrafana-admin-credentialsExternalSecret (grafana.admin.existingSecret). Grafana applies it only when its database is first created, so after a rotation also reset the stored one:kubectl -n monitoring exec deploy/monitoring-grafana -c grafana -- \ grafana cli admin reset-admin-password "$(gcloud secrets versions access latest \ --secret=tomoda-grafana-admin-password --project=development-485000)" -
Node, disk and RAID alerts come from the chart's default rules;
KubeCPUOvercommitandKubeMemoryOvercommitare disabled because they misfire on a single node. - Disabling the chart loses CRDs. As with cert-manager, the kube-prometheus-stack ships its CRDs. Uninstalling cascades to every
ServiceMonitor,PrometheusRule, etc. in the cluster.