Skip to content

Monitoring

Prometheus, Alertmanager, Grafana, the operator, node-exporter and kube-state-metrics: one Helm chart, one Argo CD Application, one namespace.

Installed by k8s/envs/platform/monitoring/application.yaml, configured by k8s/envs/platform/monitoring/values.yaml. Custom alerting rules live in k8s/envs/platform/manifests/alerting-rules.yaml and are deployed by the sys-resources Application.

Chart and source

Field Value
Helm chart kube-prometheus-stack
Repository https://prometheus-community.github.io/helm-charts
Version 87.17.0 (operator v0.92, Prometheus v3.13)
Destination namespace monitoring (created by Argo CD)
Argo CD Application monitoring
Server-side apply Yes (ServerSideApply=true; the chart's CRDs exceed the client-side annotation size limit)

What you get from this single chart:

  • Prometheus: central scrape and TSDB. 15-day retention capped at 25 GB on a 30Gi PVC; scrapes ServiceMonitor and PodMonitor resources from every namespace.
  • Alertmanager: alert routing. 1Gi PVC.
  • Grafana: dashboards and exploration UI.
  • node-exporter: per-node host metrics (CPU, memory, disk, network).
  • kube-state-metrics: cluster object metrics (deployments, pods, PVCs, etc.).
  • prometheus-operator: reconciles the CR types above.

Scrape configuration

By default the operator restricts ServiceMonitor discovery to resources labelled release: monitoring. That's disabled here:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}
    podMonitorSelectorNilUsesHelmValues: false
    podMonitorSelector: {}
    podMonitorNamespaceSelector: {}

Empty selectors = "match everything". Any ServiceMonitor or PodMonitor in any namespace is scraped. That's how the Traefik ServiceMonitor (in traefik-system), the Loki ServiceMonitor (in monitoring), and the CNPG PodMonitor (in cnpg-system) all light up automatically.

Grafana

Grafana is exposed at https://grafana.tomoda.life through Traefik, gated by the same two oauth2-proxy middlewares as Argo CD (see oauth2-proxy):

ingress:
  ingressClassName: traefik
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: sys-oauth2-proxy-errors@kubernetescrd,sys-oauth2-proxy-auth@kubernetescrd
  hosts:
    - grafana.tomoda.life

Two layers of auth:

  1. oauth2-proxy stops unauthenticated traffic at Traefik.
  2. Grafana's auth.proxy trusts the X-Auth-Request-Email header that oauth2-proxy injects, and auto-creates a Grafana user from it (auto_sign_up: true).

A separate auth.google block is configured but disabled (enabled: false); the proxy header path is the active one. Its client secret is still wired in from the google-oauth-credentials Secret, populated by External Secrets from GSM tomoda-google-client-secret.

A 2Gi PVC backs /var/lib/grafana, so dashboards and users survive restarts.

Dashboards

The chart's bundled dashboards are off (defaultDashboardsEnabled: false). Every dashboard is a JSON file in k8s/apps/grafana-dashboards/, packed into one sidecar ConfigMap per folder and filed by its grafana_folder annotation (sidecar.dashboards.folderAnnotation). Inventory and conventions: Dashboards.

A Loki datasource is added as an additionalDataSource (http://loki:3100), so logs and metrics share the same Grafana; see Loki. A Tempo datasource (http://tempo:3200) is also wired in with bidirectional jumps to Loki; see operations/observability/tempo and prometheus scrape coverage.

Custom alerting rules

k8s/envs/platform/manifests/alerting-rules.yaml defines a PrometheusRule named tomoda-alerts; postgres-alerts.yaml and redis-alerts.yaml hold the data-tier rules and location-harvest-alerts.yaml the place-harvester rules. Full inventory: Alerting.

Group Sample alerts
tomoda.application ArgoCDSyncFailed, BackendDown, BackendHighErrorRate (>5% 5xx), BackendHighLatency (p95 > 2s)
tomoda.database (postgres-alerts.yaml) PostgresDown, PostgresHighConnections, PostgresWALArchivingFailing, PostgresBaseBackupStale, PostgresLongRunningTransaction, PostgresXIDAgeHigh, PostgresDiskFillingUp, OTABackupStale
tomoda.redis (redis-alerts.yaml) RedisDown, RedisHighMemory, RedisRejectingWrites, RedisPersistenceFailing, RedisRejectedConnections
tomoda.data.containers, tomoda.data.node DataStoreMemoryNearLimit, DataStoreRestarted, NodeRootFilesystemLow
tomoda.scaling PostgresCPUHigh, PostgresReplicaLagNearBudget, PostgresConnectionsNearMax, HPAPinnedAtMax, PodMemoryNearLimit
tomoda.business WSHubConnectionDrop
tomoda.platform IngressDown, ExternalSecretSyncFailed, ArgoCDAppDegraded, PhotonDown, EmbeddingsDown
tomoda.semantic SemanticHighErrorRate, SemanticAuthFailing, SemanticRateLimited, SemanticP95Slow, SemanticDailySpendHigh, SemanticCacheHitLow, SearchEnrichmentHighErrorRate, SearchEnrichmentAuthFailing, SearchEnrichmentRateLimited
tomoda.certificates CertificateExpiringSoon, CertificateRenewalFailed
tomoda.observability BlackboxProbeFailed
tomoda.messaging NATSDown, NATSSlowConsumers, NATSHighMemory, RealtimeBroadcastLatencyHigh

The release: monitoring label on the rule is what the Alertmanager picks up via the kube-prometheus-stack's default ruleSelector. Routing to Discord is defined in an AlertmanagerConfig CR; see Alerting. Cost series and alerts are a separate PrometheusRule, cost-alerts; see Cost monitoring.

Operational notes

  • Grafana admin password. The public hostname signs users in through oauth2-proxy; the local admin login only matters on a direct port-forward. Its password comes from GCP SM tomoda-grafana-admin-password through the grafana-admin-credentials ExternalSecret (grafana.admin.existingSecret). Grafana applies it only when its database is first created, so after a rotation also reset the stored one:

    kubectl -n monitoring exec deploy/monitoring-grafana -c grafana -- \
      grafana cli admin reset-admin-password "$(gcloud secrets versions access latest \
        --secret=tomoda-grafana-admin-password --project=development-485000)"
    
  • Node, disk and RAID alerts come from the chart's default rules; KubeCPUOvercommit and KubeMemoryOvercommit are disabled because they misfire on a single node.

  • Disabling the chart loses CRDs. As with cert-manager, the kube-prometheus-stack ships its CRDs. Uninstalling cascades to every ServiceMonitor, PrometheusRule, etc. in the cluster.