Skip to content

Prometheus scrape coverage

This page tracks what the cluster's Prometheus actually scrapes. The deployment itself is documented in kubernetes/system/monitoring.md; this page focuses on who is scraped, how, and where the ServiceMonitor / PodMonitor / Probe CR lives.

kube-prometheus-stack's serviceMonitorSelector is set to {} (match everything in any namespace), so any of the resources below light up automatically once they exist.

ServiceMonitor inventory

Target Resource type Owned by Namespace
Tomoda backend (tomoda-api) ServiceMonitor tomoda-backend k8s/apps/tomoda/base/servicemonitor.yaml tomoda-dev, tomoda-prod
Tomoda async workers (tomoda-async) PodMonitor tomoda-async k8s/apps/tomoda/base/servicemonitor.yaml tomoda-dev, tomoda-prod
External Secrets (controller, webhook, cert-controller) ServiceMonitor (via Helm serviceMonitor.enabled) k8s/envs/platform/external-secrets/application.yaml external-secrets
Tempo (own internal metrics) ServiceMonitor (via Helm serviceMonitor.enabled: true) k8s/envs/platform/tempo/values.yaml monitoring
Loki ServiceMonitor (via Helm) k8s/envs/platform/loki/values.yaml monitoring
Traefik ServiceMonitor (via Helm) k8s/envs/platform/traefik/values.yaml monitoring
Redis (dev) ServiceMonitor (via Bitnami metrics.serviceMonitor.enabled: true) k8s/envs/dev/redis/values.yaml monitoring (cross-ns selector → data)
Redis (prod) ServiceMonitor (via Bitnami) k8s/envs/prod/redis/values.yaml monitoring (cross-ns selector → data)
Argo CD (controller, server, repo-server, applicationSet, notifications) ServiceMonitor (committed manifests) k8s/envs/platform/manifests/argocd-servicemonitors.yaml argocd
Postgres (CNPG primary + standby) PodMonitor (CNPG built-in) k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml cnpg-system
Blackbox-exporter (own internal metrics) ServiceMonitor (via Helm) k8s/envs/platform/blackbox-exporter/values.yaml monitoring
Photon (/status via blackbox) Probe photon k8s/envs/platform/manifests/photon-probe.yaml monitoring
Loki (/ready via blackbox) Probe loki-ready k8s/envs/platform/manifests/loki-probe.yaml monitoring
Tempo (/ready via blackbox) Probe tempo-ready k8s/envs/platform/manifests/tempo-probe.yaml monitoring

The release: monitoring label is set on each of these by convention; the operator's selector is empty.

Tomoda backend specifics

The backend exposes /metrics on the same listener as /health (port 8080) when OBSERVABILITY_METRICS_ENABLED=true. That env var is set in both the dev and prod overlay kustomizations:

- name: OBSERVABILITY_METRICS_ENABLED
  value: "true"
- name: OBSERVABILITY_TRACING_ENABLED
  value: "true"
- name: OBSERVABILITY_OTLP_ENDPOINT
  value: "tempo.monitoring.svc.cluster.local:4317"
- name: OBSERVABILITY_OTLP_INSECURE
  value: "true"
- name: OBSERVABILITY_SAMPLE_RATE
  value: "0.1"

The ServiceMonitor selects app: tomoda-api; async pods have no Service, so a PodMonitor selects app: tomoda-async. The overlays' commonLabels: env: dev|prod lands on the Service and pods, and targetLabels: [env] / podTargetLabels: [env] copy it onto every scraped series. The app's own metrics carry no env label, so alerts, SLOs and dashboards rely on this to tell dev from prod.

NetworkPolicy

tomoda-api-policy and tomoda-async-policy (k8s/apps/tomoda/base/network-policy.yaml) allow inbound :8080 from the monitoring namespace, alongside Traefik. Without that rule, scrapes are silently dropped and the targets show up == 0 with no obvious cause.

Argo CD specifics

The Argo CD Helm values (k8s/envs/platform/argocd/values.yaml) enable per-component metrics Services, and k8s/envs/platform/manifests/argocd-servicemonitors.yaml adds five ServiceMonitors in the argocd namespace, one each for:

  • argocd-application-controller: sync queue depth, app health counts
  • argocd-server: API request rates
  • argocd-repo-server: repo cache stats, manifest generation latency
  • argocd-applicationset-controller
  • argocd-notifications-controller

The Platform overview dashboard lists apps that are not Healthy + Synced from argocd_app_info.

Redis

Both envs enable the redis-exporter sidecar and its ServiceMonitor (k8s/envs/{dev,prod}/redis/values.yaml):

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    namespace: monitoring
    namespaceSelector:
      matchNames:
        - data
    additionalLabels:
      release: monitoring

Series carry job="redis-{dev,prod}-metrics".

Blackbox probes: service readiness

The prometheus-blackbox-exporter Helm chart in k8s/envs/platform/blackbox-exporter/ deploys an HTTP prober. A Probe CR then tells Prometheus to have blackbox hit a service's health endpoint on an interval and record probe_success. This covers two cases:

Probe Target Why blackbox and not a ServiceMonitor
photon http://photon.platform.svc:2322/status Photon exposes no Prometheus metrics at all
loki-ready http://loki.monitoring.svc:3100/ready catches a crashlooping / stale pod behind a Healthy Argo app
tempo-ready http://tempo.monitoring.svc:3200/ready Same: Argo's object-level check can't see the endpoint

The resulting metrics, labelled with the target and a service label:

  • probe_success: 1 if the request succeeded, 0 otherwise
  • probe_duration_seconds: how long the request took
  • probe_http_status_code: last response code

Why the /ready probes exist. Argo CD's health check trusts the Kubernetes object, not the running endpoint: a StatefulSet can report Healthy while its pod crashloops or runs a stale spec. A blackbox probe on the service's own /ready closes that gap: it goes red the moment the pod stops actually serving.

One generic alert covers all of them: BlackboxProbeFailed (probe_success == 0 for 2m, in the tomoda.observability group of alerting-rules.yaml). The failing service label names which one.

Debugging "where are my metrics?"

# Is the ServiceMonitor visible to Prometheus?
kubectl get servicemonitor -A | grep <name>

# Does Prometheus actually have the target?
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090/targets → look for the job

# Is the pod's /metrics endpoint working?
kubectl port-forward -n <ns> <pod> 8080:8080
curl localhost:8080/metrics | head -20

# Is a NetworkPolicy blocking the scrape?
kubectl get networkpolicy -n <ns>

If a ServiceMonitor exists but the target is missing from Prometheus, the most common causes are: (a) wrong port name (endpoints[].port must match the Service's ports[].name, not the number), (b) the Service's selector doesn't actually hit any pods, © a NetworkPolicy blocks ingress from monitoring.