Prometheus scrape coverage¶
This page tracks what the cluster's Prometheus actually scrapes. The deployment itself is documented in kubernetes/system/monitoring.md; this page focuses on who is scraped, how, and where the ServiceMonitor / PodMonitor / Probe CR lives.
kube-prometheus-stack's serviceMonitorSelector is set to {} (match everything in any namespace), so any of the resources below light up automatically once they exist.
ServiceMonitor inventory¶
| Target | Resource type | Owned by | Namespace |
|---|---|---|---|
Tomoda backend (tomoda-api) |
ServiceMonitor tomoda-backend |
k8s/apps/tomoda/base/servicemonitor.yaml |
tomoda-dev, tomoda-prod |
Tomoda async workers (tomoda-async) |
PodMonitor tomoda-async |
k8s/apps/tomoda/base/servicemonitor.yaml |
tomoda-dev, tomoda-prod |
| External Secrets (controller, webhook, cert-controller) | ServiceMonitor (via Helm serviceMonitor.enabled) |
k8s/envs/platform/external-secrets/application.yaml |
external-secrets |
| Tempo (own internal metrics) | ServiceMonitor (via Helm serviceMonitor.enabled: true) |
k8s/envs/platform/tempo/values.yaml |
monitoring |
| Loki | ServiceMonitor (via Helm) | k8s/envs/platform/loki/values.yaml |
monitoring |
| Traefik | ServiceMonitor (via Helm) | k8s/envs/platform/traefik/values.yaml |
monitoring |
| Redis (dev) | ServiceMonitor (via Bitnami metrics.serviceMonitor.enabled: true) |
k8s/envs/dev/redis/values.yaml |
monitoring (cross-ns selector → data) |
| Redis (prod) | ServiceMonitor (via Bitnami) | k8s/envs/prod/redis/values.yaml |
monitoring (cross-ns selector → data) |
| Argo CD (controller, server, repo-server, applicationSet, notifications) | ServiceMonitor (committed manifests) | k8s/envs/platform/manifests/argocd-servicemonitors.yaml |
argocd |
| Postgres (CNPG primary + standby) | PodMonitor (CNPG built-in) | k8s/envs/{dev,prod}/postgres/manifests/cluster.yaml |
cnpg-system |
| Blackbox-exporter (own internal metrics) | ServiceMonitor (via Helm) | k8s/envs/platform/blackbox-exporter/values.yaml |
monitoring |
Photon (/status via blackbox) |
Probe photon |
k8s/envs/platform/manifests/photon-probe.yaml |
monitoring |
Loki (/ready via blackbox) |
Probe loki-ready |
k8s/envs/platform/manifests/loki-probe.yaml |
monitoring |
Tempo (/ready via blackbox) |
Probe tempo-ready |
k8s/envs/platform/manifests/tempo-probe.yaml |
monitoring |
The release: monitoring label is set on each of these by convention; the operator's selector is empty.
Tomoda backend specifics¶
The backend exposes /metrics on the same listener as /health (port 8080) when OBSERVABILITY_METRICS_ENABLED=true. That env var is set in both the dev and prod overlay kustomizations:
- name: OBSERVABILITY_METRICS_ENABLED
value: "true"
- name: OBSERVABILITY_TRACING_ENABLED
value: "true"
- name: OBSERVABILITY_OTLP_ENDPOINT
value: "tempo.monitoring.svc.cluster.local:4317"
- name: OBSERVABILITY_OTLP_INSECURE
value: "true"
- name: OBSERVABILITY_SAMPLE_RATE
value: "0.1"
The ServiceMonitor selects app: tomoda-api; async pods have no Service, so a PodMonitor selects app: tomoda-async. The overlays' commonLabels: env: dev|prod lands on the Service and pods, and targetLabels: [env] / podTargetLabels: [env] copy it onto every scraped series. The app's own metrics carry no env label, so alerts, SLOs and dashboards rely on this to tell dev from prod.
NetworkPolicy¶
tomoda-api-policy and tomoda-async-policy (k8s/apps/tomoda/base/network-policy.yaml) allow inbound :8080 from the monitoring namespace, alongside Traefik. Without that rule, scrapes are silently dropped and the targets show up == 0 with no obvious cause.
Argo CD specifics¶
The Argo CD Helm values (k8s/envs/platform/argocd/values.yaml) enable per-component metrics Services, and k8s/envs/platform/manifests/argocd-servicemonitors.yaml adds five ServiceMonitors in the argocd namespace, one each for:
argocd-application-controller: sync queue depth, app health countsargocd-server: API request ratesargocd-repo-server: repo cache stats, manifest generation latencyargocd-applicationset-controllerargocd-notifications-controller
The Platform overview dashboard lists apps that are not Healthy + Synced from argocd_app_info.
Redis¶
Both envs enable the redis-exporter sidecar and its ServiceMonitor (k8s/envs/{dev,prod}/redis/values.yaml):
metrics:
enabled: true
serviceMonitor:
enabled: true
namespace: monitoring
namespaceSelector:
matchNames:
- data
additionalLabels:
release: monitoring
Series carry job="redis-{dev,prod}-metrics".
Blackbox probes: service readiness¶
The prometheus-blackbox-exporter Helm chart in k8s/envs/platform/blackbox-exporter/ deploys an HTTP prober. A Probe CR then tells Prometheus to have blackbox hit a service's health endpoint on an interval and record probe_success. This covers two cases:
| Probe | Target | Why blackbox and not a ServiceMonitor |
|---|---|---|
photon |
http://photon.platform.svc:2322/status |
Photon exposes no Prometheus metrics at all |
loki-ready |
http://loki.monitoring.svc:3100/ready |
catches a crashlooping / stale pod behind a Healthy Argo app |
tempo-ready |
http://tempo.monitoring.svc:3200/ready |
Same: Argo's object-level check can't see the endpoint |
The resulting metrics, labelled with the target and a service label:
probe_success: 1 if the request succeeded, 0 otherwiseprobe_duration_seconds: how long the request tookprobe_http_status_code: last response code
Why the /ready probes exist. Argo CD's health check trusts the Kubernetes object, not the running endpoint: a StatefulSet can report Healthy while its pod crashloops or runs a stale spec. A blackbox probe on the service's own /ready closes that gap: it goes red the moment the pod stops actually serving.
One generic alert covers all of them: BlackboxProbeFailed (probe_success == 0 for 2m, in the tomoda.observability group of alerting-rules.yaml). The failing service label names which one.
Debugging "where are my metrics?"¶
# Is the ServiceMonitor visible to Prometheus?
kubectl get servicemonitor -A | grep <name>
# Does Prometheus actually have the target?
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090/targets → look for the job
# Is the pod's /metrics endpoint working?
kubectl port-forward -n <ns> <pod> 8080:8080
curl localhost:8080/metrics | head -20
# Is a NetworkPolicy blocking the scrape?
kubectl get networkpolicy -n <ns>
If a ServiceMonitor exists but the target is missing from Prometheus, the most common causes are: (a) wrong port name (endpoints[].port must match the Service's ports[].name, not the number), (b) the Service's selector doesn't actually hit any pods, © a NetworkPolicy blocks ingress from monitoring.