Skip to content

Prometheus Adapter

Prometheus Adapter serves the custom metrics API (custom.metrics.k8s.io), letting an HPA scale on a Prometheus metric instead of only CPU and memory. Manifests: k8s/envs/platform/prometheus-adapter/ (Helm chart prometheus-adapter v4.11.0), deployed by the platform Argo CD Application into the monitoring namespace next to Prometheus.

What it does

The kube-prometheus-stack scrapes app metrics into Prometheus, but the HPA controller can't query Prometheus directly. Prometheus Adapter bridges that gap: it registers as a Kubernetes APIService for custom.metrics.k8s.io/v1beta1, translates a metric request from the HPA into a PromQL query, and returns the value as a first-class Kubernetes metric.

Without it, only the CPU and memory signals from metrics-server are available to autoscalers.

Metric exposed

The chart's default rule catalogue is turned off (rules.default: false) because it exposes a large, expensive set of container metrics we don't autoscale on. Only one custom rule ships (values.yaml):

Field Value
Series tomoda_ws_connections_active{namespace!="",pod!=""}
Exposed as tomoda_ws_connections_active (per pod)
Query sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)

The query sums live WebSocket connections per pod (across the hub_type label), so the HPA sees total sockets held by each pod. Prometheus is reached at http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090.

Who consumes it

The tomoda-api HPA (k8s/apps/tomoda/overlays/prod/hpa-api.yaml) scales on this metric alongside CPU (70%) and memory (80%):

- type: Pods
  pods:
    metric:
      name: tomoda_ws_connections_active
    target:
      type: AverageValue
      averageValue: "8000"   # scale out above ~8k live sockets/pod

WebSocket connections are the signal that saturates first for a socket-bound workload: a pod can sit near its socket/memory ceiling while CPU looks calm. See Scaling and Tomoda WS Pool Split, which uses the same adapter for a connection-count HPA once the WS pool is split out.

To add another custom-metric HPA (for example Asynq queue depth on the async pool), add a rule to values.yaml mapping the Prometheus series to a metric name, then reference that name from the HPA.

Verify

# The custom-metrics API is registered and lists the metric
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq '.resources[].name'
#   .../tomoda_ws_connections_active

# Live value the HPA sees for the tomoda-api pods
kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/tomoda-prod/pods/*/tomoda_ws_connections_active" | jq

# The HPA is reading it (not <unknown>)
kubectl get hpa tomoda-api-hpa -n tomoda-prod

If the metric shows <unknown> on the HPA, confirm the adapter pod is running in monitoring, that Prometheus actually has the tomoda_ws_connections_active series (the backend must be exposing it), and that the APIService above resolves.