Skip to content

NATS

Core NATS is the realtime messaging backbone. The backend's WebSocket Hub uses it as a pub/sub fan-out bus so a message published on one backend pod reaches sockets held by every other pod (WS_BACKPLANE=nats). Manifests: k8s/envs/platform/nats/manifests.yaml, deployed by the nats Argo CD Application into the platform namespace.

What it does

The Hub keeps one connection per backend pod, not one per user. When a pod broadcasts (a chat message, a notification), it publishes to a subject; NATS fans the message out to the other pods, which then write to their local sockets. This is the seam that lets the backend scale horizontally without every pod needing a socket to every user.

It runs Core NATS with no JetStream: pure at-most-once pub/sub, no persistence, no stream storage, no quorum. Realtime fan-out doesn't need delivery guarantees (a dropped broadcast is a missed frame, not lost data), so there is nothing to replicate and a single node is sufficient.

Shape

Piece Value
Workload StatefulSet nats-bus, image nats:2.15.1-alpine, 1 replica
Service nats-bus.platform.svc.cluster.local, headless (clusterIP: None)
Client port 4222
Monitor port 8222 (/healthz, varz/connz/subz)
Metrics port 7777 (exporter sidecar)
Config nats-bus-config ConfigMap (nats.conf)
Auth One account per env; passwords from the nats-bus-auth ExternalSecret

A headless Service is a StatefulSet requirement, not a load-balancer: clients connect to nats-bus.platform.svc.cluster.local:4222 and DNS returns the pod IP directly.

Metrics

The pod runs a prometheus-nats-exporter:0.20.2 sidecar. It scrapes the NATS monitoring endpoint (localhost:8222) with -varz -connz -subz -routez and re-exposes it as Prometheus metrics on :7777. A ServiceMonitor named nats in platform (matched cluster-wide by the operator's empty serviceMonitorSelector) scrapes that port every 30s. The NATS Grafana dashboard (k8s/apps/grafana-dashboards/data/nats.json) visualises it.

Auth

Each env is a separate NATS account, so subjects never cross: the dev credential can't subscribe to or publish prod's client.<uuid> and session.<uuid> subjects.

Account User Server password (GSM) Client (NATS_URL in backend-nats)
DEV tomoda-dev tomoda-nats-password-dev tomoda-nats-url-dev
PROD tomoda-prod tomoda-nats-password-prod tomoda-nats-url-prod

The client URL carries the same password: nats://tomoda-<env>:<password>@nats-bus.platform.svc.cluster.local:4222. The NATS client reads credentials from the URL. Both clients, tomoda-api and tomoda-async, read NATS_URL from the backend-nats Secret (its own ExternalSecret per env) through a required envFrom. The backend's built-in default URL (nats.platform.svc.cluster.local) names a server that no longer exists, so a pod without the Secret stays in CreateContainerConfigError instead of starting without realtime. Passwords are openssl rand -hex 32, so they need no URL escaping.

nats-bus reads both passwords from nats-bus-auth. Until both GSM secrets have a version, the pod waits in CreateContainerConfigError. It starts on its own once External Secrets creates the Secret.

Password expansion must stay unquoted

nats.conf expands each password via password: $NATS_DEV_PASSWORD. nats-server treats a quoted "$VAR" as a literal string, not an environment expansion, so the reference must remain unquoted.

Rotating a password

Changing the server password and the client URL can't happen atomically, so a rotation briefly drops realtime. Add both versions, then restart nats-bus and the env's tomoda-api and tomoda-async together. The other env reconnects to nats-bus with its unchanged password:

kubectl annotate externalsecret nats-bus-auth -n platform force-sync=$(date +%s) --overwrite
kubectl annotate externalsecret backend-nats -n tomoda-<env> force-sync=$(date +%s) --overwrite
kubectl rollout restart statefulset/nats-bus -n platform
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-<env>

Guardrails

Set in the nats.conf ConfigMap:

Setting Value Why
max_payload 262144 (256 KiB) Chat/notification envelopes are small; cap well below the 1 MB default
max_connections 4096 Generous headroom (one connection per backend pod)
write_deadline 10s Disconnect a consumer the server can't write to in time, so one slow pod can't wedge the bus

Per-connection subscriptions are left unlimited so the per-subject interest model can subscribe to many subjects per pod.

The nats-ingress NetworkPolicy allows 4222 only from tomoda-api and tomoda-async pods in tomoda-dev and tomoda-prod, and 7777 only from monitoring. Nothing else reaches the pod.

Scaling

NATS is not autoscaled. Churn on a realtime bus causes reconnects, so there is no HPA on it. Connection growth is a backend-HPA lever, not a NATS one: one connection per backend pod means a single node handles a large backend fleet. The backend tomoda-api pool scales on WebSocket connections per pod (prometheus-adapter custom metric), and NATS rides that without needing to scale itself.

The server runs without cluster routes. Moving to HA means adding a cluster block with peer routes and authorization (its own user and password), a 6222 rule between NATS pods in nats-ingress, and replicas: 3.

Alerting

The tomoda.messaging group in k8s/envs/platform/manifests/alerting-rules.yaml covers the bus:

Alert Severity Trigger
NATSDown critical up{job="nats-bus"} == 0 for 2m; realtime stops propagating across pods
NATSSlowConsumers warning gnatsd_varz_slow_consumers > 0 for 5m; a pod isn't draining fast enough
NATSHighMemory warning gnatsd_varz_mem > 800MB for 10m; approaching the 1Gi limit
RealtimeBroadcastLatencyHigh warning WS broadcast p99 > 1s for 5m; fan-out backing up

The gnatsd_* metric names come from the exporter sidecar. See Alerting.

Verify

# Pod and headless Service
kubectl get statefulset,svc -n platform -l app.kubernetes.io/name=nats-bus

# Server health, and which account each backend pod is in
kubectl exec -n platform nats-bus-0 -c nats -- wget -qO- http://localhost:8222/varz
kubectl exec -n platform nats-bus-0 -c nats -- wget -qO- 'http://localhost:8222/connz?auth=1'

# Metrics the ServiceMonitor scrapes
kubectl exec -n platform nats-bus-0 -c metrics -- wget -qO- http://localhost:7777/metrics | grep gnatsd_varz