NATS¶
Core NATS is the realtime messaging backbone. The backend's WebSocket Hub uses it
as a pub/sub fan-out bus so a message published on one backend pod reaches sockets
held by every other pod (WS_BACKPLANE=nats). Manifests:
k8s/envs/platform/nats/manifests.yaml, deployed by the nats Argo CD
Application into the platform namespace.
What it does¶
The Hub keeps one connection per backend pod, not one per user. When a pod broadcasts (a chat message, a notification), it publishes to a subject; NATS fans the message out to the other pods, which then write to their local sockets. This is the seam that lets the backend scale horizontally without every pod needing a socket to every user.
It runs Core NATS with no JetStream: pure at-most-once pub/sub, no persistence, no stream storage, no quorum. Realtime fan-out doesn't need delivery guarantees (a dropped broadcast is a missed frame, not lost data), so there is nothing to replicate and a single node is sufficient.
Shape¶
| Piece | Value |
|---|---|
| Workload | StatefulSet nats-bus, image nats:2.15.1-alpine, 1 replica |
| Service | nats-bus.platform.svc.cluster.local, headless (clusterIP: None) |
| Client port | 4222 |
| Monitor port | 8222 (/healthz, varz/connz/subz) |
| Metrics port | 7777 (exporter sidecar) |
| Config | nats-bus-config ConfigMap (nats.conf) |
| Auth | One account per env; passwords from the nats-bus-auth ExternalSecret |
A headless Service is a StatefulSet requirement, not a load-balancer: clients
connect to nats-bus.platform.svc.cluster.local:4222 and DNS returns the pod IP
directly.
Metrics¶
The pod runs a prometheus-nats-exporter:0.20.2 sidecar. It scrapes the NATS
monitoring endpoint (localhost:8222) with -varz -connz -subz -routez and
re-exposes it as Prometheus metrics on :7777. A ServiceMonitor named nats
in platform (matched cluster-wide by the operator's empty
serviceMonitorSelector) scrapes that port every 30s. The NATS Grafana
dashboard (k8s/apps/grafana-dashboards/data/nats.json) visualises it.
Auth¶
Each env is a separate NATS account, so subjects never cross: the dev
credential can't subscribe to or publish prod's client.<uuid> and
session.<uuid> subjects.
| Account | User | Server password (GSM) | Client (NATS_URL in backend-nats) |
|---|---|---|---|
DEV |
tomoda-dev |
tomoda-nats-password-dev |
tomoda-nats-url-dev |
PROD |
tomoda-prod |
tomoda-nats-password-prod |
tomoda-nats-url-prod |
The client URL carries the same password:
nats://tomoda-<env>:<password>@nats-bus.platform.svc.cluster.local:4222. The
NATS client reads credentials from the URL. Both clients, tomoda-api and
tomoda-async, read NATS_URL from the backend-nats Secret (its own
ExternalSecret per env) through a required envFrom. The backend's built-in
default URL (nats.platform.svc.cluster.local) names a server that no longer
exists, so a pod without the Secret stays in CreateContainerConfigError
instead of starting without realtime. Passwords are openssl rand -hex 32, so
they need no URL escaping.
nats-bus reads both passwords from nats-bus-auth. Until both GSM secrets
have a version, the pod waits in CreateContainerConfigError. It starts on its
own once External Secrets creates the Secret.
Password expansion must stay unquoted
nats.conf expands each password via password: $NATS_DEV_PASSWORD.
nats-server treats a quoted "$VAR" as a literal string, not an environment
expansion, so the reference must remain unquoted.
Rotating a password¶
Changing the server password and the client URL can't happen atomically, so a
rotation briefly drops realtime. Add both versions, then restart nats-bus and
the env's tomoda-api and tomoda-async together. The other env reconnects to
nats-bus with its unchanged password:
kubectl annotate externalsecret nats-bus-auth -n platform force-sync=$(date +%s) --overwrite
kubectl annotate externalsecret backend-nats -n tomoda-<env> force-sync=$(date +%s) --overwrite
kubectl rollout restart statefulset/nats-bus -n platform
kubectl rollout restart deployment/tomoda-api deployment/tomoda-async -n tomoda-<env>
Guardrails¶
Set in the nats.conf ConfigMap:
| Setting | Value | Why |
|---|---|---|
max_payload |
262144 (256 KiB) | Chat/notification envelopes are small; cap well below the 1 MB default |
max_connections |
4096 | Generous headroom (one connection per backend pod) |
write_deadline |
10s | Disconnect a consumer the server can't write to in time, so one slow pod can't wedge the bus |
Per-connection subscriptions are left unlimited so the per-subject interest model can subscribe to many subjects per pod.
The nats-ingress NetworkPolicy allows 4222 only from tomoda-api and
tomoda-async pods in tomoda-dev and tomoda-prod, and 7777 only from
monitoring. Nothing else reaches the pod.
Scaling¶
NATS is not autoscaled. Churn on a realtime bus causes reconnects, so there
is no HPA on it. Connection growth is a backend-HPA lever, not a NATS one:
one connection per backend pod means a single node handles a large backend fleet.
The backend tomoda-api pool scales on WebSocket connections per pod
(prometheus-adapter custom metric), and NATS rides that
without needing to scale itself.
The server runs without cluster routes. Moving to HA means adding a cluster
block with peer routes and authorization (its own user and password), a 6222
rule between NATS pods in nats-ingress, and replicas: 3.
Alerting¶
The tomoda.messaging group in
k8s/envs/platform/manifests/alerting-rules.yaml covers the bus:
| Alert | Severity | Trigger |
|---|---|---|
NATSDown |
critical | up{job="nats-bus"} == 0 for 2m; realtime stops propagating across pods |
NATSSlowConsumers |
warning | gnatsd_varz_slow_consumers > 0 for 5m; a pod isn't draining fast enough |
NATSHighMemory |
warning | gnatsd_varz_mem > 800MB for 10m; approaching the 1Gi limit |
RealtimeBroadcastLatencyHigh |
warning | WS broadcast p99 > 1s for 5m; fan-out backing up |
The gnatsd_* metric names come from the exporter sidecar. See
Alerting.
Verify¶
# Pod and headless Service
kubectl get statefulset,svc -n platform -l app.kubernetes.io/name=nats-bus
# Server health, and which account each backend pod is in
kubectl exec -n platform nats-bus-0 -c nats -- wget -qO- http://localhost:8222/varz
kubectl exec -n platform nats-bus-0 -c nats -- wget -qO- 'http://localhost:8222/connz?auth=1'
# Metrics the ServiceMonitor scrapes
kubectl exec -n platform nats-bus-0 -c metrics -- wget -qO- http://localhost:7777/metrics | grep gnatsd_varz
Related¶
- Tomoda WS Pool Split: the dedicated WS pool that uses this backplane
- prometheus-adapter: the custom-metrics adapter the backend WS HPA scales on
- Scaling: backend pool autoscaling