Skip to content

NATS

Core NATS is the realtime messaging backbone. The backend's WebSocket Hub uses it as a pub/sub fan-out bus so a message published on one backend pod reaches sockets held by every other pod (WS_BACKPLANE=nats). Manifests: k8s/envs/platform/nats/manifests.yaml, deployed by the platform Argo CD Application into the platform namespace.

What it does

The Hub keeps one connection per backend pod, not one per user. When a pod broadcasts (a chat message, a notification), it publishes to a subject; NATS fans the message out to the other pods, which then write to their local sockets. This is the seam that lets the backend scale horizontally without every pod needing a socket to every user.

It runs Core NATS with no JetStream: pure at-most-once pub/sub, no persistence, no stream storage, no quorum. Realtime fan-out doesn't need delivery guarantees (a dropped broadcast is a missed frame, not lost data), so there is nothing to replicate and a single node is sufficient.

Shape

Piece Value
Workload StatefulSet nats, image nats:2.11-alpine
Replicas 1 (cluster-ready config seeds peer routes for nats-0..2)
Service nats.platform.svc.cluster.local, headless (clusterIP: None)
Client port 4222
Cluster port 6222 (peer routes)
Monitor port 8222 (/healthz, varz/connz/subz/routez)
Metrics port 7777 (exporter sidecar)
Config nats-config ConfigMap (nats.conf)
Auth nats-auth ExternalSecret (token)

The Service is headless so each pod is addressable by a stable DNS name (nats-0.nats.platform.svc.cluster.local), which the cluster routes below rely on for peer discovery. A headless Service is a StatefulSet requirement, not a load-balancer: clients connect to nats.platform.svc.cluster.local:4222 and DNS returns the pod IPs directly.

Metrics

The pod runs a prometheus-nats-exporter:0.20.1 sidecar. It scrapes the NATS monitoring endpoint (localhost:8222) with -varz -connz -subz -routez and re-exposes it as Prometheus metrics on :7777. A ServiceMonitor named nats in platform (matched cluster-wide by the operator's empty serviceMonitorSelector) scrapes that port every 30s. The community NATS dashboard (grafana.com gnetId: 2279) visualises it, provisioned via k8s/envs/platform/monitoring/values.yaml.

Auth

Client connections are gated by a shared token. The value lives in GCP Secret Manager (tomoda-nats-auth-token) and is projected by External Secrets into the nats-auth Secret.

Side Env var Source
Server NATS_AUTH_TOKEN nats-auth Secret (this manifest)
Client (tomoda-api) NATS_TOKEN same GSM secret, projected in the app namespace

Token expansion must stay unquoted

nats.conf expands the token via token: $NATS_AUTH_TOKEN. nats-server treats a quoted "$VAR" as a literal string, not an environment expansion, so the reference must remain unquoted.

Guardrails

Set in the nats.conf ConfigMap:

Setting Value Why
max_payload 262144 (256 KiB) Chat/notification envelopes are small; cap well below the 1 MB default
max_connections 4096 Generous headroom (one connection per backend pod)
write_deadline 10s Disconnect a consumer the server can't write to in time, so one slow pod can't wedge the bus

Per-connection subscriptions are left unlimited so the per-subject interest model can subscribe to many subjects per pod.

Ingress is restricted by the nats-ingress NetworkPolicy to ports 4222 (client), 6222 (peer cluster), and 7777 (metrics scrape). Client access is namespace-agnostic to match the existing cross-namespace backend connections.

Cluster routes and scaling

The config seeds peer routes for three replicas (nats-0..2) on port 6222. nats-server ignores routes to itself and to peers that don't exist yet, so the same config forms a mesh at any replica count from 1 to 3. Extend the route list to grow past 3.

NATS is not autoscaled. Churn on a realtime bus causes reconnects, so there is no HPA on it. Connection growth is a backend-HPA lever, not a NATS one: one connection per backend pod means a single node handles a large backend fleet. The backend tomoda-api pool scales on WebSocket connections per pod (prometheus-adapter custom metric), and NATS rides that without needing to scale itself.

Moving to HA is a deliberate, fixed bump: edit replicas from 1 to 3 in the StatefulSet. That is the lever if throughput or a single-point-of-failure concern warrants it, not an automatic response to load.

Alerting

The tomoda.messaging group in k8s/envs/platform/manifests/alerting-rules.yaml covers the bus:

Alert Severity Trigger
NATSDown critical up{job="nats"} == 0 for 2m; realtime stops propagating across pods
NATSSlowConsumers warning gnatsd_varz_slow_consumers > 0 for 5m; a pod isn't draining fast enough
NATSHighMemory warning gnatsd_varz_mem > 800MB for 10m; approaching the 1Gi limit
RealtimeBroadcastLatencyHigh warning WS broadcast p99 > 1s for 5m; fan-out backing up

The gnatsd_* metric names come from the exporter sidecar. See Alerting.

Verify

# Pod and headless Service
kubectl get statefulset,svc -n platform -l app.kubernetes.io/name=nats

# Server health and connection/route state
kubectl exec -n platform nats-0 -c nats -- wget -qO- http://localhost:8222/varz
kubectl exec -n platform nats-0 -c nats -- wget -qO- http://localhost:8222/routez

# Metrics the ServiceMonitor scrapes
kubectl exec -n platform nats-0 -c metrics -- wget -qO- http://localhost:7777/metrics | grep gnatsd_varz