Skip to content

Tomoda WS Pool Split (Planned)

Planned topology, not deployed: split the WebSocket Hub out of tomoda-api into its own pool (tomoda-ws). The backend already ships the api-hub and ws-hub modes; what's missing is the k8s wiring, a path-based ingress rule and a WS-pool HPA. Don't pre-build it.

Why split

Today HTTP API and WebSocket termination share the tomoda-api pool (multi-hub). Its HPA scales on CPU, memory and WS connections per pod, so one pool has to be sized for two load shapes.

Splitting tomoda-api into tomoda-api (HTTP only) and tomoda-ws (WebSocket only) gives us:

  • Independent scaling shape. WS pool scales on connection count (long-lived, sticky, memory-bound). API pool keeps scaling on CPU/RPS.
  • Different pod sizing. WS pods can be sized for connection state (more memory, less CPU). API pods stay lean.
  • Blast-radius reduction. A WS-related rollout (Hub change, WebSocket library upgrade) doesn't touch the API pool, and vice versa.

The cost is more moving parts: another Deployment, another Service, an ingress-level routing rule, and a custom WS-connection metric for the HPA.

When to do this

Trigger criteria, any of which is sufficient:

  • WS-connection count regularly exceeds 10k during peak.
  • tomoda-api HPA spends meaningful time pinned at maxReplicas while the HTTP request-rate metrics show idle pods. That gap is WS-driven load not visible to CPU-based scaling.
  • WS-related changes cause API-side incidents (a Hub bug bricks the API pool too) more than once.

Until one of these fires, the multi-hub topology is correct.

Target topology

Traefik
↙   ↘
/ws/*
tomoda-ws HPA on ws_connections
everything else
tomoda-api HPA on CPU + RPS
Path-based ingress split lets each pool scale on its own signal: long-lived WS connections (memory-bound) vs short bursty HTTP requests (CPU-bound).

Both pools run the same image. The Hub on tomoda-ws keeps the NATS backplane (WS_BACKPLANE=nats) that tomoda-api uses today for cross-pod fanout.

tomoda-async (worker + scheduler) is untouched.

Manifest sketch

In k8s/apps/tomoda/base/:

# backend-ws-deployment.yaml: Deployment tomoda-ws (mode ws-hub)
# Service ws-service (clusterIP, port 8080, selector app: tomoda-ws)

# backend-api-deployment.yaml: change SERVER_MODE multi-hub → api-hub
# (api-hub serves /api/v1/* + /health, no /ws/* routes, no Hub)

Update kustomization.yaml to add the new file.

In network-policy.yaml, add:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tomoda-ws-policy
spec:
  podSelector:
    matchLabels:
      app: tomoda-ws
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik-system
      ports:
        - protocol: TCP
          port: 8080

In ingress.yaml, replace the single-path backend-service rule with two paths, /ws/* → ws-service and default → backend-service:

- host: api.tomoda.life
  http:
    paths:
      - path: /ws
        pathType: Prefix
        backend:
          service:
            name: ws-service
            port: { number: 8080 }
      - path: /
        pathType: Prefix
        backend:
          service:
            name: backend-service
            port: { number: 8080 }

Traefik picks the longer prefix first, so /ws/chat/... lands on the WS pool and everything else stays on the API pool.

HPA: custom WS connection metric

prometheus-adapter already exposes tomoda_ws_connections_active per pod (the prod tomoda-api HPA scales on it); ws-hub pods emit it too. Reference it from tomoda-ws-hpa:

spec:
  metrics:
    - type: Pods
      pods:
        metric:
          name: tomoda_ws_connections_active
        target:
          type: AverageValue
          averageValue: "500"   # scale to keep <=500 conns/pod

Tune averageValue from steady-state telemetry.

Migration plan (when triggered)

Sequence so we don't drop connections:

  1. Land the new tomoda-ws Deployment + Service in the cluster (replicas 0). Verify it builds and reconciles via Argo CD without any traffic.
  2. Bring it to 2 replicas, confirm /health succeeds on the new pods.
  3. Add the ingress /ws path pointing at ws-service. New WS connections start landing on the WS pool; existing connections remain on tomoda-api pods until they reconnect (Hub is per-pod; old pods still own their sockets).
  4. Flip tomoda-api from multi-hub to api-hub. Existing sockets on api pods are torn down on the rollout; clients reconnect to the WS pool via the new ingress path.
  5. Drop unused metrics / HPA from tomoda-api; deploy custom-metric HPA for tomoda-ws.

Before step 1, verify the frontend reconnect path handles the rolling pod swap (see docs/architecture/realtime.md in the tomoda repo).