Skip to content

Node Lifecycle

Adding, removing and replacing machines in the k3s cluster. Every node is set up the same way: order it, install the OS, run scripts/ovh/bootstrap-node.sh. Terraform manages Cloudflare only; servers are ordered by hand and bootstrapped over SSH.

Node types

Type Runs etcd member
Server Control plane (API server, scheduler, embedded etcd) plus workloads Yes
Agent Workloads only No

etcd needs a majority of its members up to accept writes. That makes the server count matter:

Servers Survives losing Notes
1 Nothing (workloads keep running, the API stops) Today's shape
2 Nothing, and either loss stops the API Worse than 1. Never stay here
3 1 server The HA shape

So a second machine joins as an agent, unless a third server follows right away.

Before adding any node

  1. Order a Rise server in Beauharnois (BHS), monthly billing.
  2. Install from the OVH template: Ubuntu Server 24.04, RAID1 across both disks, / taking all remaining space, your ED25519 key, hostname tomoda-bhs-N, no post-install script.
  3. Generate a one-off Tailscale auth key for the node (Server Access → Auth key per node). Keys are single-use and never stored.
  4. Secrets always go to a node through /root/bootstrap.env, never on the command line (Bootstrap → Step 5). Below, "send secrets" means writing that file with the keys listed.
  5. Note the new node's public IP and the existing nodes' public IPs. Nodes talk to each other over their public IPs, encrypted by flannel's WireGuard backend; PRIVATE_CIDR opens the node ports to the listed peers only.

Adding an agent

  1. Open the first server to the new node. Re-run the bootstrap there with the new peer in PRIVATE_CIDR (comma-separated, one entry per other node):

    Send secrets: GHCR_USER, GHCR_TOKEN. Then:

    sudo PRIVATE_CIDR=<new-node-ip>/32 NODE_ROLE=server ./bootstrap-node.sh
    

    Tailscale is already joined there, so no auth key is needed.

  2. Read the join token on the first server: sudo cat /var/lib/rancher/k3s/server/node-token.

  3. Bootstrap the new node as the image's ubuntu user, with the files from scripts/ovh/ copied over:

    Send secrets: GHCR_USER, GHCR_TOKEN, TS_AUTHKEY (the new key), K3S_TOKEN (the join token). Then:

    sudo PRIVATE_CIDR=<first-server-ip>/32 K3S_URL=https://<first-server-ip>:6443 \
      NODE_ROLE=agent ./bootstrap-node.sh
    

    admin-users.sh runs as part of it, so the same people can log in.

  4. Verify: kubectl get nodes -o wide shows the node Ready with label tomoda.life/role=worker (from scripts/ovh/k3s-agent.yaml).

Workloads have no node selectors, so new pods start landing on the agent at once. Pods with local volumes stay where their data is (see Storage moves with nodes).

Adding servers for HA

Go from one server straight to three.

  1. Bootstrap two new machines as above, but with NODE_ROLE=server and K3S_URL (K3S_TOKEN still goes in the secrets file). The script drops cluster-init and joins the existing etcd.
  2. Every node's PRIVATE_CIDR lists all the other nodes: re-run the bootstrap on each existing node with the full list.
  3. Check etcd: kubectl get nodes shows three control-plane,etcd nodes, and kubectl get --raw '/readyz?verbose' reports etcd ok.
  4. Raise instances on postgres-prod (k8s/envs/prod/postgres/manifests/cluster.yaml) to 2 or 3 for a streaming standby and automatic failover, and raise the HPA ceilings.

Storage moves with nodes

PVCs use local-path: each volume is a directory on the node where it was first scheduled, and its pod can only run there. Before draining a node, deal with each volume on it:

Workload Move by
Postgres (CNPG) Add an instance (instances: +1) so a replica builds on another node, then kubectl cnpg promote <cluster> <new-instance> -n data, then lower instances again. CNPG removes the old instance's PVC
Photon Nothing. A new pod downloads the index from R2 (an hour or two of geocoding outage for the planet index)
Prometheus Delete the PVC; metric history starts over. Acceptable
Loki, Tempo Nothing to move; data lives in R2. Delete the PVC
Redis Delete the PVC; it is a cache, users may need to sign in again
OTA databases (xavia-ota-db-{dev,prod}) Gap: plain Postgres with no backups. Take a pg_dump before the move and restore it after
pgAdmin Delete the PVC; saved server definitions are recreated

Deleting a PVC lets its pod reschedule on another node with a fresh volume.

Draining and removing a node

  1. Move or delete the node's volumes (table above).
  2. Drain it:

    kubectl cordon <node>
    kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
    
  3. Remove it from the cluster: kubectl delete node <node>. For a server, k3s also removes its etcd member. Check kubectl get nodes and kubectl get --raw '/readyz?verbose'.

  4. On the old machine, uninstall k3s: sudo /usr/local/bin/k3s-uninstall.sh (server) or sudo /usr/local/bin/k3s-agent-uninstall.sh (agent).
  5. Remove the machine from the tailnet (Tailscale admin console → Machines).
  6. Re-run the bootstrap on the remaining nodes without its IP in PRIVATE_CIDR.
  7. Purge the disks before the server leaves our hands (NIST SP 800-88): boot OVHcloud rescue mode and run nvme format --ses=1 /dev/nvmeXn1 on each NVMe (--ses=2, a cryptographic erase, where nvme id-ctrl reports support).
  8. Cancel the server in the OVH Control Panel.

Never remove servers in a way that leaves exactly two.

Replacing the single server

For moving to a bigger machine while there is only one server. Two ways:

Fresh cluster and restore (recommended). Build the new machine as a new cluster (bootstrap-node.sh, then bootstrap-argocd.sh), restore Postgres from R2 with Disaster Recovery, take and restore the OTA dumps, then switch over. The cost is a short write freeze while the final backup restores. etcd never runs with two members.

Live move. Keeps everything running, but etcd spends the move with two members, and losing either machine in that window stops the API. Keep it short.

  1. Order and install the new machine, then join it as a server (NODE_ROLE=server, K3S_URL, K3S_TOKEN, each node's PRIVATE_CIDR listing the other). Joining as an agent first doesn't work: the agent uninstall wipes /var/lib/rancher/k3s, which holds the local volumes.
  2. Move Postgres with a CNPG replica and switchover, so the primary runs on the new machine.
  3. Take the OTA dumps, then delete the Photon, Prometheus, Loki, Tempo, Redis, OTA and pgAdmin PVCs.
  4. Drain the old server; pods come back on the new machine. Restore the OTA dumps.
  5. Remove the old server (kubectl delete node, uninstall, remove from the tailnet, cancel). etcd is back to one member, on the new machine.

Either way, DNS follows on its own: the Traefik LoadBalancer Service reports the new node's IP, and external-dns republishes the records.

Hourly capacity through Terraform (not set up)

For short-lived extra capacity, OVH Public Cloud instances in BHS bill by the hour and fit Terraform: the OVH provider's ovh_cloud_project_instance creates the instance, and cloud-init runs the agent join. The costs: OVH API keys in CI, the k3s join token in Terraform state and cloud-init, and peer firewall rules that change with every instance. Worth adding once there is real burst load, such as extra API pods or CI runners.