Node Lifecycle¶
Adding, removing and replacing machines in the k3s cluster. Every node is set up the same way: order it, install the OS, run scripts/ovh/bootstrap-node.sh. Terraform manages Cloudflare only; servers are ordered by hand and bootstrapped over SSH.
Node types¶
| Type | Runs | etcd member |
|---|---|---|
| Server | Control plane (API server, scheduler, embedded etcd) plus workloads | Yes |
| Agent | Workloads only | No |
etcd needs a majority of its members up to accept writes. That makes the server count matter:
| Servers | Survives losing | Notes |
|---|---|---|
| 1 | Nothing (workloads keep running, the API stops) | Today's shape |
| 2 | Nothing, and either loss stops the API | Worse than 1. Never stay here |
| 3 | 1 server | The HA shape |
So a second machine joins as an agent, unless a third server follows right away.
Before adding any node¶
- Order a Rise server in Beauharnois (BHS), monthly billing.
- Install from the OVH template: Ubuntu Server 24.04, RAID1 across both disks,
/taking all remaining space, your ED25519 key, hostnametomoda-bhs-N, no post-install script. - Generate a one-off Tailscale auth key for the node (Server Access → Auth key per node). Keys are single-use and never stored.
- Secrets always go to a node through
/root/bootstrap.env, never on the command line (Bootstrap → Step 5). Below, "send secrets" means writing that file with the keys listed. - Note the new node's public IP and the existing nodes' public IPs. Nodes talk to each other over their public IPs, encrypted by flannel's WireGuard backend;
PRIVATE_CIDRopens the node ports to the listed peers only.
Adding an agent¶
-
Open the first server to the new node. Re-run the bootstrap there with the new peer in
PRIVATE_CIDR(comma-separated, one entry per other node):Send secrets:
GHCR_USER,GHCR_TOKEN. Then:sudo PRIVATE_CIDR=<new-node-ip>/32 NODE_ROLE=server ./bootstrap-node.shTailscale is already joined there, so no auth key is needed.
-
Read the join token on the first server:
sudo cat /var/lib/rancher/k3s/server/node-token. -
Bootstrap the new node as the image's
ubuntuuser, with the files fromscripts/ovh/copied over:Send secrets:
GHCR_USER,GHCR_TOKEN,TS_AUTHKEY(the new key),K3S_TOKEN(the join token). Then:sudo PRIVATE_CIDR=<first-server-ip>/32 K3S_URL=https://<first-server-ip>:6443 \ NODE_ROLE=agent ./bootstrap-node.shadmin-users.shruns as part of it, so the same people can log in. -
Verify:
kubectl get nodes -o wideshows the nodeReadywith labeltomoda.life/role=worker(fromscripts/ovh/k3s-agent.yaml).
Workloads have no node selectors, so new pods start landing on the agent at once. Pods with local volumes stay where their data is (see Storage moves with nodes).
Adding servers for HA¶
Go from one server straight to three.
- Bootstrap two new machines as above, but with
NODE_ROLE=serverandK3S_URL(K3S_TOKENstill goes in the secrets file). The script dropscluster-initand joins the existing etcd. - Every node's
PRIVATE_CIDRlists all the other nodes: re-run the bootstrap on each existing node with the full list. - Check etcd:
kubectl get nodesshows threecontrol-plane,etcdnodes, andkubectl get --raw '/readyz?verbose'reportsetcd ok. - Raise
instancesonpostgres-prod(k8s/envs/prod/postgres/manifests/cluster.yaml) to 2 or 3 for a streaming standby and automatic failover, and raise the HPA ceilings.
Storage moves with nodes¶
PVCs use local-path: each volume is a directory on the node where it was first scheduled, and its pod can only run there. Before draining a node, deal with each volume on it:
| Workload | Move by |
|---|---|
| Postgres (CNPG) | Add an instance (instances: +1) so a replica builds on another node, then kubectl cnpg promote <cluster> <new-instance> -n data, then lower instances again. CNPG removes the old instance's PVC |
| Photon | Nothing. A new pod downloads the index from R2 (an hour or two of geocoding outage for the planet index) |
| Prometheus | Delete the PVC; metric history starts over. Acceptable |
| Loki, Tempo | Nothing to move; data lives in R2. Delete the PVC |
| Redis | Delete the PVC; it is a cache, users may need to sign in again |
OTA databases (xavia-ota-db-{dev,prod}) |
Gap: plain Postgres with no backups. Take a pg_dump before the move and restore it after |
| pgAdmin | Delete the PVC; saved server definitions are recreated |
Deleting a PVC lets its pod reschedule on another node with a fresh volume.
Draining and removing a node¶
- Move or delete the node's volumes (table above).
-
Drain it:
kubectl cordon <node> kubectl drain <node> --ignore-daemonsets --delete-emptydir-data -
Remove it from the cluster:
kubectl delete node <node>. For a server, k3s also removes its etcd member. Checkkubectl get nodesandkubectl get --raw '/readyz?verbose'. - On the old machine, uninstall k3s:
sudo /usr/local/bin/k3s-uninstall.sh(server) orsudo /usr/local/bin/k3s-agent-uninstall.sh(agent). - Remove the machine from the tailnet (Tailscale admin console → Machines).
- Re-run the bootstrap on the remaining nodes without its IP in
PRIVATE_CIDR. - Purge the disks before the server leaves our hands (NIST SP 800-88): boot OVHcloud rescue mode and run
nvme format --ses=1 /dev/nvmeXn1on each NVMe (--ses=2, a cryptographic erase, wherenvme id-ctrlreports support). - Cancel the server in the OVH Control Panel.
Never remove servers in a way that leaves exactly two.
Replacing the single server¶
For moving to a bigger machine while there is only one server. Two ways:
Fresh cluster and restore (recommended). Build the new machine as a new cluster (bootstrap-node.sh, then bootstrap-argocd.sh), restore Postgres from R2 with Disaster Recovery, take and restore the OTA dumps, then switch over. The cost is a short write freeze while the final backup restores. etcd never runs with two members.
Live move. Keeps everything running, but etcd spends the move with two members, and losing either machine in that window stops the API. Keep it short.
- Order and install the new machine, then join it as a server (
NODE_ROLE=server,K3S_URL,K3S_TOKEN, each node'sPRIVATE_CIDRlisting the other). Joining as an agent first doesn't work: the agent uninstall wipes/var/lib/rancher/k3s, which holds the local volumes. - Move Postgres with a CNPG replica and switchover, so the primary runs on the new machine.
- Take the OTA dumps, then delete the Photon, Prometheus, Loki, Tempo, Redis, OTA and pgAdmin PVCs.
- Drain the old server; pods come back on the new machine. Restore the OTA dumps.
- Remove the old server (
kubectl delete node, uninstall, remove from the tailnet, cancel). etcd is back to one member, on the new machine.
Either way, DNS follows on its own: the Traefik LoadBalancer Service reports the new node's IP, and external-dns republishes the records.
Hourly capacity through Terraform (not set up)¶
For short-lived extra capacity, OVH Public Cloud instances in BHS bill by the hour and fit Terraform: the OVH provider's ovh_cloud_project_instance creates the instance, and cloud-init runs the agent join. The costs: OVH API keys in CI, the k3s join token in Terraform state and cloud-init, and peer firewall rules that change with every instance. Worth adding once there is real burst load, such as extra API pods or CI runners.