Skip to content

OVH bare metal (k3s)

The cluster runs on one OVH Rise bare-metal server in Beauharnois, Quebec (BHS), with k3s as the Kubernetes distribution. Argo CD reconciles everything from k8s/; storage is local NVMe (local-path) and object storage is Cloudflare R2.

OVH Beauharnois · RISE-GAME-2
k3s
tomoda api / async
CNPG Postgres local NVMe
Photon local NVMe
cloudflared
Cloudflare R2 backups, logs, media, index
GCP Secret Manager via ESO
Cloudflare DNS + proxy api / app
API and app traffic reaches Traefik on the server's public IP (prod proxied through Cloudflare with origin mTLS); data of record sits on local NVMe and streams to R2; secrets come from GCP Secret Manager.

Server

Item Value
Model RISE-GAME-2: Ryzen 7 5800X (8c/16t), 64 GB DDR4 ECC, 2× 960 GB NVMe (software RAID1)
OS Ubuntu 24.04
Disk encryption None (accepted risk, Encryption at Rest). Encrypt the data volume (LUKS2) when the node is next reprovisioned
Billing Monthly, no commitment (one-time setup fee)
Network 1 Gbps public, unmetered. Node-to-node traffic (only once a second node joins) runs over WireGuard; PRIVATE_CIDR lists the other nodes' addresses, comma-separated

k3s

Config lives in scripts/ovh/k3s-server.yaml and scripts/ovh/k3s-agent.yaml:

  • Embedded etcd (cluster-init) so more servers can join without a datastore migration; snapshots every 6h, kept 20, on local disk and in R2 (tomoda-db-backups/k3s-etcd/, credentials from the kube-system/k3s-etcd-s3 ExternalSecret).
  • wireguard-native flannel backend, encrypted node-to-node traffic.
  • Bundled Traefik disabled (Traefik comes from Argo CD). ServiceLB stays on and binds the Traefik LoadBalancer Service to the node's public IP on 80/443; external-dns publishes that IP.
  • Kubelet reserves 1 CPU / 3 GiB for the system and evicts below 1 GiB free memory.
  • Pod CIDR 10.42.0.0/16 (CNPG pg_hba allows it), service CIDR 10.43.0.0/16.
  • Storage class local-path (k3s default): PVCs are directories on the RAID1 NVMe.
  • GHCR pulls authenticate at node level via /etc/rancher/k3s/registries.yaml.

Bootstrap

# On the server (copy scripts/ovh/ over first). First login is the image's `ubuntu` user.
# Secrets (GHCR token, Tailscale key) go in /root/bootstrap.env first: see Operations → Bootstrap.
sudo ADMIN_CIDRS="<your-ip>/32" NODE_ROLE=server ./bootstrap-node.sh

# From a workstation, over Tailscale
task kube:auth
export KUBECONFIG=~/.kube/tomoda-ovh.yaml
GCPSM_KEY_FILE=./eso-reader.json ./scripts/ovh/bootstrap-argocd.sh

bootstrap-node.sh hardens SSH, creates the admin logins (admin-users.sh, below), joins Tailscale, enables unattended upgrades, sets the firewall (80/443 open; SSH and the API server only over Tailscale, plus any ADMIN_CIDRS), adds the Tailscale name and IP to the API server certificate, applies Photon's vm.max_map_count, and installs a pinned k3s. bootstrap-argocd.sh creates the Secret Manager key Secret, installs Argo CD from k8s/envs/platform/argocd/values.yaml and applies k8s/envs/bootstrap.yaml.

Server logins

scripts/ovh/admin-users.sh gives each person in its ADMIN_USERS list a login (<github-user>[:<login>]; the login defaults to the GitHub name, lowercased) with passwordless sudo through the ops group. SSH keys come from https://github.com/<user>.keys. Once at least one admin has keys, root SSH login is turned off.

Re-run it as root to pick up changed GitHub keys, add someone, or remove someone (their account is locked and their keys deleted). SSH and the k3s API are reachable only over Tailscale. Login steps, onboarding and offboarding: Server Access.

Scheduling and resources

  • No node selectors or taints: every workload can run on any node. Add soft affinity on tomoda.life/role when a second node exists.
  • Requests reflect typical use; memory limits are the safety net.
  • No CPU limits on services (API, async, Postgres, Photon, Redis, NATS, monitoring) so latency-sensitive pods never get throttled. CPU limits stay on batch work: CI runners, Trivy scans, the seed load Job.

Capacity

Sized for about 100k MAU on this one server. At about 30k DAU and 100-150 API requests/s at peak, the API, Postgres and Photon together need roughly 6-8 of the 16 threads.

Resource Allocatable Requested
CPU ~15 threads ~7 steady, ~1 more for prod HPA max and cron, ~6 for CI at max
Memory ~60 GB ~29 GB steady, ~1 GB more for HPA max and cron, ~16 GB for CI at max

CPU requests are the binding limit while real use stays near 1 CPU outside CI: the CI caps already use the spare CPU requests (budget). Memory runs out next as Photon (planet index) and Postgres grow. Past that, add a second server (below) or move to the 128 GB option.

Disk budget (about 890 GB usable)

Use Size
OS, k3s, images ~40 GB
Photon planet index + tarball during an update ~145–230 GB
Postgres prod (curated catalog plus harvested places) 10–50 GB
Monitoring, dev data, OTA, CI caches ~70 GB

The worst case is about 390 GB. Photon uses UPDATE_STRATEGY=SEQUENTIAL, so uploading a new tarball to R2 causes a short geocoding outage on the next daily poll.

Growing

  1. Second machine: joins as a k3s agent (two servers would make etcd need both up). Photon or the API can move there.
  2. HA: three servers for etcd quorum, with Postgres at instances: 2 or more for a standby and failover.
  3. Bigger boxes (RISE-M/L or Advance) for the database once queries need it.

Adding, removing and replacing nodes step by step: Node Lifecycle.