OVH bare metal (k3s)¶
The cluster runs on one OVH Rise bare-metal server in Beauharnois, Quebec (BHS), with k3s as the Kubernetes distribution. Argo CD reconciles everything from k8s/; storage is local NVMe (local-path) and object storage is Cloudflare R2.
Server¶
| Item | Value |
|---|---|
| Model | RISE-GAME-2: Ryzen 7 5800X (8c/16t), 64 GB DDR4 ECC, 2× 960 GB NVMe (software RAID1) |
| OS | Ubuntu 24.04 |
| Disk encryption | None (accepted risk, Encryption at Rest). Encrypt the data volume (LUKS2) when the node is next reprovisioned |
| Billing | Monthly, no commitment (one-time setup fee) |
| Network | 1 Gbps public, unmetered. Node-to-node traffic (only once a second node joins) runs over WireGuard; PRIVATE_CIDR lists the other nodes' addresses, comma-separated |
k3s¶
Config lives in scripts/ovh/k3s-server.yaml and scripts/ovh/k3s-agent.yaml:
- Embedded etcd (
cluster-init) so more servers can join without a datastore migration; snapshots every 6h, kept 20, on local disk and in R2 (tomoda-db-backups/k3s-etcd/, credentials from thekube-system/k3s-etcd-s3ExternalSecret). wireguard-nativeflannel backend, encrypted node-to-node traffic.- Bundled Traefik disabled (Traefik comes from Argo CD). ServiceLB stays on and binds the Traefik
LoadBalancerService to the node's public IP on 80/443; external-dns publishes that IP. - Kubelet reserves 1 CPU / 3 GiB for the system and evicts below 1 GiB free memory.
- Pod CIDR
10.42.0.0/16(CNPGpg_hbaallows it), service CIDR10.43.0.0/16. - Storage class
local-path(k3s default): PVCs are directories on the RAID1 NVMe. - GHCR pulls authenticate at node level via
/etc/rancher/k3s/registries.yaml.
Bootstrap¶
# On the server (copy scripts/ovh/ over first). First login is the image's `ubuntu` user.
# Secrets (GHCR token, Tailscale key) go in /root/bootstrap.env first: see Operations → Bootstrap.
sudo ADMIN_CIDRS="<your-ip>/32" NODE_ROLE=server ./bootstrap-node.sh
# From a workstation, over Tailscale
task kube:auth
export KUBECONFIG=~/.kube/tomoda-ovh.yaml
GCPSM_KEY_FILE=./eso-reader.json ./scripts/ovh/bootstrap-argocd.sh
bootstrap-node.sh hardens SSH, creates the admin logins (admin-users.sh, below), joins Tailscale, enables unattended upgrades, sets the firewall (80/443 open; SSH and the API server only over Tailscale, plus any ADMIN_CIDRS), adds the Tailscale name and IP to the API server certificate, applies Photon's vm.max_map_count, and installs a pinned k3s. bootstrap-argocd.sh creates the Secret Manager key Secret, installs Argo CD from k8s/envs/platform/argocd/values.yaml and applies k8s/envs/bootstrap.yaml.
Server logins¶
scripts/ovh/admin-users.sh gives each person in its ADMIN_USERS list a login (<github-user>[:<login>]; the login defaults to the GitHub name, lowercased) with passwordless sudo through the ops group. SSH keys come from https://github.com/<user>.keys. Once at least one admin has keys, root SSH login is turned off.
Re-run it as root to pick up changed GitHub keys, add someone, or remove someone (their account is locked and their keys deleted). SSH and the k3s API are reachable only over Tailscale. Login steps, onboarding and offboarding: Server Access.
Scheduling and resources¶
- No node selectors or taints: every workload can run on any node. Add soft affinity on
tomoda.life/rolewhen a second node exists. - Requests reflect typical use; memory limits are the safety net.
- No CPU limits on services (API, async, Postgres, Photon, Redis, NATS, monitoring) so latency-sensitive pods never get throttled. CPU limits stay on batch work: CI runners, Trivy scans, the seed load Job.
Capacity¶
Sized for about 100k MAU on this one server. At about 30k DAU and 100-150 API requests/s at peak, the API, Postgres and Photon together need roughly 6-8 of the 16 threads.
| Resource | Allocatable | Requested |
|---|---|---|
| CPU | ~15 threads | ~7 steady, ~1 more for prod HPA max and cron, ~6 for CI at max |
| Memory | ~60 GB | ~29 GB steady, ~1 GB more for HPA max and cron, ~16 GB for CI at max |
CPU requests are the binding limit while real use stays near 1 CPU outside CI: the CI caps already use the spare CPU requests (budget). Memory runs out next as Photon (planet index) and Postgres grow. Past that, add a second server (below) or move to the 128 GB option.
Disk budget (about 890 GB usable)¶
| Use | Size |
|---|---|
| OS, k3s, images | ~40 GB |
| Photon planet index + tarball during an update | ~145–230 GB |
| Postgres prod (curated catalog plus harvested places) | 10–50 GB |
| Monitoring, dev data, OTA, CI caches | ~70 GB |
The worst case is about 390 GB. Photon uses UPDATE_STRATEGY=SEQUENTIAL, so uploading a new tarball to R2 causes a short geocoding outage on the next daily poll.
Growing¶
- Second machine: joins as a k3s agent (two servers would make etcd need both up). Photon or the API can move there.
- HA: three servers for etcd quorum, with Postgres at
instances: 2or more for a standby and failover. - Bigger boxes (RISE-M/L or Advance) for the database once queries need it.
Adding, removing and replacing nodes step by step: Node Lifecycle.
Related¶
- Cloudflare (Tunnel, R2)
- Secrets management
- Decisions