Server Access¶
How developers reach the OVH server and the k3s cluster: who has access, how to log in, how to connect kubectl, and how access is granted and removed.
How it works¶
| Layer | Mechanism | Managed in |
|---|---|---|
| Network | Tailscale. SSH (22) and the k3s API (6443) are open only on the tailscale0 interface; the public firewall allows just 80/443 |
Tailnet policy in infrastructure/tailscale/; members in the admin console; firewall in scripts/ovh/bootstrap-node.sh |
| Login | One Linux account per person, passwordless sudo via the ops group, SSH keys pulled from GitHub. Root SSH login is off, and the image's default ubuntu user is locked after the first bootstrap |
ADMIN_USERS in scripts/ovh/admin-users.sh |
| Cluster | The k3s admin kubeconfig, fetched over SSH | task kube:auth |
Getting in takes all three: a Tailscale account in the tailnet, a login in ADMIN_USERS with a key on GitHub, and then the kubeconfig.
Who has access¶
Each entry in ADMIN_USERS (scripts/ovh/admin-users.sh) is <github-user>[:<login>]: the GitHub account whose keys are pulled, and an optional server login that defaults to the GitHub name, lowercased. That list is the source of truth for who has server access.
First-time setup (each developer)¶
- SSH key on GitHub. Add your public key under GitHub → Settings → SSH and GPG keys. Check it is published:
curl https://github.com/<github-user>.keys. If you had no key when the server was set up, ask an admin to re-runsudo ./admin-users.shafter you add one. - Accept the tailnet invite. An admin invites your
tomoda.lifeGoogle account from the Tailscale admin console; accept it from the email. -
Install Tailscale and log in with the same Google account. Pick one:
Platform Install Log in macOS (menu-bar app, recommended) brew install --cask tailscale, or the App Store / standalone downloadOpen Tailscale from Applications, then Log in in the menu bar macOS (CLI only, no app) brew install tailscalethensudo brew services start tailscalesudo tailscale upand open the printed linkLinux curl -fsSL https://tailscale.com/install.sh \| shsudo tailscale upand open the printed linkWindows Installer Tray icon → Log in The commands below use the
tailscaleCLI. The macOS app doesn't put it on yourPATHby default: turn it on in the app (Settings → CLI integration), or addalias tailscale=/Applications/Tailscale.app/Contents/MacOS/Tailscaleto your shell profile. -
Keep MagicDNS on: in the Tailscale app, "Use Tailscale DNS settings" must be checked (CLI:
tailscale set --accept-dns=true), sotomoda-bhs-1resolves. -
Check you can see the server:
tailscale status | grep tomoda-bhs-1 # 100.85.88.103 tomoda-bhs-1 ... linux tailscale ping tomoda-bhs-1
The tailnet is taile95de2.ts.net, so the full name is tomoda-bhs-1.taile95de2.ts.net; the short name works when MagicDNS is on.
Log in¶
ssh <login>@tomoda-bhs-1 # first time: accept the host key
sudo -i # root shell when needed
Optional ~/.ssh/config entry:
Host tomoda-bhs-1
User <login>
IdentityFile ~/.ssh/id_ed25519
tomoda-bhs-1 resolves through Tailscale MagicDNS. If it doesn't, use the server's Tailscale IP from tailscale status.
Connect to the cluster¶
task kube:auth # writes ~/.kube/tomoda-ovh.yaml over Tailscale
task kube:status # or plain kubectl with KUBECONFIG=~/.kube/tomoda-ovh.yaml
Tasks log in as your GitHub username, lowercased. If your server login differs, set it once: export TOMODA_SSH_USER=<login>. task setup:doctor checks the whole chain. All shortcuts: Task Catalog.
The kubeconfig points at https://tomoda-bhs-1:6443, which only works while Tailscale is connected. It is cluster-admin: keep the file at mode 0600 and never commit or paste it.
Granting access¶
- Invite the person to the tailnet (Tailscale admin console → Users → Invite).
- Add
<github-user>[:<login>]toADMIN_USERSinscripts/ovh/admin-users.sh, update the table above, and merge. - Run
task server:users(copies the script over and applies it).
Key changes need only step 3: the person updates their keys on GitHub, then someone runs task server:users.
Removing access¶
- Remove the person from the tailnet. This alone cuts their network path.
- Remove them from
ADMIN_USERS, merge, and runtask server:users: their account is locked and their keys deleted. - If they had the kubeconfig, rotate the k3s client certificates:
sudo k3s certificate rotateandsudo systemctl restart k3s, then everyone re-runstask kube:auth.
Tailnet policy¶
The access policy is code: infrastructure/tailscale/policy.hujson, applied by the infrastructure/tailscale Terraform stack (state prefix tailscale/state). Change it through a PR, never in the admin console's Access controls editor: a console edit is reverted by the next apply.
The policy lets members reach only the server's admin ports:
"tagOwners": { "tag:server": ["autogroup:admin"] },
"grants": [
{ "src": ["autogroup:member"], "dst": ["tag:server"], "ip": ["tcp:22", "tcp:6443"] }
],
There is no ssh section; logins use normal SSH keys.
| Managed in Terraform | Not managed |
|---|---|
Policy (tailscale_acl), DNS (tailscale_dns_configuration: MagicDNS on, no custom nameservers), tailnet settings (tailscale_tailnet_settings) |
Users and invites (admin console), auth keys (see below), device tags (set by the auth key at join) |
CI posts a plan on every PR that touches infrastructure/tailscale/. Merging applies it. To check a change locally before opening the PR:
cd infrastructure/tailscale
export TAILSCALE_OAUTH_CLIENT_ID=$(gcloud secrets versions access latest --secret=tomoda-tailscale-oauth-client-id --project=development-485000)
export TAILSCALE_OAUTH_CLIENT_SECRET=$(gcloud secrets versions access latest --secret=tomoda-tailscale-oauth-client-secret --project=development-485000)
terraform init && terraform plan -lock=false
Bootstrap (once)¶
- Create the tailnet with Google sign-in on
tomoda.life. - The secret containers
tomoda-tailscale-oauth-client-idand-secretare ininfrastructure/gcp/secrets.tf; the gcp stack creates them on merge. Don'tgcloud secrets createthem by hand. Until they hold a value, CI skips the tailscale plan with a notice. - Admin console → Settings → OAuth clients: create a client with scopes
policy_file:write,dns:writeandfeature_settings:write. - Set each value from stdin:
pbpaste | gcloud secrets versions add <name> --project=development-485000 --data-file=-. - Re-run the plan (CI or the commands above). If
tailscale_acl.thisshows an update, reconcilepolicy.hujsonwith the live policy until the plan is 3 imports, 0 changes. - The first apply on merge imports the three resources; check the plan on the PR shows only those imports and expected policy changes.
Auth key per node¶
Each node joins the tailnet with its own auth key, generated right before its bootstrap and never stored. Auth keys stay out of Terraform: a tailscale_tailnet_key would keep the key in state.
- Tailscale admin console → Settings → Keys → Generate auth key: not reusable, not ephemeral, pre-approved, tag
tag:server, expiry 1 day. - Copy it. The bootstrap command reads it from the clipboard (Bootstrap → Step 5) and clears it.
The key is spent once the node joins; tagged nodes stay in the tailnet without expiring.
Break-glass¶
- Tailscale down or the server dropped off the tailnet: OVH Control Panel → the server → IPMI / KVM gives a remote console. Log in as your user (set a password first with
sudo passwd <login>while you still have access, or use rescue mode). - Public IP fallback: re-running
bootstrap-node.shwithADMIN_CIDRS=<your-ip>/32opens SSH and 6443 to that IP. Remove it again afterwards.