Skip to content

Server Access

How developers reach the OVH server and the k3s cluster: who has access, how to log in, how to connect kubectl, and how access is granted and removed.

How it works

Layer Mechanism Managed in
Network Tailscale. SSH (22) and the k3s API (6443) are open only on the tailscale0 interface; the public firewall allows just 80/443 Tailnet policy in infrastructure/tailscale/; members in the admin console; firewall in scripts/ovh/bootstrap-node.sh
Login One Linux account per person, passwordless sudo via the ops group, SSH keys pulled from GitHub. Root SSH login is off, and the image's default ubuntu user is locked after the first bootstrap ADMIN_USERS in scripts/ovh/admin-users.sh
Cluster The k3s admin kubeconfig, fetched over SSH task kube:auth

Getting in takes all three: a Tailscale account in the tailnet, a login in ADMIN_USERS with a key on GitHub, and then the kubeconfig.

Who has access

Each entry in ADMIN_USERS (scripts/ovh/admin-users.sh) is <github-user>[:<login>]: the GitHub account whose keys are pulled, and an optional server login that defaults to the GitHub name, lowercased. That list is the source of truth for who has server access.

First-time setup (each developer)

  1. SSH key on GitHub. Add your public key under GitHub → Settings → SSH and GPG keys. Check it is published: curl https://github.com/<github-user>.keys. If you had no key when the server was set up, ask an admin to re-run sudo ./admin-users.sh after you add one.
  2. Accept the tailnet invite. An admin invites your tomoda.life Google account from the Tailscale admin console; accept it from the email.
  3. Install Tailscale and log in with the same Google account. Pick one:

    Platform Install Log in
    macOS (menu-bar app, recommended) brew install --cask tailscale, or the App Store / standalone download Open Tailscale from Applications, then Log in in the menu bar
    macOS (CLI only, no app) brew install tailscale then sudo brew services start tailscale sudo tailscale up and open the printed link
    Linux curl -fsSL https://tailscale.com/install.sh \| sh sudo tailscale up and open the printed link
    Windows Installer Tray icon → Log in

    The commands below use the tailscale CLI. The macOS app doesn't put it on your PATH by default: turn it on in the app (Settings → CLI integration), or add alias tailscale=/Applications/Tailscale.app/Contents/MacOS/Tailscale to your shell profile.

  4. Keep MagicDNS on: in the Tailscale app, "Use Tailscale DNS settings" must be checked (CLI: tailscale set --accept-dns=true), so tomoda-bhs-1 resolves.

  5. Check you can see the server:

    tailscale status | grep tomoda-bhs-1      # 100.85.88.103  tomoda-bhs-1  ... linux
    tailscale ping tomoda-bhs-1
    

The tailnet is taile95de2.ts.net, so the full name is tomoda-bhs-1.taile95de2.ts.net; the short name works when MagicDNS is on.

Log in

ssh <login>@tomoda-bhs-1      # first time: accept the host key
sudo -i                       # root shell when needed

Optional ~/.ssh/config entry:

Host tomoda-bhs-1
  User <login>
  IdentityFile ~/.ssh/id_ed25519

tomoda-bhs-1 resolves through Tailscale MagicDNS. If it doesn't, use the server's Tailscale IP from tailscale status.

Connect to the cluster

task kube:auth            # writes ~/.kube/tomoda-ovh.yaml over Tailscale
task kube:status          # or plain kubectl with KUBECONFIG=~/.kube/tomoda-ovh.yaml

Tasks log in as your GitHub username, lowercased. If your server login differs, set it once: export TOMODA_SSH_USER=<login>. task setup:doctor checks the whole chain. All shortcuts: Task Catalog.

The kubeconfig points at https://tomoda-bhs-1:6443, which only works while Tailscale is connected. It is cluster-admin: keep the file at mode 0600 and never commit or paste it.

Granting access

  1. Invite the person to the tailnet (Tailscale admin console → Users → Invite).
  2. Add <github-user>[:<login>] to ADMIN_USERS in scripts/ovh/admin-users.sh, update the table above, and merge.
  3. Run task server:users (copies the script over and applies it).

Key changes need only step 3: the person updates their keys on GitHub, then someone runs task server:users.

Removing access

  1. Remove the person from the tailnet. This alone cuts their network path.
  2. Remove them from ADMIN_USERS, merge, and run task server:users: their account is locked and their keys deleted.
  3. If they had the kubeconfig, rotate the k3s client certificates: sudo k3s certificate rotate and sudo systemctl restart k3s, then everyone re-runs task kube:auth.

Tailnet policy

The access policy is code: infrastructure/tailscale/policy.hujson, applied by the infrastructure/tailscale Terraform stack (state prefix tailscale/state). Change it through a PR, never in the admin console's Access controls editor: a console edit is reverted by the next apply.

The policy lets members reach only the server's admin ports:

"tagOwners": { "tag:server": ["autogroup:admin"] },
"grants": [
  { "src": ["autogroup:member"], "dst": ["tag:server"], "ip": ["tcp:22", "tcp:6443"] }
],

There is no ssh section; logins use normal SSH keys.

Managed in Terraform Not managed
Policy (tailscale_acl), DNS (tailscale_dns_configuration: MagicDNS on, no custom nameservers), tailnet settings (tailscale_tailnet_settings) Users and invites (admin console), auth keys (see below), device tags (set by the auth key at join)

CI posts a plan on every PR that touches infrastructure/tailscale/. Merging applies it. To check a change locally before opening the PR:

cd infrastructure/tailscale
export TAILSCALE_OAUTH_CLIENT_ID=$(gcloud secrets versions access latest --secret=tomoda-tailscale-oauth-client-id --project=development-485000)
export TAILSCALE_OAUTH_CLIENT_SECRET=$(gcloud secrets versions access latest --secret=tomoda-tailscale-oauth-client-secret --project=development-485000)
terraform init && terraform plan -lock=false

Bootstrap (once)

  1. Create the tailnet with Google sign-in on tomoda.life.
  2. The secret containers tomoda-tailscale-oauth-client-id and -secret are in infrastructure/gcp/secrets.tf; the gcp stack creates them on merge. Don't gcloud secrets create them by hand. Until they hold a value, CI skips the tailscale plan with a notice.
  3. Admin console → Settings → OAuth clients: create a client with scopes policy_file:write, dns:write and feature_settings:write.
  4. Set each value from stdin: pbpaste | gcloud secrets versions add <name> --project=development-485000 --data-file=-.
  5. Re-run the plan (CI or the commands above). If tailscale_acl.this shows an update, reconcile policy.hujson with the live policy until the plan is 3 imports, 0 changes.
  6. The first apply on merge imports the three resources; check the plan on the PR shows only those imports and expected policy changes.

Auth key per node

Each node joins the tailnet with its own auth key, generated right before its bootstrap and never stored. Auth keys stay out of Terraform: a tailscale_tailnet_key would keep the key in state.

  1. Tailscale admin console → Settings → Keys → Generate auth key: not reusable, not ephemeral, pre-approved, tag tag:server, expiry 1 day.
  2. Copy it. The bootstrap command reads it from the clipboard (Bootstrap → Step 5) and clears it.

The key is spent once the node joins; tagged nodes stay in the tailnet without expiring.

Break-glass

  • Tailscale down or the server dropped off the tailnet: OVH Control Panel → the server → IPMI / KVM gives a remote console. Log in as your user (set a password first with sudo passwd <login> while you still have access, or use rescue mode).
  • Public IP fallback: re-running bootstrap-node.sh with ADMIN_CIDRS=<your-ip>/32 opens SSH and 6443 to that IP. Remove it again afterwards.