Skip to content

Bootstrap

One-time steps to bring the platform up from nothing: the pieces Terraform and Argo CD depend on, then the server, then GitOps. A rebuild after losing the server repeats only Steps 5 and 6.

What lives outside Terraform

Resource Why
gs://tomoda-tfstate Holds Terraform state itself (chicken-and-egg)
GCP Secret Manager values Secret values never live in git or state; the secret names are Terraform in infrastructure/gcp/
The OVH server Ordered in the OVH console (monthly billing); provisioned by scripts/ovh/
gcpsm-credentials and repo-devops Secrets The two Secrets created by hand; everything else comes from ESO

Everything else is Terraform: Cloudflare DNS, Workers, Tunnel, Access and R2 buckets in infrastructure/cloudflare/; GCP secrets, service accounts, WIF, buckets and APIs in infrastructure/gcp/ (GCP). The Photon index and seed buckets carry prevent_destroy, and R2 refuses to delete non-empty buckets.

Step 1: Terraform state bucket

Already exists. To recreate it:

gcloud storage buckets create gs://tomoda-tfstate \
  --project=development-485000 --location=northamerica-northeast1 \
  --uniform-bucket-level-access
gcloud storage buckets update gs://tomoda-tfstate --versioning

Each stack uses its own prefix (cloudflare/state, gcp/state, github/state, tailscale/state). Then bootstrap the gcp stack (GCP → Bootstrap); it adopts the bucket and adds the lifecycle rule.

Step 2: Secrets in GCP Secret Manager

./scripts/setup-gcp-secrets.sh        # generates the auto values (and creates any name Terraform has not)

Then set the values that come from outside (full inventory in Secrets management):

  • R2 tokens: tomoda-r2-cluster-credentials, tomoda-r2-app-{dev,prod}-credentials (JSON {"access_key_id","secret_access_key"}).
  • Asset signing keys: tomoda-asset-signing-key-{dev,prod} (openssl rand -base64 48). Shared by the backend and the assets Workers.
  • tomoda-ghcr-read-credentials (<user>:<read:packages PAT>).
  • tomoda-cloudflare-api-token, with the permissions listed in Cloudflare.

The ESO reader service account and its secretAccessor role come from the gcp stack (infrastructure/gcp/service_accounts.tf); its key is created by hand in Step 6.

Step 3: GitHub App for repo access + ARC runners

Required. This single GitHub App serves two purposes: it is the credential Argo CD uses to clone this private repo (Step 6), and it authenticates the self-hosted ARC runner pool in k8s/envs/platform/arc-*/. Apps are scoped, rotatable, and don't tie access to a single human's GitHub account.

Argo CD needs this App even without ARC runners: it can't sync without repo read access.

  1. Go to https://github.com/organizations/tomoda-labs/settings/apps → New GitHub App.
  2. Fill in:
    • GitHub App name: tomoda-ci-automation
    • Homepage URL: https://github.com/tomoda-labs/devops
    • Webhook URL: disable webhooks (uncheck "Active")
  3. Permissions: Repository permissions:
    • Contents: Read-only (Argo CD clones the repo with this)
    • Actions: Read and write
    • Administration: Read and write
    • Checks: Read
    • Metadata: Read (auto-selected)
  4. Where can this GitHub App be installed? → Only on this account.
  5. Click Create GitHub App.
  6. On the next page, note the App ID (top of the page, ~6 digits).
  7. Click Generate a private key → downloads tomoda-ci-automation.<date>.private-key.pem. Treat this file like a password.
  8. Left sidebar → Install App → install on the tomoda-labs org → choose "All repositories", or "Only select repositories" and include devops (Argo can't clone a repo outside the installation's scope).
  9. After install, the URL is https://github.com/organizations/tomoda-labs/settings/installations/<installation-id>. Note the Installation ID (the numeric path segment).

Push all three to GCP SM:

PROJECT_ID=development-485000

# The secrets exist (infrastructure/gcp/secrets.tf); add values.
# App ID and Installation ID: paste each, then Ctrl-D.
gcloud secrets versions add tomoda-github-app-id --project="${PROJECT_ID}" --data-file=-
gcloud secrets versions add tomoda-github-app-installation-id --project="${PROJECT_ID}" --data-file=-

# Private key: the whole PEM file from step 7, then delete the local copy.
gcloud secrets versions add tomoda-github-app-private-key --project="${PROJECT_ID}" \
  --data-file=tomoda-ci-automation.<date>.private-key.pem

# Verify
gcloud secrets list --project="${PROJECT_ID}" --filter="name~tomoda-github-app"
# Expected: tomoda-github-app-id, tomoda-github-app-installation-id, tomoda-github-app-private-key

After uploading, delete the local PEM file; the only copy now lives in GCP SM:

shred -u tomoda-ci-automation.<date>.private-key.pem   # Linux
rm -P tomoda-ci-automation.<date>.private-key.pem      # macOS

If you rotate the App's private key in the GitHub UI later, push the new PEM via gcloud secrets versions add tomoda-github-app-private-key --data-file=…, then restart the ARC controller + listener pods to pick up the new key without waiting for ESO's 1h refresh:

kubectl rollout restart deploy/arc-controller-gha-rs-controller -n arc-systems
kubectl delete pod -n arc-runners -l app.kubernetes.io/component=runner-set-listener

Step 4: Cloudflare Terraform and data

Merge the change to main; terraform-apply.yml applies the cloudflare stack (R2 buckets, Workers, custom domains). The R2 data (Photon index, seed sets, email logos) survives a server rebuild. Into empty buckets, build the index with Photon indexing and publish seed sets with Location seed load.

Step 5: Server

Before you start: the tailnet and its policy (Server Access → Tailnet policy), and tomoda-ghcr-read-credentials filled in Secret Manager.

  1. Order and install. RISE-GAME-2 (64 GB, 2× 960 GB NVMe) in Beauharnois. In the OVH Control Panel install Ubuntu Server 24.04 from the OVHcloud template: RAID1, / takes all remaining space, hostname tomoda-bhs-1, your ED25519 public key, everything else empty.
  2. Check the box (first login is the image's ubuntu user):

    ssh ubuntu@<server-ip> 'lsblk; cat /proc/mdstat; df -h /; nproc; free -g'
    

    Expect both RAID arrays [UU] and about 880 GB free on /.

  3. Generate a Tailscale auth key and copy it (Server Access → Auth key per node). It is not stored anywhere.

  4. Copy the scripts, send the secrets, run the bootstrap. Secrets travel over SSH stdin into /root/bootstrap.env (mode 600), which bootstrap-node.sh reads and deletes on exit. They never appear in shell history or in the server's process list; never pass them as sudo VAR=... arguments, which ps shows. ADMIN_CIDRS keeps your current IP open as a fallback in case Tailscale fails to join. The run takes a few minutes; it detaches and logs to /root/bootstrap.log.

    scp -r scripts/ovh ubuntu@<server-ip>:
    G=$(gcloud secrets versions access latest --secret=tomoda-ghcr-read-credentials --project=development-485000)
    printf 'GHCR_USER=%q\nGHCR_TOKEN=%q\nTS_AUTHKEY=%q\n' "${G%%:*}" "${G#*:}" "$(pbpaste)" \
      | ssh ubuntu@<server-ip> 'sudo install -m 600 /dev/stdin /root/bootstrap.env'
    unset G; pbcopy </dev/null
    ssh ubuntu@<server-ip> 'cd ovh && sudo ADMIN_CIDRS=<your-ip>/32 NODE_ROLE=server \
      setsid nohup bash -c "./bootstrap-node.sh >/root/bootstrap.log 2>&1" </dev/null >/dev/null 2>&1 &'
    ssh ubuntu@<server-ip> 'sudo tail -f /root/bootstrap.log'   # Ctrl-C once it prints "k3s server installed"
    
  5. Verify from your laptop with Tailscale connected:

    tailscale status | grep tomoda-bhs-1
    ssh <your-user>@tomoda-bhs-1 'sudo k3s kubectl get nodes; sudo ufw status'
    ssh root@tomoda-bhs-1 true      # should fail: root login is off
    task kube:auth
    export KUBECONFIG=~/.kube/tomoda-ovh.yaml
    kubectl get nodes
    
  6. Close the fallback once Tailscale works: send the GHCR secrets again (step 4, with an empty TS_AUTHKEY) and re-run the bootstrap without ADMIN_CIDRS. From here on use your own login over Tailscale (<login>@tomoda-bhs-1): the bootstrap locks the image's ubuntu user once a listed admin has keys.

Step 6: Argo CD and the app-of-apps

  1. Make sure the devops-owned images exist on GHCR. images.yml builds an image only when its directory changes on main, so after a fresh clone or the first merge some may be missing. CNPG cannot start without tomoda-postgres, the CI runners need tomoda-arc-runner, and the OTA servers need xavia-ota:

    task images:list -- tomoda-postgres      # repeat for tomoda-arc-runner, xavia-ota
    task images:build -- tomoda-postgres     # only for the ones that are missing
    
  2. Install Argo CD:

    gcloud iam service-accounts keys create eso-reader.json \
      --iam-account=eso-reader@development-485000.iam.gserviceaccount.com
    GCPSM_KEY_FILE=./eso-reader.json ./scripts/ovh/bootstrap-argocd.sh
    rm -P eso-reader.json   # the cluster Secret is now the only copy you need
    

The script creates external-secrets/gcpsm-credentials, installs Argo CD with k8s/envs/platform/argocd/values.yaml, creates the repo-devops repo credential from the GitHub App keys in GCP SM, and applies k8s/envs/bootstrap.yaml. From then on Argo CD manages its own release through the argocd Application, so the script is not re-run for upgrades. Reconciliation takes about 10 minutes, longer on first boot while Photon downloads the planet index (kubectl get applications -n argocd -w).

Verification checklist

  • kubectl get nodes shows the server Ready
  • kubectl get applications -n argocd shows every app Synced / Healthy
  • kubectl get externalsecret -A all SecretSynced
  • kubectl get cluster -n data shows postgres-prod healthy and a first backup in tomoda-db-backups
  • curl -sI https://api.tomoda.life/health returns 200
  • A private media URL from the app loads; the same URL with a changed sig returns 403