Bootstrap¶
One-time steps to bring the platform up from nothing: the pieces Terraform and Argo CD depend on, then the server, then GitOps. A rebuild after losing the server repeats only Steps 5 and 6.
What lives outside Terraform¶
| Resource | Why |
|---|---|
gs://tomoda-tfstate |
Holds Terraform state itself (chicken-and-egg) |
| GCP Secret Manager values | Secret values never live in git or state; the secret names are Terraform in infrastructure/gcp/ |
| The OVH server | Ordered in the OVH console (monthly billing); provisioned by scripts/ovh/ |
gcpsm-credentials and repo-devops Secrets |
The two Secrets created by hand; everything else comes from ESO |
Everything else is Terraform: Cloudflare DNS, Workers, Tunnel, Access and R2 buckets in infrastructure/cloudflare/; GCP secrets, service accounts, WIF, buckets and APIs in infrastructure/gcp/ (GCP). The Photon index and seed buckets carry prevent_destroy, and R2 refuses to delete non-empty buckets.
Step 1: Terraform state bucket¶
Already exists. To recreate it:
gcloud storage buckets create gs://tomoda-tfstate \
--project=development-485000 --location=northamerica-northeast1 \
--uniform-bucket-level-access
gcloud storage buckets update gs://tomoda-tfstate --versioning
Each stack uses its own prefix (cloudflare/state, gcp/state, github/state, tailscale/state). Then bootstrap the gcp stack (GCP → Bootstrap); it adopts the bucket and adds the lifecycle rule.
Step 2: Secrets in GCP Secret Manager¶
./scripts/setup-gcp-secrets.sh # generates the auto values (and creates any name Terraform has not)
Then set the values that come from outside (full inventory in Secrets management):
- R2 tokens:
tomoda-r2-cluster-credentials,tomoda-r2-app-{dev,prod}-credentials(JSON{"access_key_id","secret_access_key"}). - Asset signing keys:
tomoda-asset-signing-key-{dev,prod}(openssl rand -base64 48). Shared by the backend and the assets Workers. tomoda-ghcr-read-credentials(<user>:<read:packages PAT>).tomoda-cloudflare-api-token, with the permissions listed in Cloudflare.
The ESO reader service account and its secretAccessor role come from the gcp stack (infrastructure/gcp/service_accounts.tf); its key is created by hand in Step 6.
Step 3: GitHub App for repo access + ARC runners¶
Required. This single GitHub App serves two purposes: it is the credential Argo CD uses to clone this private repo (Step 6), and it authenticates the self-hosted ARC runner pool in k8s/envs/platform/arc-*/. Apps are scoped, rotatable, and don't tie access to a single human's GitHub account.
Argo CD needs this App even without ARC runners: it can't sync without repo read access.
- Go to https://github.com/organizations/tomoda-labs/settings/apps → New GitHub App.
- Fill in:
- GitHub App name:
tomoda-ci-automation - Homepage URL:
https://github.com/tomoda-labs/devops - Webhook URL: disable webhooks (uncheck "Active")
- GitHub App name:
- Permissions: Repository permissions:
- Contents: Read-only (Argo CD clones the repo with this)
- Actions: Read and write
- Administration: Read and write
- Checks: Read
- Metadata: Read (auto-selected)
- Where can this GitHub App be installed? → Only on this account.
- Click Create GitHub App.
- On the next page, note the App ID (top of the page, ~6 digits).
- Click Generate a private key → downloads
tomoda-ci-automation.<date>.private-key.pem. Treat this file like a password. - Left sidebar → Install App → install on the
tomoda-labsorg → choose "All repositories", or "Only select repositories" and includedevops(Argo can't clone a repo outside the installation's scope). - After install, the URL is
https://github.com/organizations/tomoda-labs/settings/installations/<installation-id>. Note the Installation ID (the numeric path segment).
Push all three to GCP SM:
PROJECT_ID=development-485000
# The secrets exist (infrastructure/gcp/secrets.tf); add values.
# App ID and Installation ID: paste each, then Ctrl-D.
gcloud secrets versions add tomoda-github-app-id --project="${PROJECT_ID}" --data-file=-
gcloud secrets versions add tomoda-github-app-installation-id --project="${PROJECT_ID}" --data-file=-
# Private key: the whole PEM file from step 7, then delete the local copy.
gcloud secrets versions add tomoda-github-app-private-key --project="${PROJECT_ID}" \
--data-file=tomoda-ci-automation.<date>.private-key.pem
# Verify
gcloud secrets list --project="${PROJECT_ID}" --filter="name~tomoda-github-app"
# Expected: tomoda-github-app-id, tomoda-github-app-installation-id, tomoda-github-app-private-key
After uploading, delete the local PEM file; the only copy now lives in GCP SM:
shred -u tomoda-ci-automation.<date>.private-key.pem # Linux
rm -P tomoda-ci-automation.<date>.private-key.pem # macOS
If you rotate the App's private key in the GitHub UI later, push the new PEM via gcloud secrets versions add tomoda-github-app-private-key --data-file=…, then restart the ARC controller + listener pods to pick up the new key without waiting for ESO's 1h refresh:
kubectl rollout restart deploy/arc-controller-gha-rs-controller -n arc-systems
kubectl delete pod -n arc-runners -l app.kubernetes.io/component=runner-set-listener
Step 4: Cloudflare Terraform and data¶
Merge the change to main; terraform-apply.yml applies the cloudflare stack (R2 buckets, Workers, custom domains). The R2 data (Photon index, seed sets, email logos) survives a server rebuild. Into empty buckets, build the index with Photon indexing and publish seed sets with Location seed load.
Step 5: Server¶
Before you start: the tailnet and its policy (Server Access → Tailnet policy), and tomoda-ghcr-read-credentials filled in Secret Manager.
- Order and install. RISE-GAME-2 (64 GB, 2× 960 GB NVMe) in Beauharnois. In the OVH Control Panel install Ubuntu Server 24.04 from the OVHcloud template: RAID1,
/takes all remaining space, hostnametomoda-bhs-1, your ED25519 public key, everything else empty. -
Check the box (first login is the image's
ubuntuuser):ssh ubuntu@<server-ip> 'lsblk; cat /proc/mdstat; df -h /; nproc; free -g'Expect both RAID arrays
[UU]and about 880 GB free on/. -
Generate a Tailscale auth key and copy it (Server Access → Auth key per node). It is not stored anywhere.
-
Copy the scripts, send the secrets, run the bootstrap. Secrets travel over SSH stdin into
/root/bootstrap.env(mode 600), whichbootstrap-node.shreads and deletes on exit. They never appear in shell history or in the server's process list; never pass them assudo VAR=...arguments, whichpsshows.ADMIN_CIDRSkeeps your current IP open as a fallback in case Tailscale fails to join. The run takes a few minutes; it detaches and logs to/root/bootstrap.log.scp -r scripts/ovh ubuntu@<server-ip>: G=$(gcloud secrets versions access latest --secret=tomoda-ghcr-read-credentials --project=development-485000) printf 'GHCR_USER=%q\nGHCR_TOKEN=%q\nTS_AUTHKEY=%q\n' "${G%%:*}" "${G#*:}" "$(pbpaste)" \ | ssh ubuntu@<server-ip> 'sudo install -m 600 /dev/stdin /root/bootstrap.env' unset G; pbcopy </dev/null ssh ubuntu@<server-ip> 'cd ovh && sudo ADMIN_CIDRS=<your-ip>/32 NODE_ROLE=server \ setsid nohup bash -c "./bootstrap-node.sh >/root/bootstrap.log 2>&1" </dev/null >/dev/null 2>&1 &' ssh ubuntu@<server-ip> 'sudo tail -f /root/bootstrap.log' # Ctrl-C once it prints "k3s server installed" -
Verify from your laptop with Tailscale connected:
tailscale status | grep tomoda-bhs-1 ssh <your-user>@tomoda-bhs-1 'sudo k3s kubectl get nodes; sudo ufw status' ssh root@tomoda-bhs-1 true # should fail: root login is off task kube:auth export KUBECONFIG=~/.kube/tomoda-ovh.yaml kubectl get nodes -
Close the fallback once Tailscale works: send the GHCR secrets again (step 4, with an empty
TS_AUTHKEY) and re-run the bootstrap withoutADMIN_CIDRS. From here on use your own login over Tailscale (<login>@tomoda-bhs-1): the bootstrap locks the image'subuntuuser once a listed admin has keys.
Step 6: Argo CD and the app-of-apps¶
-
Make sure the devops-owned images exist on GHCR.
images.ymlbuilds an image only when its directory changes onmain, so after a fresh clone or the first merge some may be missing. CNPG cannot start withouttomoda-postgres, the CI runners needtomoda-arc-runner, and the OTA servers needxavia-ota:task images:list -- tomoda-postgres # repeat for tomoda-arc-runner, xavia-ota task images:build -- tomoda-postgres # only for the ones that are missing -
Install Argo CD:
gcloud iam service-accounts keys create eso-reader.json \ --iam-account=eso-reader@development-485000.iam.gserviceaccount.com GCPSM_KEY_FILE=./eso-reader.json ./scripts/ovh/bootstrap-argocd.sh rm -P eso-reader.json # the cluster Secret is now the only copy you need
The script creates external-secrets/gcpsm-credentials, installs Argo CD with k8s/envs/platform/argocd/values.yaml, creates the repo-devops repo credential from the GitHub App keys in GCP SM, and applies k8s/envs/bootstrap.yaml. From then on Argo CD manages its own release through the argocd Application, so the script is not re-run for upgrades. Reconciliation takes about 10 minutes, longer on first boot while Photon downloads the planet index (kubectl get applications -n argocd -w).
Verification checklist¶
-
kubectl get nodesshows the serverReady -
kubectl get applications -n argocdshows every appSynced/Healthy -
kubectl get externalsecret -AallSecretSynced -
kubectl get cluster -n datashowspostgres-prodhealthy and a first backup intomoda-db-backups -
curl -sI https://api.tomoda.life/healthreturns 200 - A private media URL from the app loads; the same URL with a changed
sigreturns 403