Terraform CI/CD¶
GitHub Actions plans the stacks infrastructure/cloudflare, infrastructure/gcp, infrastructure/gcp-org, infrastructure/github and infrastructure/tailscale on PRs, and applies all but gcp-org on merge. gcp-org is plan-only in CI and applied locally by an org owner.
This pipeline covers infrastructure only. Kubernetes manifests are delivered separately by Argo CD, see Argo CD. The K8s workflow here only validates manifests (kustomize build + kubeconform); it never applies them.
Workflows¶
| Workflow | File | Trigger | Does |
|---|---|---|---|
| Terraform Plan | .github/workflows/terraform-plan.yml |
PR touching infrastructure/** or a Worker source (assets-worker/, og-preview/, synthetic/) |
fmt -check + plan each stack, posts the plan as a PR comment. gcp-org is skipped with a notice until terraform-ci has its org read roles |
| Terraform Apply | .github/workflows/terraform-apply.yml |
merge to main touching the same paths, or manual dispatch (gh workflow run terraform-apply.yml) after rotating a secret it reads |
applies the cloudflare, gcp, github and tailscale stacks |
| K8s Validate | .github/workflows/k8s-validate.yml |
PR touching k8s/** |
kustomize build + kubeconform, no apply |
| Secret Scan | .github/workflows/secret-scan.yml |
PR or push to main |
Runs Gitleaks to detect committed secrets/keys/credentials |
| Infra Lint & Security | .github/workflows/infra-lint.yml |
PR to main |
TFLint, Trivy IaC scan, yamllint, ShellCheck, Worker tests |
| Images | .github/workflows/images.yml |
push to main touching an image dir, PR, or dispatch |
Builds tomoda-postgres, tomoda-arc-runner, xavia-ota to GHCR, tagged from each image dir's VERSION; existing tags are never overwritten. On PRs, checks that changed images bump VERSION and manifests reference it |
| GHCR cleanup | .github/workflows/ghcr-cleanup.yml |
weekly (Monday 03:47 UTC), or dispatch (dry run by default) | Deletes untagged versions of this repo's images; see Image retention |
| Notify prod deploy | .github/workflows/notify-prod-deploy.yml |
push to main changing the tomoda prod pin (k8s/apps/tomoda/overlays/prod/kustomization.yaml) |
Sends a prod-deployed dispatch with the tag to the tomoda repo, which publishes that release's prod OTA (see Deploy) |
Terraform Apply and the image build job run on GitHub-hosted runners, so a broken cluster never blocks an infra fix or the images needed to rebuild it, and privileged Docker builds stay off the prod server. Terraform Plan, the lint and validation jobs, GHCR cleanup and the image versions / changes jobs run on the shared in-cluster tomoda-arc pool; Notify prod deploy holds the release App key and runs on tomoda-arc-trusted (see ARC runners).
Auth: no long-lived keys¶
GCP Workload Identity Federation: GitHub's OIDC token is exchanged for short-lived credentials impersonating a service account. The github-actions pool's provider accepts only assertion.repository == tomoda-labs/devops. The pool, provider and SAs are managed by the gcp stack (infrastructure/gcp/wif.tf); see GCP and IAM.
| SA | Who can impersonate | Used by | Access |
|---|---|---|---|
terraform-ci |
Any workflow run in the repo, PR branches included | Terraform Plan | Read-only: project read roles, tfstate read, the plan input secrets |
terraform-apply |
Only terraform-apply.yml running on refs/heads/main (job_workflow_ref) |
Terraform Apply | Project IAM, secrets, SAs, WIF, APIs; tfstate write |
A plan on any branch can read GCP but change nothing there. Applies happen only from main.
Accepted residual risks:
- Plans load write credentials for Cloudflare, Tailscale and GitHub. Those providers need a token to refresh state, and the same Cloudflare API token, Tailscale OAuth client and
tomoda-terraformApp key serve plan and apply. Code on a PR branch can read them throughterraform-ciand change DNS, Access, Workers, the tailnet policy or repo settings. - Anyone who can push
maincan apply. A push tomainrunsterraform-apply.ymlasterraform-apply; the Free plan has no branch protection to stop a direct push (see Gate limitations).
CI needs GCP for two things only: the state bucket (gs://tomoda-tfstate, prefixes cloudflare/state, gcp/state, gcp-org/state, github/state and tailscale/state) and the provider secrets it reads from Secret Manager at run time (Cloudflare API token, Discord webhook, probe password, asset signing keys, the tomoda-terraform GitHub App key, the Tailscale OAuth client). The gcp and gcp-org stacks need no inputs. Nothing is stored as a GitHub secret. Repo variables: GCP_WIF_PROVIDER, GCP_TERRAFORM_CI_SA, GCP_TERRAFORM_APPLY_SA, TERRAFORM_APP_ID (and GCP_RELEASE_SA for notify-prod-deploy.yml).
The github stack authenticates with a short-lived installation token for the tomoda-terraform App, minted per run by actions/create-github-app-token from the GSM key. The stack joins the matrix only when the TERRAFORM_APP_ID variable is set; until then CI plans and applies the other stacks without it. See GitHub.
Dev/prod split¶
The cloudflare stack is a single copy; resources that differ by env are keyed by env inside it (for_each over dev/prod for the assets buckets, Workers, custom domains and CORS rules). There are no workspaces.
Applying¶
Merging a PR to main that touched infrastructure/** applies the cloudflare, gcp, github and tailscale stacks. Each run applies every stack in the matrix, not only the one the PR touched. The gate is PR review: the Plan workflow has already posted the real diff on that PR. Review it before approving.
The gcp apply runs with -parallelism=1: concurrent project IAM policy writes fail with Error 409: There were concurrent policy changes.
Gate limitations on the free plan¶
Two GitHub features this pipeline would otherwise use require a paid plan and are unavailable:
- Branch protection / required status checks: can't force the Plan job to pass before merge (403 "Upgrade to GitHub Pro or make this repository public"). Mitigation: PR review is the enforced gate; the reviewer must eyeball the posted plan.
- Environment approval gates: can't insert a human-approval pause before apply. Mitigation: PR review.
If the repo upgrades to Pro, add branch protection requiring plan (cloudflare), plan (gcp), plan (gcp-org), plan (github) and plan (tailscale), and wrap apply in an Environment with required reviewers.
Serialization¶
Plans run with -lock=false (they write nothing), so concurrent PR plans never block each other; a failed plan fails the job and its output is still posted. Applies share the GCS state lock. The Apply workflow uses a tf-apply concurrency group and max-parallel: 1 on the stack matrix so runs queue rather than collide on the lock. cancel-in-progress: false: a half-finished apply is never cancelled.
Image retention¶
GHCR keeps every version forever, and the org's private package storage is capped, so ghcr-cleanup.yml prunes tomoda-postgres, tomoda-arc-runner and xavia-ota weekly:
| Version | Kept |
|---|---|
Any tag (the image dir's VERSION) |
Always (manifests and rollbacks pin them) |
| Untagged | Only if under a day old, or a platform or attestation manifest of a tag |
Builds attach a provenance attestation, so each tag points at an image index whose platform and attestation manifests appear as untagged versions in the GitHub UI. The cleanup action resolves those children and deletes them only with their parent, so every tag stays pullable. The job summary lists the tags kept and the versions deleted.
A manual run (gh workflow run ghcr-cleanup.yml) is a dry run by default; pass -f dry_run=false to delete. Deleting needs the Admin role for this repo under each package's Package settings → Manage Actions access. A deleted version can be restored for 30 days through the GitHub REST API (restore package version); the cleanup log prints each deleted version ID. The tomoda repo prunes tomoda-backend and tomoda-frontend with its own copy of the workflow.
Image vulnerability scanning¶
Not part of this Terraform pipeline. Running images are scanned in-cluster by the Trivy Operator; see Vulnerability scanning.
Adding infra that needs a new secret or variable¶
- Non-secret var (like
project_id): set it inline in the workflow's stack-vars step. - Secret (provider token, password): put it in GCP Secret Manager and add a
gcloud secrets versions accessline to that stack's "Load … secrets" step. Never add it as a GitHub secret. See Secrets Management.