Skip to content

ARC Self-Hosted Runners

tomoda-ci-automation GitHub App avatar

GitHub Actions runners on the k3s cluster instead of GitHub-hosted. Saves the free-plan Actions-minutes limit and gives CI the same cluster + network access as the other workloads.

Powered by Actions Runner Controller (ARC). Runner pods are ephemeral: spawn on a queued job, register with GitHub, run, terminate. Scales to 0 between jobs.

Architecture

GitHub Actions queue
        │
        │  poll for jobs
        ▼
┌─────────────────────────────┐
│ ARC controller              │   (arc-systems namespace)
│ gha-runner-scale-set-       │   - Always-on, ~50m CPU / 64Mi RAM
│ controller                  │   - Leader-elected operator
└─────────────────────────────┘
        │  manages
        ▼
┌─────────────────────────────┐
│ Runner Scale Set listeners  │   (arc-runners namespace)
│ tomoda-arc, -db, -trusted   │   - one per scale set, polls GitHub API
└─────────────────────────────┘
        │  spawns one pod per job, up to maxRunners
        ▼
┌─────────────────────────────┐
│ Runner pods (ephemeral)     │   (arc-runners namespace)
│ runs-on: tomoda-arc         │   - 1 CPU / 2.5Gi requested, 1.5 / 6Gi limit
│ tomoda-arc-runner:<VERSION> │   - custom image: stock runner + gcc + Docker CLI
└─────────────────────────────┘   - ephemeral: tears down between jobs
┌─────────────────────────────┐
│ Runner pods + Postgres      │   (arc-runners namespace)
│ runs-on: tomoda-arc-db      │   - same runner, plus a per-job Postgres sidecar
│ + tomoda-postgres sidecar   │   - sidecar: 100m / 256Mi requested, 1.5 / 1Gi limit
│                             │   - runner limit 4Gi
└─────────────────────────────┘
┌─────────────────────────────┐
│ Trusted runner pods         │   (arc-runners namespace)
│ runs-on: tomoda-arc-trusted │   - same requests and limits as tomoda-arc
│                             │   - no hostPath caches, one job at a time
└─────────────────────────────┘

Three scale sets share the controller, namespace and GitHub App credentials:

Scale set Argo app Values For
tomoda-arc arc-runners k8s/envs/platform/arc-runners/values.yaml Every CI job that needs no database
tomoda-arc-db arc-runners-db k8s/envs/platform/arc-runners-db/values.yaml Jobs that need a real Postgres (the tomoda backend integration suite and schema drift check)
tomoda-arc-trusted arc-runners-trusted k8s/envs/platform/arc-runners-trusted/values.yaml Jobs that hold prod credentials, from main only (see Trusted pool)

Jobs that stay on GitHub-hosted runners: terraform-apply.yml (break-glass path when the node is down, strongest credentials) and the build jobs of both repos' images.yml (they need Docker, and a broken cluster must still be able to build the images it needs).

Cost

Runners use spare capacity on the server, so they cost nothing extra. Each runner pod requests 1 CPU / 2.5Gi, about what a job uses at its peak (7-day p50 1.0 CPU, p90 2.4 GiB), so the scheduler counts CI against prod. It is capped at 1.5 CPU / 6Gi (the Expo OTA export runs Node with a 3 GiB heap; tomoda-arc-db runners cap at 4Gi). There are no node selectors. With all pools full (3 + 2 + 1 pods) CI requests 6.2 CPU / about 15.5 Gi; the budget is in Server capacity.

Priority and QoS

All runner pods use the ci-runner PriorityClass (k8s/envs/platform/manifests/priority-classes.yaml): value -1000, below every other workload, and preemptionPolicy: Never, so a runner never evicts anything. When prod needs room (HPA scale-out, a rollout surge) the scheduler preempts runners first, and node-pressure eviction picks them first. A preempted job fails and is re-run. Runners are Burstable, so under memory pressure the OOM killer also takes them (usage above request) before prod pods, which sit under their requests.

Capping behaviour

  • minRunners: 0 in k8s/envs/platform/arc-runners/values.yaml: scales to zero between jobs
  • maxRunners: 3 (tomoda-arc), 2 (tomoda-arc-db) and 1 (tomoda-arc-trusted): at most 6 concurrent jobs. Raising any of them needs headroom in the CI capacity budget: every extra runner reserves 1 CPU
  • Per-pod: requests 1 / 2560Mi, limits 1500m / 6Gi (4Gi on tomoda-arc-db); a tomoda-arc-db pod adds 100m / 256Mi, limits 1500m / 1Gi for Postgres. Native iOS / Android builds run off-cluster

Opting a workflow in

In any workflow YAML, replace the runs-on: value:

# Before
jobs:
  build:
    runs-on: ubuntu-24.04

# After
jobs:
  build:
    runs-on: tomoda-arc

The next git push to a branch with that workflow queues a job; ARC spawns a runner pod within ~10s; it registers as tomoda-arc, runs the job, terminates.

The runner label matches the chart release name (releaseName in each scale set's application.yaml). Renaming a release means updating every workflow's runs-on:.

The runner image has no gh or gcloud. Jobs that need gh run the repo's ./.github/actions/setup-gh (a pinned, checksum-verified download into $RUNNER_TEMP, a no-op once the image ships gh); jobs that call gcloud add google-github-actions/setup-gcloud. GitHub OIDC tokens are issued per job whatever the runner, so Workload Identity Federation works from these pods; the WIF providers condition on repository, ref and workflow file, not on the runner type.

Trusted pool (tomoda-arc-trusted)

For jobs that hold prod credentials: the release App key or the tomoda-release service account (via WIF). Today these are the tomoda release.yml jobs (cut, bump-prod, ota-gate), the prod-pin job and the prod run of the publish job of tomoda ota.yml, and devops notify-prod-deploy.yml.

  • Never PR code. Those workflows trigger only on workflow_dispatch, workflow_call, repository_dispatch or push to main, and each trusted job carries a job-level if: on refs/heads/main. terraform-plan.yml runs on pull requests, so it uses tomoda-arc, not this pool.
  • No shared caches. The pod mounts no hostPath: the shared pool runs PR code and writes the node cache, so a trusted job must not read it.
  • One job at a time (maxRunners: 1), scales to zero.

The credentials themselves are gated in GCP, not by the runner: the tomoda-release WIF bindings accept only the named workflow files on refs/heads/main (infrastructure/gcp/wif.tf), so a job on any pool from another branch or workflow gets no token.

Limit: the label is not access control. On GitHub Free an org has only the Default runner group, so any workflow in an org repo, a PR branch included, can request runs-on: tomoda-arc-trusted. Such a job gets the pod but not the prod credentials (see above). Creating more runner groups, and restricting a group to selected workflows, needs a paid plan (GitHub Team or above). With one, the org owner would:

  1. Org settings → Actions → Runner groups → New runner group: name trusted, selected repositories tomoda and devops, and under workflow access choose selected workflows: tomoda-labs/tomoda/.github/workflows/release.yml@refs/heads/main, tomoda-labs/tomoda/.github/workflows/ota.yml@refs/heads/main, tomoda-labs/devops/.github/workflows/notify-prod-deploy.yml@refs/heads/main.
  2. Set runnerGroup: trusted in arc-runners-trusted/values.yaml and merge; the scale set re-registers in that group.

Postgres sidecar (tomoda-arc-db)

A tomoda-arc-db pod runs the production Postgres image (ghcr.io/tomoda-labs/tomoda-postgres, PostGIS + PGroonga + pgvector) as a native sidecar next to the runner:

  • Ready before the job. The sidecar's startupProbe holds the runner container until Postgres answers, so the job never sees a half-initialised database. Setup (initdb, the extensions postgis, postgis_topology, pgroonga, vector, unaccent in template1, then createdb tomoda) runs on a socket-only server, so TCP answers only once it is done.
  • Connect as tomoda to localhost:5432, database tomoda. Trust auth, so any password works. Databases created later inherit the extensions.
  • Loopback only. listen_addresses=localhost and the default pg_hba allow only 127.0.0.1 / ::1: the database is reachable from containers in the same pod, not over the pod IP.
  • Unprivileged. uid 26, read-only root filesystem, all capabilities dropped, no privilege escalation, RuntimeDefault seccomp. Data lives in an emptyDir (4 Gi cap) with fsync off; it is deleted with the pod, so each job starts empty.
  • Image tag is pinned in arc-runners-db/values.yaml; keep it at k8s/envs/platform/postgres-image/VERSION (scripts/check-image-versions.sh fails the PR otherwise).
jobs:
  integration:
    runs-on: tomoda-arc-db
    steps:
      - run: psql -h localhost -U tomoda -d tomoda -c 'select postgis_version()'

Verification

# 1. Controller is running
kubectl get pods -n arc-systems
#   NAME                                READY   STATUS
#   arc-gha-rs-controller-...           1/1     Running

# 2. Runner Scale Set is registered with GitHub
kubectl get autoscalingrunnerset -n arc-runners
#   NAME         MINIMUM   MAXIMUM   CURRENT   STATE
#   tomoda-arc      0         3         0         <pending until first job>
#   tomoda-arc-db   0         2         0         <pending until first job>

# 3. Listener pod is polling GitHub
kubectl get pods -n arc-runners -l app.kubernetes.io/component=runner-set-listener
#   NAME                                READY   STATUS
#   tomoda-arc-...-listener             1/1     Running
#   tomoda-arc-db-...-listener          1/1     Running

# 4. The ESO-projected GitHub App secret exists
kubectl get secret arc-github-app-credentials -n arc-runners
#   NAME                          TYPE     DATA   AGE
#   arc-github-app-credentials    Opaque   3      <new>

# 5. Trigger a test job
#    In a workflow YAML, change runs-on to tomoda-arc + push.
#    Watch ARC spawn a pod:
kubectl get pods -n arc-runners -w
#   New pod appears within ~10s of the job being queued
#   Pod completes + terminates within seconds of the job finishing

If a runner pod never spawns when a job is queued, check the listener pod logs:

kubectl logs -n arc-runners -l app.kubernetes.io/component=runner-set-listener --tail=50

Most likely cause: GitHub App permissions are missing or wrong. Re-verify the App has Repository: Actions: Read+Write and Administration: Read+Write.

Runner image

Runners use a thin custom image, ghcr.io/tomoda-labs/tomoda-arc-runner, built from stock ghcr.io/actions/actions-runner by k8s/envs/platform/arc-runner-image/Dockerfile (.github/workflows/images.yml), tagged from that directory's VERSION and pinned in arc-runners/values.yaml. It adds what stock lacks:

  • build-essential: a C toolchain, so go test -race (needs cgo) runs.
  • postgresql-client-17 (PGDG apt repo): psql, pg_dump, pg_isready matching the Postgres 17 server, for jobs on tomoda-arc-db.
  • Docker CLI + buildx + compose: the client for the dind daemon below.
  • Pinned CI lint/scan tools: shellcheck, gitleaks, kubeconform, yq, conftest, yamllint. Fetched, checksum-verified, and extracted in a builder stage; only the resulting binaries are copied into the final image. Bumping a version means editing the pinned ARG + checksum in the Dockerfile, a reviewed diff rather than an unpinned fetch at job runtime. infra-lint.yml, k8s-validate.yml, and secret-scan.yml assume these are already on $PATH.

Kept thin (a few MB over stock) so scale-to-zero cold-start pulls stay fast.

The stock base is pinned to a runner release. GitHub stops dispatching jobs to runners that fall too far behind the current release: the runner exits at startup, ARC marks it Failed after 5 attempts, and queued jobs never start. Dependabot (docker ecosystem in .github/dependabot.yml) opens a weekly PR to move the pin; merge it with a VERSION bump.

Docker-in-Docker is available but off. The image ships the Docker CLI, so enabling it is a one-line flip in arc-runners/values.yaml (uncomment containerMode: dind), no rebuild. It adds a privileged docker:dind sidecar and wires DOCKER_HOST for docker build / buildx and service containers. Caveat: containerMode: dind makes every runner pod privileged; weigh that (or split a separate dind scale set) before enabling.

What runners CAN'T do (current limits)

  • macOS-only jobs: runner pods are Linux containers. Native store builds run locally, not in CI.
  • Docker in jobs: dind is off (see above); docker build and service containers have no daemon. For a database use tomoda-arc-db. Image builds run on GitHub-hosted runners (images.yml).
  • Persistent caches (partial): Go build cache (~/.cache/go-build) and golangci-lint cache (~/.cache/golangci-lint) persist via per-node hostPath volumes and survive job-to-job on the same node. Go module cache (~/go/pkg/mod) is intentionally not mounted (a root-owned hostPath breaks setup-go's ~/go/bin mkdir); it is handled by a separate GHA cache step. Docker layer caches, $HOME pip wheels, and other per-user caches don't survive job-to-job.

Most CI workloads (Go tests, ESLint, Playwright headless, docs build) run fine on Linux ARC pods.

Rotation

  • GitHub App private key: in the App settings page, generate a new key + delete the old. Push the new PEM to GCP SM:

    gcloud secrets versions add tomoda-github-app-private-key \
      --project=development-485000 --data-file=new-key.pem
    

    Then either wait 1h (ESO sync) or force restart:

    kubectl rollout restart deploy -n arc-systems
    kubectl delete pod -n arc-runners -l app.kubernetes.io/component=runner-set-listener
    
  • GitHub App itself: deleting the App invalidates all credentials immediately. Re-create per the bootstrap doc, push new IDs + key to GCP SM.

Argo CD sync

The arc-runners Application runs with prune: false. The ARC controller creates an AutoscalingListener (+ its Role/RoleBinding) that inherit the app's Argo tracking label but aren't in git; with prune on, Argo and the controller fight in a permanent sync loop (actions-runner-controller#2949). selfHeal stays on; only reaping controller-owned resources is disabled.