Skip to content

ARC Self-Hosted Runners

Tomoda ARC Runners GitHub App avatar

GitHub Actions runners on the GKE cluster instead of GitHub-hosted. Saves the free-plan Actions-minutes limit and gives CI the same cluster + network access as the other workloads.

Powered by Actions Runner Controller (ARC). Runner pods are ephemeral: spawn on a queued job, register with GitHub, run, terminate. Scales to 0 between jobs.

Architecture

GitHub Actions queue
        │
        │  poll for jobs
        ▼
┌─────────────────────────────┐
│ ARC controller              │   (arc-systems namespace)
│ gha-runner-scale-set-       │   - Always-on, ~50m CPU / 64Mi RAM
│ controller                  │   - Leader-elected operator
└─────────────────────────────┘
        │  manages
        ▼
┌─────────────────────────────┐
│ Runner Scale Set listener   │   (arc-runners namespace)
│ Service: tomoda-arc         │   - Polls GitHub API
└─────────────────────────────┘
        │  spawns N pods per job, up to maxRunners
        ▼
┌─────────────────────────────┐
│ Runner pods (ephemeral)     │   (arc-runners namespace)
│ runs-on: tomoda-arc         │   - 200m CPU / 1.5Gi requested, 1.5/2Gi limit
│ tomoda-arc-runner:latest    │   - custom image: stock runner + gcc + Docker CLI
└─────────────────────────────┘   - ephemeral: tears down between jobs

Cost (us-central1 spot pricing)

Component Idle cost Active cost
ARC controller ~$0.50/mo (always-on tiny pod) same — does no real work between jobs
Listener pod bundled in controller cost same
Runner pods $0 (terminated between jobs) spot t2a-standard-2 time per concurrent job
Extra node capacity (autoscale-up when concurrent jobs exceed current nodes) $0 most of the time spot t2a-standard-2 time when peaking
Realistic monthly total ~$0.50 ~$2-5/mo for 5-10 PRs/day

Math: 5 PRs × 4 workflows × 8 min ≈ 160 runner-min/day = 80 hrs/mo of compute. At spot pricing in us-central1, ~$1-2 of job time on top of the $0.50 controller baseline.

Controller + listener run on the on-demand pool=system (a preempted listener drops its GitHub session); runner pods pin to the spot pool=burst via nodeSelector: {pool: burst} (k8s/envs/platform/arc-runners/values.yaml). Mid-job preemption is fine since GitHub Actions retries. See GKE node pools.

Capping behaviour

  • minRunners: 0 in k8s/envs/platform/arc-runners/values.yaml — scales to zero between jobs
  • maxRunners: 3 — no more than 3 concurrent jobs. Two reasons:
    1. Bounds the worst-case compute spend
    2. Prevents starving prod (or dev tomoda) of node capacity under a CI storm
  • Per-pod limits: 1500m CPU / 2 Gi RAM — heavy Go builds + Node lint fit; native iOS / Android builds would need a larger pool

Opting a workflow in

In any workflow YAML, replace the runs-on: value:

# Before
jobs:
  build:
    runs-on: ubuntu-latest

# After
jobs:
  build:
    runs-on: tomoda-arc

The next git push to a branch with that workflow queues a job; ARC spawns a runner pod within ~10s; it registers as tomoda-arc, runs the job, terminates.

The runner label tomoda-arc matches the chart release name in arc-runners/application.yaml (releaseName: tomoda-arc). Renaming the release means updating every workflow's runs-on:.

Verification

# 1. Controller is running
kubectl get pods -n arc-systems
#   NAME                                READY   STATUS
#   arc-gha-rs-controller-...           1/1     Running

# 2. Runner Scale Set is registered with GitHub
kubectl get autoscalingrunnerset -n arc-runners
#   NAME         MINIMUM   MAXIMUM   CURRENT   STATE
#   tomoda-arc   0         3         0         <pending until first job>

# 3. Listener pod is polling GitHub
kubectl get pods -n arc-runners -l app.kubernetes.io/component=runner-set-listener
#   NAME                                READY   STATUS
#   tomoda-arc-...-listener             1/1     Running

# 4. The ESO-projected GitHub App secret exists
kubectl get secret arc-github-app-credentials -n arc-runners
#   NAME                          TYPE     DATA   AGE
#   arc-github-app-credentials    Opaque   3      <new>

# 5. Trigger a test job
#    In a workflow YAML, change runs-on to tomoda-arc + push.
#    Watch ARC spawn a pod:
kubectl get pods -n arc-runners -w
#   New pod appears within ~10s of the job being queued
#   Pod completes + terminates within seconds of the job finishing

If a runner pod never spawns when a job is queued, check the listener pod logs:

kubectl logs -n arc-runners -l app.kubernetes.io/component=runner-set-listener --tail=50

Most likely cause: GitHub App permissions are missing or wrong. Re-verify the App has Repository: Actions: Read+Write and Administration: Read+Write.

Runner image

Runners use a thin custom image, tomoda-arc-runner:latest, built from stock ghcr.io/actions/actions-runner by k8s/envs/platform/arc-runner-image/Dockerfile (Cloud Build arc-runner-image-push-trigger, pushed to both AR repos). It adds what stock lacks:

  • build-essential — a C toolchain, so go test -race (needs cgo) runs.
  • Docker CLI + buildx + compose — the client for the dind daemon below.
  • Pinned CI lint/scan tools — shellcheck, gitleaks, kubeconform, yq, conftest, yamllint. Fetched, checksum-verified, and extracted in a builder stage; only the resulting binaries are copied into the final image. Bumping a version means editing the pinned ARG + checksum in the Dockerfile — a reviewed diff, not an unpinned .../latest/... fetch at job runtime. infra-lint.yml, k8s-validate.yml, and secret-scan.yml assume these are already on $PATH.

Kept thin (a few MB over stock) so scale-to-zero cold-start pulls stay fast.

Docker-in-Docker is available but off. The image ships the Docker CLI, so enabling it is a one-line flip in arc-runners/values.yaml (uncomment containerMode: dind), no rebuild. It adds a privileged docker:dind sidecar and wires DOCKER_HOST for docker build / buildx and service containers. Caveat: containerMode: dind makes every runner pod privileged — weigh that (or split a separate dind scale set) before enabling.

What runners CAN'T do (current limits)

  • macOS-only jobs — runner pods are Linux containers. iOS builds needing a Mac stay on GitHub-hosted runs-on: macos-latest (billed at 10× the Linux rate; watch the minutes meter).
  • Docker in jobs — dind is off (see above); docker build has no daemon until enabled. Use kaniko / buildah, or enable dind.
  • Persistent caches (partial) — Go build cache (~/.cache/go-build) and golangci-lint cache (~/.cache/golangci-lint) persist via per-node hostPath volumes and survive job-to-job on the same node. Go module cache (~/go/pkg/mod) is intentionally not mounted (a root-owned hostPath breaks setup-go's ~/go/bin mkdir); it is handled by a separate GHA cache step. Docker layer caches, $HOME pip wheels, and other per-user caches don't survive job-to-job.

Most CI workloads (Go tests, ESLint, Playwright headless, docs build) run fine on Linux ARC pods.

Rotation

  • GitHub App private key: in the App settings page, generate a new key + delete the old. Push the new PEM to GCP SM:

    gcloud secrets versions add tomoda-github-app-private-key \
  --project=development-485000 --data-file=new-key.pem

    Then either wait 1h (ESO sync) or force restart:

    kubectl rollout restart deploy -n arc-systems
kubectl delete pod -n arc-runners -l app.kubernetes.io/component=runner-set-listener

  • GitHub App itself: deleting the App invalidates all credentials immediately. Re-create per the bootstrap doc, push new IDs + key to GCP SM.

Argo CD sync

The arc-runners Application runs with prune: false. The ARC controller creates an AutoscalingListener (+ its Role/RoleBinding) that inherit the app's Argo tracking label but aren't in git; with prune on, Argo and the controller fight in a permanent sync loop (actions-runner-controller#2949). selfHeal stays on; only reaping controller-owned resources is disabled.