Skip to content

Vulnerability scanning

How container images in the cluster get scanned for CVEs, how findings reach the team, and the security properties of that pipeline.

Mental model

trivy-operator ──scans──▶ VulnerabilityReport CRDs ──read──▶ trivy-issue-reporter ──▶ GitHub issues
      ▲                                                              (06:00 UTC)         (tomoda-labs/devops)
      │ pulls image                                                                            │
   ar-pull imagePullSecret ◀──writes── ar-pull-refresh (CronJob, node AR token)                 │
                                                                                                  ▼
                                                                          trivy-triage (07:00 UTC) ──▶ labels + scans
                                                                                                        candidate fixes,
                                                                                                        overwrites pinned
                                                                                                        report issue #368

Four pieces, all in trivy-system / trivy-issue-reporter namespaces:

Piece What it does Manifest
trivy-operator Scans running images, writes VulnerabilityReport / ConfigAuditReport CRDs k8s/envs/platform/trivy-operator/
ar-pull-refresh Mints a short-lived Artifact Registry pull token so trivy can pull private tomoda-* images k8s/envs/platform/ar-pull-refresh/
trivy-issue-reporter Daily job turning findings into GitHub issues k8s/envs/platform/trivy-issue-reporter/
trivy-triage Daily job that checks whether an upstream fix actually exists for each open finding, and rolls the result into a pinned report issue k8s/envs/platform/trivy-issue-reporter/ (same manifest file)

Why ar-pull-refresh exists

trivy-operator authenticates to a registry only through a workload's imagePullSecrets (directly or via its ServiceAccount). It does not use the node service account or Workload Identity. Our tomoda-* pods carry no imagePullSecret (they pull through the node compute SA), so trivy could not pull them and produced zero reports for tomoda-dev / tomoda-prod.

ar-pull-refresh closes that gap. Every 45 min a CronJob:

  1. Mints an Artifact Registry access token from the node compute SA via the metadata server (gcloud auth print-access-token). That SA holds only roles/artifactregistry.reader (infrastructure/gcp/gke_iam.tf).
  2. Writes it as the ar-pull dockerconfigjson secret in each tenant namespace.
  3. Attaches ar-pull to that namespace's default ServiceAccount, so trivy scan jobs inherit it.

trivy-issue-reporter

A daily CronJob (schedule: 0 6 * * *) reads the VulnerabilityReport CRDs and files GitHub issues on tomoda-labs/devops:

  • Scope: our images (tomoda-*) at HIGH+CRITICAL; third-party images at CRITICAL only.
  • Dedup: each finding keys on CVE + image + package, stored in an HTML comment marker in the issue body. Still-open findings are skipped; findings that disappear from the latest scan get their issue closed.
  • Auth: the shared GitHub App (same App ARC uses, id from GSM tomoda-github-app-*) signs a JWT and exchanges it for a short-lived installation token. The App needs Issues: write on the repo.

Reads the CRDs through the in-cluster Kubernetes API with the job's ServiceAccount token, so the image is just python:3.12-slim + openssl.

trivy-triage

A daily CronJob (schedule: 0 7 * * *, an hour after the reporter) that answers the question the reporter's Fixed in: field can't: does an upstream build that actually contains the fix exist yet? A higher tag number is not proof — tag-naming conventions vary per project and none of them reliably say "this build contains fix commit X" — so the only trustworthy signal is a real scan of a real candidate image (scripts/trivy-triage.py):

  1. For each open trivy-labeled issue not yet triaged, resolve the image's upstream GitHub repo from a static map (UPSTREAM_MAP in the script — hand-maintained, not auto-discovered).
  2. List that repo's releases and pick the closest newer release on the same major.minor line as what's deployed (narrowest possible diff), falling back to the newest stable release if none exists on that line.
  3. Run trivy image directly against that candidate image reference on the registry (no local pull/daemon) and check whether the specific CVE is still reported.
  4. Label the issue fix-available or upstream-unfixed, with a comment recording the scan result. fix-available and node-image are terminal — a re-scan tells us nothing new. upstream-unfixed is not terminal: upstream ships fixes on its own schedule, so it's re-checked every RECHECK_AFTER_DAYS (default 7) via a timestamp marker in the triage comment, instead of being labeled once and never revisited. Images riding the GKE node image (gke-release/*, prometheus-to-sd, etc.) are labeled node-image and skipped — no repo PR can fix those, only a node pool bump. A single issue's triage failure (API error, scan timeout) is caught and reported as needs-manual, not fatal to the run — the other issues and the report still get processed.
  5. Overwrite the pinned report issue (#368) with a structured summary — actionable fix-available items first (image, CVE, candidate image ref, scan evidence), then what needs manual triage, node image findings, and upstream-unfixed findings collapsed. Written for a cold read: a weekly session can open it and start bump PRs without re-deriving anything the daily runs already checked.

Base image is aquasec/trivy (the scanner itself); apk add python3 openssl runs at container start since the base has neither by default.

Security properties

The pipeline was reviewed for the risks its own moving parts introduce.

Property Assessment
No long-lived registry key ar-pull holds a metadata-server token that expires within ~1h. A leak only grants artifactregistry.reader (read images), because that is all the node SA has.
GitHub auth is short-lived The App JWT is exchanged for an installation token per run; no PAT is stored. The App private key lives in GSM and is projected via ESO, never written to git.
Least-privilege RBAC ar-pull-refresh can touch only secrets + serviceaccounts, bound in tomoda-dev / tomoda-prod only (RoleBindings, not a cluster-wide grant). trivy-issue-reporter has read-only get/list on vulnerabilityreports.
TLS verified The reporter reads the Kubernetes API with the projected CA (ssl.create_default_context(cafile=...)), not an unverified context.
Finding data → GitHub CVE ids, titles, image and package names from scan results are sent as JSON (json.dumps), so a hostile string in a report cannot inject into the GitHub API. Worst case is a malformed dedup marker (duplicate issue), not code execution.
trivy-triage stays read-only + scan-only Talks only to the GitHub API and public registries; no registry credentials, no cluster write access beyond posting labels/comments/the report issue. Candidate images are scanned, never run — trivy image inspects layers, it doesn't execute the image. The upstream-repo map is a static, hand-maintained list, not resolved from an untrusted source.

GitHub App permission

Changing the App's permissions requires an org owner to approve the update on the installation before the new scope (Issues: write) takes effect. Until then the reporter's token has no issue scope and the job 403s.