Skip to content

Vulnerability scanning

How container images in the cluster get scanned for CVEs, how findings reach the team, and the security properties of that pipeline.

Mental model

trivy-operator ──scans──▶ VulnerabilityReport CRDs ──read──▶ trivy-issue-reporter ──▶ GitHub issues
      ▲                                                              (06:00 UTC)         (tomoda-labs/devops)
      │ pulls image                                                                            │
   ghcr-pull imagePullSecret ◀──syncs── External Secrets (GHCR read token)                     │
                                                                                                  ▼
                                                                          trivy-triage (07:00 UTC) ──▶ labels + scans
                                                                                                        candidate fixes,
                                                                                                        overwrites pinned
                                                                                                        report issue #368

Four pieces, all in trivy-system / trivy-issue-reporter namespaces:

Piece What it does Manifest
trivy-operator Scans running images, writes VulnerabilityReport / ConfigAuditReport CRDs k8s/envs/platform/trivy-operator/
ghcr-pull GHCR pull secret attached to the tomoda Deployments so trivy can pull the private tomoda-* images k8s/apps/tomoda/base/ghcr-pull.yaml
trivy-issue-reporter Daily job turning findings into GitHub issues k8s/apps/trivy-issue-reporter/
trivy-triage Daily job that checks whether an upstream fix actually exists for each open finding, and rolls the result into a pinned report issue k8s/apps/trivy-issue-reporter/ (same manifest file)

Why ghcr-pull exists

trivy-operator authenticates to a registry only through a workload's imagePullSecrets (directly or via its ServiceAccount). Nodes pull GHCR images with node-level credentials (k3s registries.yaml), which trivy can't see, so without a pull secret it produces no reports for tomoda-dev / tomoda-prod.

ghcr-pull closes that gap: an ExternalSecret builds a dockerconfigjson from tomoda-ghcr-read-credentials in each tenant namespace, and the API, async and frontend Deployments list it under imagePullSecrets. The token is a read:packages PAT, so no refresh job is needed.

trivy-issue-reporter

A daily CronJob (schedule: 0 6 * * *) reads the VulnerabilityReport CRDs and files GitHub issues on tomoda-labs/devops:

  • Scope: our images (tomoda-*) at HIGH+CRITICAL; third-party images at CRITICAL only.
  • Dedup: each finding keys on CVE + image + package, stored in an HTML comment marker in the issue body. Still-open findings are skipped; findings that disappear from the latest scan get their issue closed.
  • Auth: the shared GitHub App (same App ARC uses, id from GSM tomoda-github-app-*) signs a JWT and exchanges it for a short-lived installation token. The App needs Issues: write on the repo.

Reads the CRDs through the in-cluster Kubernetes API with the job's ServiceAccount token, so the image is just python:3.12-slim + openssl.

trivy-triage

A daily CronJob (schedule: 0 7 * * *, an hour after the reporter) answers what the reporter's Fixed in: field can't: does an upstream build that contains the fix exist yet? A higher tag number is not proof, so it scans a real candidate image. The script is k8s/apps/trivy-issue-reporter/trivy-triage.py; a kustomize configMapGenerator turns it into the trivy-triage-script ConfigMap the CronJob mounts, so edits to the file roll out on the next sync.

  1. For each open, untriaged trivy issue, resolve the image's upstream GitHub repo from the hand-maintained UPSTREAM_MAP.
  2. Pick the closest newer release on the deployed major.minor line, else the newest stable release.
  3. Run trivy image against that candidate on the registry (no pull, no daemon) and check whether the CVE is still reported.
  4. Label the issue and comment the scan result:
    • fix-available: terminal.
    • upstream-unfixed: re-checked every RECHECK_AFTER_DAYS (default 7).
    • node-image: rancher/* system images, fixed only by a k3s upgrade; terminal.
    • needs-manual: that issue's triage failed (API error, scan timeout); the run continues.
  5. Overwrite the pinned report issue (#368): fix-available items first (image, CVE, candidate ref, evidence), then needs-manual, node-image, and upstream-unfixed collapsed. A weekly session can start bump PRs from it without re-deriving anything.

Base image is aquasec/trivy; apk add python3 openssl runs at container start.

Security properties

Property Assessment
Read-only registry key ghcr-pull holds a classic PAT with only read:packages. A leak grants image reads, nothing else; rotate it in GCP SM and ESO resyncs within the hour.
GitHub auth is short-lived The App JWT is exchanged for an installation token per run; no PAT is stored. The App private key lives in GSM and is projected via ESO, never written to git.
Least-privilege RBAC trivy-issue-reporter has read-only get/list on vulnerabilityreports.
TLS verified The reporter reads the Kubernetes API with the projected CA (ssl.create_default_context(cafile=...)), not an unverified context.
Finding data → GitHub CVE ids, titles, image and package names from scan results are sent as JSON (json.dumps), so a hostile string in a report cannot inject into the GitHub API. Worst case is a malformed dedup marker (duplicate issue), not code execution.
trivy-triage stays read-only + scan-only Talks only to the GitHub API and public registries; no registry credentials, no cluster write access beyond posting labels/comments/the report issue. Candidate images are scanned, never run: trivy image inspects layers without executing them. The upstream-repo map is a static, hand-maintained list, not resolved from an untrusted source.

GitHub App permission

Changing the App's permissions requires an org owner to approve the update on the installation before the new scope (Issues: write) takes effect. Until then the reporter's token has no issue scope and the job 403s.