Vulnerability scanning¶
How container images in the cluster get scanned for CVEs, how findings reach the team, and the security properties of that pipeline.
Mental model¶
trivy-operator ──scans──▶ VulnerabilityReport CRDs ──read──▶ trivy-issue-reporter ──▶ GitHub issues
▲ (06:00 UTC) (tomoda-labs/devops)
│ pulls image │
ghcr-pull imagePullSecret ◀──syncs── External Secrets (GHCR read token) │
▼
trivy-triage (07:00 UTC) ──▶ labels + scans
candidate fixes,
overwrites pinned
report issue #368
Four pieces, all in trivy-system / trivy-issue-reporter namespaces:
| Piece | What it does | Manifest |
|---|---|---|
trivy-operator |
Scans running images, writes VulnerabilityReport / ConfigAuditReport CRDs |
k8s/envs/platform/trivy-operator/ |
ghcr-pull |
GHCR pull secret attached to the tomoda Deployments so trivy can pull the private tomoda-* images |
k8s/apps/tomoda/base/ghcr-pull.yaml |
trivy-issue-reporter |
Daily job turning findings into GitHub issues | k8s/apps/trivy-issue-reporter/ |
trivy-triage |
Daily job that checks whether an upstream fix actually exists for each open finding, and rolls the result into a pinned report issue | k8s/apps/trivy-issue-reporter/ (same manifest file) |
Why ghcr-pull exists¶
trivy-operator authenticates to a registry only through a workload's imagePullSecrets (directly or via its ServiceAccount). Nodes pull GHCR images with node-level credentials (k3s registries.yaml), which trivy can't see, so without a pull secret it produces no reports for tomoda-dev / tomoda-prod.
ghcr-pull closes that gap: an ExternalSecret builds a dockerconfigjson from tomoda-ghcr-read-credentials in each tenant namespace, and the API, async and frontend Deployments list it under imagePullSecrets. The token is a read:packages PAT, so no refresh job is needed.
trivy-issue-reporter¶
A daily CronJob (schedule: 0 6 * * *) reads the VulnerabilityReport CRDs and
files GitHub issues on tomoda-labs/devops:
- Scope: our images (
tomoda-*) at HIGH+CRITICAL; third-party images at CRITICAL only. - Dedup: each finding keys on
CVE + image + package, stored in an HTML comment marker in the issue body. Still-open findings are skipped; findings that disappear from the latest scan get their issue closed. - Auth: the shared GitHub App (same App ARC uses, id from GSM
tomoda-github-app-*) signs a JWT and exchanges it for a short-lived installation token. The App needs Issues: write on the repo.
Reads the CRDs through the in-cluster Kubernetes API with the job's ServiceAccount
token, so the image is just python:3.12-slim + openssl.
trivy-triage¶
A daily CronJob (schedule: 0 7 * * *, an hour after the reporter) answers what the reporter's Fixed in: field can't: does an upstream build that contains the fix exist yet? A higher tag number is not proof, so it scans a real candidate image. The script is k8s/apps/trivy-issue-reporter/trivy-triage.py; a kustomize configMapGenerator turns it into the trivy-triage-script ConfigMap the CronJob mounts, so edits to the file roll out on the next sync.
- For each open, untriaged
trivyissue, resolve the image's upstream GitHub repo from the hand-maintainedUPSTREAM_MAP. - Pick the closest newer release on the deployed major.minor line, else the newest stable release.
- Run
trivy imageagainst that candidate on the registry (no pull, no daemon) and check whether the CVE is still reported. - Label the issue and comment the scan result:
fix-available: terminal.upstream-unfixed: re-checked everyRECHECK_AFTER_DAYS(default 7).node-image:rancher/*system images, fixed only by a k3s upgrade; terminal.needs-manual: that issue's triage failed (API error, scan timeout); the run continues.
- Overwrite the pinned report issue (#368): fix-available items first (image, CVE, candidate ref, evidence), then needs-manual, node-image, and upstream-unfixed collapsed. A weekly session can start bump PRs from it without re-deriving anything.
Base image is aquasec/trivy; apk add python3 openssl runs at container start.
Security properties¶
| Property | Assessment |
|---|---|
| Read-only registry key | ghcr-pull holds a classic PAT with only read:packages. A leak grants image reads, nothing else; rotate it in GCP SM and ESO resyncs within the hour. |
| GitHub auth is short-lived | The App JWT is exchanged for an installation token per run; no PAT is stored. The App private key lives in GSM and is projected via ESO, never written to git. |
| Least-privilege RBAC | trivy-issue-reporter has read-only get/list on vulnerabilityreports. |
| TLS verified | The reporter reads the Kubernetes API with the projected CA (ssl.create_default_context(cafile=...)), not an unverified context. |
| Finding data → GitHub | CVE ids, titles, image and package names from scan results are sent as JSON (json.dumps), so a hostile string in a report cannot inject into the GitHub API. Worst case is a malformed dedup marker (duplicate issue), not code execution. |
| trivy-triage stays read-only + scan-only | Talks only to the GitHub API and public registries; no registry credentials, no cluster write access beyond posting labels/comments/the report issue. Candidate images are scanned, never run: trivy image inspects layers without executing them. The upstream-repo map is a static, hand-maintained list, not resolved from an untrusted source. |
GitHub App permission
Changing the App's permissions requires an org owner to approve the update on the installation before the new scope (Issues: write) takes effect. Until then the reporter's token has no issue scope and the job 403s.