Skip to content

Argo CD

Argo CD is the only thing that writes to the cluster. Every namespace, Helm release, Deployment and Secret is the output of an Application sync. Auto-sync, auto-prune and self-heal are on by default, so anything not in this repo gets reverted within minutes.

One k3s cluster, three logical tiers wired up by Argo CD: platform (cluster-wide system services), dev (the dev tenant data plane), prod (the prod tenant data plane). See Environments for the full tenancy model and tier sizing.

Argo CD itself is a Helm release with values in k8s/envs/platform/argocd/values.yaml (Dex + Google SSO, RBAC, metrics). scripts/ovh/bootstrap-argocd.sh installs it once; after that it manages itself (below). This page covers the GitOps layer on top.

Self-management

k8s/envs/platform/argocd/application.yaml is an Application for Argo CD's own chart (argo-cd 10.12.1, release argocd, namespace argocd, same values file). Because the release name and namespace match the bootstrap install, it adopts those resources rather than creating new ones.

  • Upgrade: bump targetRevision in the Application (and ARGOCD_CHART_VERSION in the bootstrap script, so a rebuild installs the same version). Values changes are ordinary PRs.
  • Safety: prune: false and no resources finalizer, so Argo CD never deletes its own components, and deleting the Application leaves Argo CD running. ServerSideApply=true handles the large CRDs.
  • Not managed: the two hand-made Secrets (argocd/repo-devops, external-secrets/gcpsm-credentials) are outside the chart.
  • Metrics: the chart's own ServiceMonitors are off (it only renders them when the monitoring CRDs exist at install time); k8s/envs/platform/manifests/argocd-servicemonitors.yaml defines them instead.

Bootstrap

One file, applied once by scripts/ovh/bootstrap-argocd.sh after Argo CD is installed (see Bootstrap):

kubectl apply -f k8s/envs/bootstrap.yaml

k8s/envs/bootstrap.yaml creates three top-level Applications: platform, dev, prod. Each recurses through its respective directory under k8s/envs/ and picks up the sub-Applications it finds.

Reconciliation takes 5–10 minutes the first time. Watch progress:

kubectl get applications -n argocd -w

To skip a tier (rare, e.g. a scratch cluster with no prod tenant), apply the bootstrap and then delete the unwanted top-level app:

kubectl delete application dev -n argocd     # or: prod

platform is required: uninstalling it takes down ingress, TLS, monitoring, and secret syncing for every tenant.

See Operations → Bootstrap for the full cluster-rebuild walkthrough.

The three apps

kubectl apply -f k8s/envs/bootstrap.yaml
↓ creates three top-level Application CRs
Application: platform
Application: dev
Application: prod
↓ directory.recurse: true: each app discovers sub-Applications
platform/* sub-Apps traefik, cert-manager, monitoring, …
dev/* sub-Apps postgres, redis, tomoda overlay
prod/* sub-Apps postgres, redis, tomoda overlay
App-of-apps: one bootstrap file creates three parents; each parent recurses into its directory and picks up the children it finds. After this, nothing is kubectl apply'd by hand; Argo CD owns reconciliation.
App Source path Scope Skippable?
platform k8s/envs/platform/ Cluster-wide system services, required by both tenants No
dev k8s/envs/dev/ Dev tenant data plane (small PVCs, no backups) Yes: kubectl delete application dev -n argocd
prod k8s/envs/prod/ Prod tenant data plane (production-sized PVCs, daily backups) Yes, rarely useful

What each contains

platform brings up Argo CD itself, cert-manager, External Secrets Operator, Traefik, External-DNS, oauth2-proxy, the CNPG operator and its Barman Cloud backup plugin, the Prometheus / Grafana / Alertmanager stack, Prometheus Adapter, Loki, Tempo, Pyrra (SLO controller), blackbox-exporter, cloudflared, Photon, the embeddings server, NATS, Xavia OTA, the ARC controller + runner pools (arc-runners, arc-runners-db, arc-runners-trusted), Trivy Operator and the Trivy issue reporter. Each subdirectory under k8s/envs/platform/ carries its own Application CR. See System services for per-service docs.

dev brings up postgres-dev (CNPG), redis-dev (Bitnami), pgAdmin, RedisInsight, and the tomoda dev overlay. Sized to validate changes cheaply; see the Environments matrix.

prod brings up postgres-prod, redis-prod, pgAdmin, RedisInsight, and the tomoda prod overlay. Real PVCs, daily Postgres backups, HPA-driven app scale-out.

Recurse + exclude

Each top-level app uses directory.recurse: true plus an exclude pattern so it doesn't double-apply manifests its child apps already own:

App exclude pattern Why
platform {manifests/**,**/manifests.yaml} The manifests/ subtree (alerting rules, ESO configs, Image Updater, SLOs) is owned by the sys-resources Application, applied separately so Helm-installed CRDs land first. Per-service manifests.yaml files belong to that service's own Application
dev {**/manifests.yaml,**/manifests/**} Sub-apps (postgres, redis-insight, etc.) own their manifests.yaml / manifests/ via their own Application
prod {**/manifests.yaml,**/manifests/**} Same as dev

These trees hold only Applications and raw manifests, never a kustomization.yaml: a recursing directory source would apply it as an object and fail every sync. A kustomize app lives under k8s/apps/<app>/, and its application.yaml in the env tree points source.path there with no directory: block, so Argo detects kustomize. tomoda, trivy-issue-reporter and grafana-dashboards follow this layout.

Image Updater

argocd-image-updater (v1.x) runs in the argocd namespace (manifest at k8s/envs/platform/manifests/image-updater.yaml) and:

  • Lists GHCR tags with the argocd/ghcr-image-updater credential (ExternalSecret over tomoda-ghcr-read-credentials).
  • Runs one ImageUpdater CR, tomoda-dev, which follows the newest dev-* tag.
  • Writes updates back with the argocd method (parameters on the Application, no git commits). Argo CD then rolls the Deployments.

Prod is not tracked: its tag is pinned in git and moved by the deploy(prod) PR the tomoda release opens. See Operations → Deploy.

Post-deploy smoke test

k8s/apps/tomoda/base/smoke-test.yaml is a PostSync hook Job: after each sync of tomoda-dev / tomoda-prod, once the app is healthy, it checks /health, the web app and the auth middleware from inside the namespace (NetworkPolicy allows app: tomoda-smoke-test). Prod also logs in with the synthetic probe account (smoke-test-probe ExternalSecret) and calls /auth/profile. A failing Job marks the sync Failed, which fires the ArgoCDSyncFailed alert.

OAuth gating on the UI

The Argo CD UI is exposed at argo-app.tomoda.life, fronted by Traefik with two layers of auth:

  1. oauth2-proxy (Traefik middlewares on the Ingress at k8s/envs/platform/argocd-ingress/ingress.yaml) gates the host at the edge. Only @tomoda.life Google accounts get past it.
  2. Argo CD Dex: once past oauth2-proxy, Dex maps the Google identity into an Argo CD RBAC role.

Sign in once with Google; oauth2-proxy plants a .tomoda.life cookie and Dex picks up the identity. See oauth2-proxy for proxy-side details.

RBAC and access hardening

Two Helm values in k8s/envs/platform/argocd/values.yaml (configs.cm / configs.rbac) lock down what an authenticated user can do:

  • exec.enabled = "false": the browser Web Terminal (exec into pods from the Argo UI) is disabled. No shell into workloads through Argo CD.
  • policy.default = "role:readonly": the default SSO policy is read-only. Every mapped Google identity can view but not mutate. Write and admin actions require an explicit grant in policy.csv.

Sync workflow

  1. Edit and push to main.
  2. Argo CD polls (default ~3 min) and detects the diff. Click Refresh in the UI to force an immediate poll.
  3. Sync runs automatically: pruned resources deleted, new resources created, drifted resources patched.
  4. Self-heal pins the cluster to whatever is in Git.

Manual sync: argocd app sync <name> or the UI's Sync button, when you don't want to wait for the next poll.