Argo CD¶
Argo CD is the only thing that writes to the cluster. Every namespace, Helm release, Deployment and Secret is the output of an Application sync. Auto-sync, auto-prune and self-heal are on by default, so anything not in this repo gets reverted within minutes.
One k3s cluster, three logical tiers wired up by Argo CD: platform (cluster-wide system services), dev (the dev tenant data plane), prod (the prod tenant data plane). See Environments for the full tenancy model and tier sizing.
Argo CD itself is a Helm release with values in k8s/envs/platform/argocd/values.yaml (Dex + Google SSO, RBAC, metrics). scripts/ovh/bootstrap-argocd.sh installs it once; after that it manages itself (below). This page covers the GitOps layer on top.
Self-management¶
k8s/envs/platform/argocd/application.yaml is an Application for Argo CD's own chart (argo-cd 10.12.1, release argocd, namespace argocd, same values file). Because the release name and namespace match the bootstrap install, it adopts those resources rather than creating new ones.
- Upgrade: bump
targetRevisionin the Application (andARGOCD_CHART_VERSIONin the bootstrap script, so a rebuild installs the same version). Values changes are ordinary PRs. - Safety:
prune: falseand no resources finalizer, so Argo CD never deletes its own components, and deleting the Application leaves Argo CD running.ServerSideApply=truehandles the large CRDs. - Not managed: the two hand-made Secrets (
argocd/repo-devops,external-secrets/gcpsm-credentials) are outside the chart. - Metrics: the chart's own ServiceMonitors are off (it only renders them when the monitoring CRDs exist at install time);
k8s/envs/platform/manifests/argocd-servicemonitors.yamldefines them instead.
Bootstrap¶
One file, applied once by scripts/ovh/bootstrap-argocd.sh after Argo CD is installed (see Bootstrap):
kubectl apply -f k8s/envs/bootstrap.yaml
k8s/envs/bootstrap.yaml creates three top-level Applications: platform, dev, prod. Each recurses through its respective directory under k8s/envs/ and picks up the sub-Applications it finds.
Reconciliation takes 5–10 minutes the first time. Watch progress:
kubectl get applications -n argocd -w
To skip a tier (rare, e.g. a scratch cluster with no prod tenant), apply the bootstrap and then delete the unwanted top-level app:
kubectl delete application dev -n argocd # or: prod
platform is required: uninstalling it takes down ingress, TLS, monitoring, and secret syncing for every tenant.
See Operations → Bootstrap for the full cluster-rebuild walkthrough.
The three apps¶
Application CRsdirectory.recurse: true: each app discovers sub-Applicationskubectl apply'd by hand; Argo CD owns reconciliation.| App | Source path | Scope | Skippable? |
|---|---|---|---|
platform |
k8s/envs/platform/ |
Cluster-wide system services, required by both tenants | No |
dev |
k8s/envs/dev/ |
Dev tenant data plane (small PVCs, no backups) | Yes: kubectl delete application dev -n argocd |
prod |
k8s/envs/prod/ |
Prod tenant data plane (production-sized PVCs, daily backups) | Yes, rarely useful |
What each contains¶
platform brings up Argo CD itself, cert-manager, External Secrets Operator, Traefik, External-DNS, oauth2-proxy, the CNPG operator and its Barman Cloud backup plugin, the Prometheus / Grafana / Alertmanager stack, Prometheus Adapter, Loki, Tempo, Pyrra (SLO controller), blackbox-exporter, cloudflared, Photon, the embeddings server, NATS, Xavia OTA, the ARC controller + runner pools (arc-runners, arc-runners-db, arc-runners-trusted), Trivy Operator and the Trivy issue reporter. Each subdirectory under k8s/envs/platform/ carries its own Application CR. See System services for per-service docs.
dev brings up postgres-dev (CNPG), redis-dev (Bitnami), pgAdmin, RedisInsight, and the tomoda dev overlay. Sized to validate changes cheaply; see the Environments matrix.
prod brings up postgres-prod, redis-prod, pgAdmin, RedisInsight, and the tomoda prod overlay. Real PVCs, daily Postgres backups, HPA-driven app scale-out.
Recurse + exclude¶
Each top-level app uses directory.recurse: true plus an exclude pattern so it doesn't double-apply manifests its child apps already own:
| App | exclude pattern |
Why |
|---|---|---|
platform |
{manifests/**,**/manifests.yaml} |
The manifests/ subtree (alerting rules, ESO configs, Image Updater, SLOs) is owned by the sys-resources Application, applied separately so Helm-installed CRDs land first. Per-service manifests.yaml files belong to that service's own Application |
dev |
{**/manifests.yaml,**/manifests/**} |
Sub-apps (postgres, redis-insight, etc.) own their manifests.yaml / manifests/ via their own Application |
prod |
{**/manifests.yaml,**/manifests/**} |
Same as dev |
These trees hold only Applications and raw manifests, never a kustomization.yaml: a recursing directory source would apply it as an object and fail every sync. A kustomize app lives under k8s/apps/<app>/, and its application.yaml in the env tree points source.path there with no directory: block, so Argo detects kustomize. tomoda, trivy-issue-reporter and grafana-dashboards follow this layout.
Image Updater¶
argocd-image-updater (v1.x) runs in the argocd namespace (manifest at k8s/envs/platform/manifests/image-updater.yaml) and:
- Lists GHCR tags with the
argocd/ghcr-image-updatercredential (ExternalSecret overtomoda-ghcr-read-credentials). - Runs one
ImageUpdaterCR,tomoda-dev, which follows the newestdev-*tag. - Writes updates back with the
argocdmethod (parameters on the Application, no git commits). Argo CD then rolls the Deployments.
Prod is not tracked: its tag is pinned in git and moved by the deploy(prod) PR the tomoda release opens. See Operations → Deploy.
Post-deploy smoke test¶
k8s/apps/tomoda/base/smoke-test.yaml is a PostSync hook Job: after each sync of tomoda-dev / tomoda-prod, once the app is healthy, it checks /health, the web app and the auth middleware from inside the namespace (NetworkPolicy allows app: tomoda-smoke-test). Prod also logs in with the synthetic probe account (smoke-test-probe ExternalSecret) and calls /auth/profile. A failing Job marks the sync Failed, which fires the ArgoCDSyncFailed alert.
OAuth gating on the UI¶
The Argo CD UI is exposed at argo-app.tomoda.life, fronted by Traefik with two layers of auth:
- oauth2-proxy (Traefik middlewares on the Ingress at
k8s/envs/platform/argocd-ingress/ingress.yaml) gates the host at the edge. Only@tomoda.lifeGoogle accounts get past it. - Argo CD Dex: once past oauth2-proxy, Dex maps the Google identity into an Argo CD RBAC role.
Sign in once with Google; oauth2-proxy plants a .tomoda.life cookie and Dex picks up the identity. See oauth2-proxy for proxy-side details.
RBAC and access hardening¶
Two Helm values in k8s/envs/platform/argocd/values.yaml (configs.cm / configs.rbac) lock down what an authenticated user can do:
exec.enabled = "false": the browser Web Terminal (exec into pods from the Argo UI) is disabled. No shell into workloads through Argo CD.policy.default = "role:readonly": the default SSO policy is read-only. Every mapped Google identity can view but not mutate. Write and admin actions require an explicit grant inpolicy.csv.
Sync workflow¶
- Edit and push to
main. - Argo CD polls (default ~3 min) and detects the diff. Click Refresh in the UI to force an immediate poll.
- Sync runs automatically: pruned resources deleted, new resources created, drifted resources patched.
- Self-heal pins the cluster to whatever is in Git.
Manual sync: argocd app sync <name> or the UI's Sync button, when you don't want to wait for the next poll.