Skip to content

Argo CD

Argo CD is the only thing that writes to the cluster. Every namespace, every Helm release, every Deployment, every Secret is the output of an Argo CD Application sync. If something is running in the cluster that is not in this repo, it is drift — and Argo CD will overwrite it on the next reconciliation (every Application here has selfHeal: true).

This page is the operational companion to the platform-level page at Infrastructure → GCP → Argo CD, which covers the underlying GCP service account, IAM, and how Argo CD itself was installed.

App-of-apps bootstrap

Two bootstrap manifests sit at the top of the tree and reference everything else. Each one declares Argo CD Application resources that themselves point Argo CD at folders full of more Application resources — that is the "app of apps" pattern.

k8s/envs/dev/bootstrap.yaml

# Two Applications:
#   dev-sys         -> path: k8s/envs/dev/sys (recurse, exclude manifests/)
#   dev-middleware  -> path: k8s/envs/dev/middleware

dev-sys recurses into every subfolder of k8s/envs/dev/sys/ and applies any Application CRs it finds. That installs Traefik, cert-manager, External-DNS, External Secrets, oauth2-proxy, the Prometheus stack, Loki, the CNPG operator, and the ArgoCD UI ingress. The manifests/ subfolder is excluded from this recursion and is instead installed via the sys-resources Application (so Helm-installed CRDs land first, then the manifests that consume them).

dev-middleware recurses into k8s/envs/dev/middleware/ and installs Postgres (CNPG Cluster), Redis, pgAdmin, RedisInsight, and the dev tomoda app.

k8s/envs/prod/bootstrap.yaml

# One Application:
#   prod-middleware -> path: k8s/envs/prod/middleware

There is no prod-sys. Production runs in a separate Kubernetes namespace (prod) on the same cluster as dev, so it consumes the dev system services (one Traefik, one cert-manager, one Prometheus). The prod bootstrap only adds the prod-side data plane and the prod app.

Dev's bootstrap is load-bearing for prod

If you uninstall dev-sys, prod loses its ingress controller, its TLS certs, its monitoring, and its secret syncing. Treat dev-sys as cluster-global infrastructure regardless of its name.

Argo CD Applications

Every subfolder under k8s/envs/{dev,prod}/{middleware,sys}/ contains an application.yaml — an argoproj.io/v1alpha1 Application resource. They come in three flavours:

Source pattern Used when Example
chart + repoURL (Helm) Upstream Helm chart, no overrides or trivial parameters cert-manager/application.yaml, external-secrets/application.yaml
Multi-source: chart + git ref: values Upstream Helm chart with values.yaml checked into this repo traefik, external-dns, oauth2-proxy, monitoring, loki
Plain repoURL + path Pure Kustomize / raw manifests sys-resources, argocd-ingress, the app Application CRs

Every Application in this repo enables auto-sync, auto-prune, and self-heal:

syncPolicy:
  automated:
    prune: true
    selfHeal: true

That means: a delete in Git deletes the resource in the cluster, and a manual kubectl edit against an Argo-managed resource gets reverted within minutes. There is no "scratch" workflow — everything edits flow through Git.

Sync workflow

  1. Edit and push — change a Helm value, a manifest, or an Application CR; push to main.
  2. Argo CD polls the repo (default ~3 minute interval) and detects the diff. You can also click Refresh in the UI to force an immediate poll.
  3. Sync happens automatically. Pruned resources are deleted, new resources created, changed resources patched.
  4. Self-heal keeps the cluster pinned to whatever is in Git, even if something out-of-band drifts.

Manual sync is still available from the UI or argocd app sync <name> — useful when you want an immediate reconcile without waiting for the poll.

Image Updater integration

The app image tags are not edited by humans. argocd-image-updater runs as a Deployment in the argocd namespace (defined in k8s/envs/dev/sys/manifests/image-updater.yaml) and:

  • Authenticates to Artifact Registry via the GKE metadata server, using the argocd-image-updater KSA bound to the GCP SA argocd-image-updater-sa@development-485000.iam.gserviceaccount.com (Workload Identity).
  • Watches the tomoda-dev-repo and tomoda-prod-repo repositories in asia-east1-docker.pkg.dev.
  • When a new image tag matches the configured constraint on a tomoda Application, the updater commits a Kustomize image-tag update back to this repo.
  • That Git commit triggers the standard Argo CD reconciliation flow above — the tomoda Application syncs, and the new image rolls out.

The image-updater itself is part of dev-sys (via the sys-resources Application). Its registry config lives in the argocd-image-updater-config ConfigMap.

OAuth gating on the UI

The Argo CD UI is not exposed by Argo's built-in HTTP service directly — it sits behind Traefik through oauth2-proxy. Two layers of authentication apply:

  1. Traefik + oauth2-proxy (k8s/envs/dev/sys/argocd-ingress/ingress.yaml) — the argo-app.tomoda.life Ingress carries two Traefik middlewares: sys-oauth2-proxy-auth (forwards to /oauth2/auth and requires a valid session) and sys-oauth2-proxy-errors (catches 401/403 and redirects to Google login). The oauth2-proxy configuration restricts logins to @tomoda.life Google accounts.
  2. Argo CD's own Dex — once you are past oauth2-proxy, Argo CD's Dex also authenticates against Google OAuth with the same tomoda.life domain restriction, and maps you into an Argo CD RBAC role.

In practice you sign in once with Google (oauth2-proxy plants a .tomoda.life cookie) and Dex picks up the identity. The double layer is intentional — oauth2-proxy stops unauthenticated traffic from ever reaching Argo's backend, and Dex maps identities to Argo permissions.

See System → oauth2-proxy for the proxy-side details.

Bootstrapping a fresh cluster

If you ever rebuild the cluster from scratch (see Disaster Recovery):

  1. Provision the GKE cluster via Terraform (infrastructure/gcp/gke.tf).
  2. Install Argo CD itself — see Infrastructure → GCP → Argo CD for the install steps and the initial admin password retrieval.
  3. Create the AWS ESO auth secret manually (it is not in Git): 
    kubectl create secret generic aws-eso-credentials -n external-secrets \
  --from-literal=access-key=<ESO_READER_ACCESS_KEY_ID> \
  --from-literal=secret-access-key=<ESO_READER_SECRET_ACCESS_KEY>
  4. Apply the dev bootstrap: 
    kubectl apply -f k8s/envs/dev/bootstrap.yaml
    Argo CD picks up dev-sys and dev-middleware, recurses through every subfolder, and reconciles the entire dev environment.
  5. Once dev-sys is healthy (all system services running), apply the prod bootstrap: 
    kubectl apply -f k8s/envs/prod/bootstrap.yaml
  6. Wait. Initial reconciliation takes ~5–10 minutes depending on Helm pulls. Watch argocd app list until everything is Healthy / Synced.

After that, no manual kubectl apply should ever be needed — every change goes through Git.