Skip to content

External Secrets

Secrets never live in Git. The External Secrets Operator (ESO) reads values from GCP Secret Manager and projects them into Kubernetes Secret objects that pods envFrom at runtime.

Installed by k8s/envs/platform/external-secrets/application.yaml. The cluster-scoped store definitions live in k8s/envs/platform/manifests/external-secrets-config.yaml and are deployed by the separate sys-resources Application.

Chart and source

Field Value
Helm chart external-secrets
Repository https://charts.external-secrets.io
Version 2.13.0
Destination namespace external-secrets (created by Argo CD)
Argo CD Application external-secrets

The chart installs the ESO controller, the webhook, and the CRDs (ClusterSecretStore, SecretStore, ExternalSecret, PushSecret).

Secret store

One cluster-wide store, gsm-tomoda:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: gsm-tomoda
spec:
  provider:
    gcpsm:
      projectID: development-485000
      auth:
        secretRef:
          secretAccessKeySecretRef:
            name: gcpsm-credentials
            namespace: external-secrets
            key: credentials.json

gcpsm-credentials holds a key for the eso-reader GCP service account (infrastructure/gcp/service_accounts.tf), which has only roles/secretmanager.secretAccessor. It is the one Secret created by hand, by scripts/ovh/bootstrap-argocd.sh (see Bootstrap).

This store backs every synced secret in the cluster: app secrets (JWT, OAuth, payments, email, Postgres, Redis), R2 tokens, the asset signing key, the GHCR read token, Cloudflare tokens, the ARC GitHub App. The full inventory is in Secrets Management.

Per-app ExternalSecret CRs

Once the stores exist, any namespace can pull values:

# k8s/envs/platform/manifests/external-secrets-config.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: google-oauth-secret
  namespace: monitoring
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: gsm-tomoda
  target:
    name: google-oauth-credentials
    creationPolicy: Owner
  data:
    - secretKey: client-secret
      remoteRef:
        key: tomoda-google-client-secret

The result is a regular K8s Secret named google-oauth-credentials in the monitoring namespace, refreshed hourly from GCP SM. Grafana's Deployment then valueFrom: secretKeyRefs it as an environment variable.

The same pattern is used by the tomoda app's backend-secrets-{dev,prod} and object-storage-credentials (in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml). The app namespaces also run a namespaced SecretStore with the Kubernetes provider to mirror the CNPG CA (postgres-ca); see Postgres.

Refresh and rotation

  • Refresh interval is per-ExternalSecret; the convention is 1h. ESO re-reads the upstream value and patches the K8s Secret if it changed.
  • Rotating a value upstream (in GCP SM) propagates into the cluster within the refresh window. Pods consuming the Secret via envFrom don't pick up the new value until they restart (env vars are read at process start). Forcing a rollout is the standard remedy: kubectl rollout restart deployment/<name>.
  • Force-refreshing an ExternalSecret immediately: annotate it with force-sync or delete the target Secret (ESO recreates it on the next reconcile).

Operational notes

  • Webhook dependency. The ESO controller's mutating webhook is required for ExternalSecret admission. If the controller is unhealthy, new ExternalSecrets cannot be created; existing K8s Secrets remain in place but stop refreshing.
  • The bootstrap key is a single point of failure. If external-secrets/gcpsm-credentials is missing or its key was deleted in GCP, every ExternalSecret fails. Existing K8s Secrets stay intact (ESO doesn't delete on auth failure) but stop refreshing, and new ones can't be created.
  • No secret values are logged. ESO logs key names and store references only.

For the secret inventory, see Secrets Management.