External Secrets¶
Secrets never live in Git. The External Secrets Operator (ESO) reads values from GCP Secret Manager and projects them into Kubernetes Secret objects that pods envFrom at runtime.
Installed by k8s/envs/platform/external-secrets/application.yaml. The cluster-scoped store definitions live in k8s/envs/platform/manifests/external-secrets-config.yaml and are deployed by the separate sys-resources Application.
Chart and source¶
| Field | Value |
|---|---|
| Helm chart | external-secrets |
| Repository | https://charts.external-secrets.io |
| Version | 2.13.0 |
| Destination namespace | external-secrets (created by Argo CD) |
| Argo CD Application | external-secrets |
The chart installs the ESO controller, the webhook, and the CRDs (ClusterSecretStore, SecretStore, ExternalSecret, PushSecret).
Secret store¶
One cluster-wide store, gsm-tomoda:
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: gsm-tomoda
spec:
provider:
gcpsm:
projectID: development-485000
auth:
secretRef:
secretAccessKeySecretRef:
name: gcpsm-credentials
namespace: external-secrets
key: credentials.json
gcpsm-credentials holds a key for the eso-reader GCP service account (infrastructure/gcp/service_accounts.tf), which has only roles/secretmanager.secretAccessor. It is the one Secret created by hand, by scripts/ovh/bootstrap-argocd.sh (see Bootstrap).
This store backs every synced secret in the cluster: app secrets (JWT, OAuth, payments, email, Postgres, Redis), R2 tokens, the asset signing key, the GHCR read token, Cloudflare tokens, the ARC GitHub App. The full inventory is in Secrets Management.
Per-app ExternalSecret CRs¶
Once the stores exist, any namespace can pull values:
# k8s/envs/platform/manifests/external-secrets-config.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: google-oauth-secret
namespace: monitoring
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: gsm-tomoda
target:
name: google-oauth-credentials
creationPolicy: Owner
data:
- secretKey: client-secret
remoteRef:
key: tomoda-google-client-secret
The result is a regular K8s Secret named google-oauth-credentials in the monitoring namespace, refreshed hourly from GCP SM. Grafana's Deployment then valueFrom: secretKeyRefs it as an environment variable.
The same pattern is used by the tomoda app's backend-secrets-{dev,prod} and object-storage-credentials (in k8s/apps/tomoda/overlays/{dev,prod}/external-secret.yaml). The app namespaces also run a namespaced SecretStore with the Kubernetes provider to mirror the CNPG CA (postgres-ca); see Postgres.
Refresh and rotation¶
- Refresh interval is per-ExternalSecret; the convention is
1h. ESO re-reads the upstream value and patches the K8s Secret if it changed. - Rotating a value upstream (in GCP SM) propagates into the cluster within the refresh window. Pods consuming the Secret via
envFromdon't pick up the new value until they restart (env vars are read at process start). Forcing a rollout is the standard remedy:kubectl rollout restart deployment/<name>. - Force-refreshing an ExternalSecret immediately: annotate it with
force-syncor delete the target Secret (ESO recreates it on the next reconcile).
Operational notes¶
- Webhook dependency. The ESO controller's mutating webhook is required for
ExternalSecretadmission. If the controller is unhealthy, new ExternalSecrets cannot be created; existing K8s Secrets remain in place but stop refreshing. - The bootstrap key is a single point of failure. If
external-secrets/gcpsm-credentialsis missing or its key was deleted in GCP, every ExternalSecret fails. Existing K8s Secrets stay intact (ESO doesn't delete on auth failure) but stop refreshing, and new ones can't be created. - No secret values are logged. ESO logs key names and store references only.
For the secret inventory, see Secrets Management.