External Secrets¶

Secrets never live in Git. The External Secrets Operator (ESO) reads values from GCP Secret Manager and AWS Secrets Manager and projects them into Kubernetes Secret objects that pods envFrom at runtime.

Installed by k8s/envs/dev/sys/external-secrets/application.yaml. The cluster-scoped store definitions live in k8s/envs/dev/sys/manifests/external-secrets-config.yaml and are deployed by the separate sys-resources Application.

Chart and source¶

Field	Value
Helm chart	`external-secrets`
Repository	`https://charts.external-secrets.io`
Version	`0.10.3`
Destination namespace	`external-secrets` (created by Argo CD)
Argo CD Application	`external-secrets`

The chart installs the ESO controller, the webhook, and the CRDs (ClusterSecretStore, SecretStore, ExternalSecret, PushSecret).

ClusterSecretStores¶

Two stores are registered cluster-wide. Both are ClusterSecretStore (not namespaced SecretStore) so any namespace can reference them.

`gsm-tomoda` — GCP Secret Manager¶

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gsm-tomoda
spec:
  provider:
    gcpsm:
      projectID: development-485000

No auth block — ESO authenticates via Workload Identity using the ESO controller's KSA, which is bound to a GCP SA with roles/secretmanager.secretAccessor on the project. That binding is set up in Terraform alongside the cluster (see Infrastructure → GCP).

This store backs every application secret in the cluster — JWT secret, OAuth client secret, Stripe keys, email API keys, Postgres password, Redis password. The full inventory is in Secrets Management.

`aws-sm-tomoda` — AWS Secrets Manager¶

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-sm-tomoda
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-northeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-eso-credentials
            namespace: external-secrets
            key: access-key
          secretAccessKeySecretRef:
            name: aws-eso-credentials
            namespace: external-secrets
            key: secret-access-key

Unlike GCP, AWS has no Workload Identity equivalent for GKE pods, so this store authenticates with a static IAM user credential pair. The user is tomoda-eso-reader-{env} (provisioned by AWS Terraform), and the credentials are stored in the K8s Secret aws-eso-credentials in the external-secrets namespace.

That bootstrap secret is not in Git — it has to be created by hand after running the AWS Terraform module:

kubectl create secret generic aws-eso-credentials -n external-secrets \
  --from-literal=access-key=<ESO_READER_ACCESS_KEY_ID> \
  --from-literal=secret-access-key=<ESO_READER_SECRET_ACCESS_KEY>

This store is used for things that only AWS can produce — currently the S3 uploader's IAM credentials.

Per-app ExternalSecret CRs¶

Once the stores exist, any namespace can pull values:

# k8s/envs/dev/sys/manifests/external-secrets-config.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: google-oauth-secret
  namespace: monitoring
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: gsm-tomoda
  target:
    name: google-oauth-credentials
    creationPolicy: Owner
  data:
    - secretKey: client-secret
      remoteRef:
        key: oauth-client-secret

The result is a regular K8s Secret named google-oauth-credentials in the monitoring namespace, refreshed hourly from GCP SM. Grafana's Deployment then valueFrom: secretKeyRefs it as an environment variable.

The same pattern is used by the tomoda app's backend-secrets-{dev,prod} Secrets (under k8s/envs/{dev,prod}/middleware/tomoda/).

Refresh and rotation¶

Refresh interval is per-ExternalSecret; the convention is 1h. ESO re-reads the upstream value and patches the K8s Secret if it changed.
Rotating a value upstream (in GCP SM or AWS SM) propagates into the cluster within the refresh window. Pods consuming the Secret via envFrom will not pick up the new value until they restart — they only read env vars at process start. Forcing a rollout is the standard remedy: kubectl rollout restart deployment/<name>.
Force-refreshing an ExternalSecret immediately: annotate it with force-sync or delete the target Secret (ESO recreates it on the next reconcile).

Operational notes¶

Webhook dependency. The ESO controller's mutating webhook is required for ExternalSecret admission. If the controller is unhealthy, new ExternalSecrets cannot be created — existing K8s Secrets remain in place but stop refreshing.
The bootstrap secret is a single point of failure for AWS. If aws-eso-credentials is missing or its keys are wrong, every AWS-backed ExternalSecret fails. Existing K8s Secrets stay intact (creationPolicy: Owner means ESO won't delete on auth failure) but new ones can't be created.
No secret values are logged. ESO logs key names and store references only.

For the secret inventory, see Secrets Management.