External Secrets¶

Secrets never live in Git. The External Secrets Operator (ESO) reads values from GCP Secret Manager and AWS Secrets Manager and projects them into Kubernetes Secret objects that pods envFrom at runtime.

Installed by k8s/envs/platform/external-secrets/application.yaml. The cluster-scoped store definitions live in k8s/envs/platform/manifests/external-secrets-config.yaml and are deployed by the separate sys-resources Application.

Chart and source¶

Field	Value
Helm chart	`external-secrets`
Repository	`https://charts.external-secrets.io`
Version	`0.10.3`
Destination namespace	`external-secrets` (created by Argo CD)
Argo CD Application	`external-secrets`

The chart installs the ESO controller, the webhook, and the CRDs (ClusterSecretStore, SecretStore, ExternalSecret, PushSecret).

Secret stores¶

One cluster-wide store (gsm-tomoda, GCP) plus a namespace-scoped AWS store per tenant.

`gsm-tomoda` — GCP Secret Manager (cluster-wide)¶

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gsm-tomoda
spec:
  provider:
    gcpsm:
      projectID: development-485000

No auth block — ESO authenticates via Workload Identity using the ESO controller's KSA, which is bound to a GCP SA with roles/secretmanager.secretAccessor on the project. That binding is set up in Terraform alongside the cluster (see Infrastructure → GCP).

This store backs every application secret in the cluster — JWT secret, OAuth client secret, Stripe keys, email API keys, Postgres password, Redis password. The full inventory is in Secrets Management.

`aws-sm-<env>` — AWS Secrets Manager (per namespace)¶

Each tenant (tomoda-dev, tomoda-prod) owns a namespace-scoped SecretStore — defined in k8s/apps/tomoda/overlays/<env>/external-secret.yaml, not the cluster-wide manifest.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-sm-dev
  namespace: tomoda-dev
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-eso-credentials-dev   # same namespace
            key: access-key
          secretAccessKeySecretRef:
            name: aws-eso-credentials-dev
            key: secret-access-key

AWS has no Workload Identity equivalent for GKE pods, so this store needs a static IAM key (user tomoda-eso-reader-<env>, provisioned by AWS Terraform). But that key is not hand-created — it lives in GCP SM (tomoda-aws-eso-*-<env>) and is projected into the namespace by an ExternalSecret running against gsm-tomoda:

kind: ExternalSecret
metadata:
  name: aws-eso-credentials-dev
  namespace: tomoda-dev
spec:
  secretStoreRef: { kind: ClusterSecretStore, name: gsm-tomoda }
  target: { name: aws-eso-credentials-dev }
  data:
    - secretKey: access-key
      remoteRef: { key: tomoda-aws-eso-access-key-dev }
    - secretKey: secret-access-key
      remoteRef: { key: tomoda-aws-eso-secret-key-dev }

So the whole chain bootstraps from Workload Identity — no kubectl create secret, everything GitOps. Per-namespace stores keep a key rotation or compromise scoped to one tenant. This store backs things only AWS can produce — currently the S3 uploader's IAM credentials. Full flow in Secrets Management.

Per-app ExternalSecret CRs¶

Once the stores exist, any namespace can pull values:

# k8s/envs/platform/manifests/external-secrets-config.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: google-oauth-secret
  namespace: monitoring
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: gsm-tomoda
  target:
    name: google-oauth-credentials
    creationPolicy: Owner
  data:
    - secretKey: client-secret
      remoteRef:
        key: oauth-client-secret

The result is a regular K8s Secret named google-oauth-credentials in the monitoring namespace, refreshed hourly from GCP SM. Grafana's Deployment then valueFrom: secretKeyRefs it as an environment variable.

The same pattern is used by the tomoda app's backend-secrets-{dev,prod} Secrets (under k8s/envs/{dev,prod}/tomoda/).

Refresh and rotation¶

Refresh interval is per-ExternalSecret; the convention is 1h. ESO re-reads the upstream value and patches the K8s Secret if it changed.
Rotating a value upstream (in GCP SM or AWS SM) propagates into the cluster within the refresh window. Pods consuming the Secret via envFrom will not pick up the new value until they restart — they only read env vars at process start. Forcing a rollout is the standard remedy: kubectl rollout restart deployment/<name>.
Force-refreshing an ExternalSecret immediately: annotate it with force-sync or delete the target Secret (ESO recreates it on the next reconcile).

Operational notes¶

Webhook dependency. The ESO controller's mutating webhook is required for ExternalSecret admission. If the controller is unhealthy, new ExternalSecrets cannot be created — existing K8s Secrets remain in place but stop refreshing.
The bootstrap secret is a single point of failure for AWS. If aws-eso-credentials is missing or its keys are wrong, every AWS-backed ExternalSecret fails. Existing K8s Secrets stay intact (creationPolicy: Owner means ESO won't delete on auth failure) but new ones can't be created.
No secret values are logged. ESO logs key names and store references only.

For the secret inventory, see Secrets Management.