Secrets Management

Mental Model

A Secret is a namespaced Kubernetes object that stores key-value data. Workloads consume it through environment variables, mounted files, image pull credentials, or controllers such as cert-manager and External Secrets Operator. The API server stores the object, etcd persists it, and the kubelet materializes it onto nodes for Pods that reference it.

The operational question is not “is this base64?” The real question is: who can read it, where is it stored, how does it rotate, and can it leak through Git, logs, tickets, shell history, dashboards, or overly broad RBAC?

Secret Types

Native Secret From Scratch

Use stringData when authoring a Secret manifest manually because it accepts plain strings and the API server writes base64-encoded data. Do not commit real plaintext credentials to Git.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: app
type: Opaque
stringData:
  username: app_user
  password: change-me-in-real-secret-store

kubectl create secret generic db-credentials -n app \
  --from-literal=username=app_user \
  --from-literal=password='REPLACE_ME'

kubectl get secret db-credentials -n app -o yaml
kubectl describe secret db-credentials -n app

# Decode only when necessary, and avoid pasting output into tickets or chat.
kubectl get secret db-credentials -n app -o jsonpath='{.data.password}' | base64 -d; echo

Consume Secrets

Prefer mounted files when the app can read configuration from files, especially for values that may rotate. Environment variables are simple but only update when the container starts.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-api
  namespace: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: api
          image: registry.example.com/web-api:1.2.3
          volumeMounts:
            - name: db-credentials
              mountPath: /run/secrets/db
              readOnly: true
          env:
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: username
      volumes:
        - name: db-credentials
          secret:
            secretName: db-credentials
            items:
              - key: password
                path: password

Rotation Behavior

kubectl create secret generic db-credentials -n app \
  --from-literal=username=app_user \
  --from-literal=password='NEW_VALUE' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deploy/web-api -n app
kubectl rollout status deploy/web-api -n app

External Secret Stores

In production, Kubernetes often should not be the original source of secret material. External Secrets Operator, Secrets Store CSI Driver, Vault Agent Injector, SOPS, Sealed Secrets, or cloud-native integrations let you keep source secrets in a system designed for audit, rotation, and access control.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/web-api/db
        property: password

RBAC Risk

Secret access is high privilege. A subject that can get or list Secrets may be able to read database passwords, API keys, TLS private keys, image pull tokens, and app credentials.

kubectl auth can-i get secrets -n app
kubectl auth can-i list secrets -n app
kubectl auth can-i get secrets --all-namespaces

kubectl auth can-i get secrets -n app \
  --as=system:serviceaccount:app:web-api

GitOps And Helm

• 1Do not commit plaintext secrets: avoid real values in Helm values.yaml, Kustomize overlays, or raw manifests.
• 2Prefer references: Git should usually contain ExternalSecret, SOPS-encrypted Secret, or SealedSecret resources.
• 3Watch rendered output: Helm template output, CI logs, ArgoCD diffs, and failed manifests can expose values.
• 4Separate config from secret: use ConfigMaps for non-sensitive settings and Secrets only for sensitive values.

Debugging Workflow

NS=<namespace>
SECRET=<secret-name>
DEPLOY=<deployment>

kubectl get secret "$SECRET" -n "$NS" -o yaml
kubectl describe secret "$SECRET" -n "$NS"
kubectl get deploy "$DEPLOY" -n "$NS" -o yaml | grep -A20 -E 'secretKeyRef|secretName|imagePullSecrets'
kubectl describe pod -n "$NS" -l app=<app-label>
kubectl logs -n "$NS" deploy/"$DEPLOY" --tail=100

# External Secrets Operator.
kubectl get externalsecret,secretstore,clustersecretstore -A
kubectl describe externalsecret "$SECRET" -n "$NS"

Symptom To Cause

Production Patterns

• 1Enable encryption at rest: use provider-supported etcd/envelope encryption.
• 2Restrict RBAC: avoid broad get/list/watch secrets permissions.
• 3Use external source of truth: keep rotation and audit in Vault, cloud secret manager, or approved platform tooling.
• 4Rotate with evidence: know which Deployments restarted and which clients picked up new values.
• 5Keep cert lifecycle separate: TLS Secret storage belongs here; certificate issuance and renewal belong on Certificate Management.

Safe Change Pattern

1. Identify the true source of the secret: cloud store, Vault, SOPS, SealedSecret, ESO, Helm, or manual Secret.
2. Update the source of truth, not just the live Kubernetes Secret, unless it is an approved emergency action.
3. Verify the synced Kubernetes Secret metadata, keys, and consuming workload spec.
4. Restart or reload workloads that read values only at startup.
5. After rotation, test the app path and revoke the old credential where the external system supports it.

Related Pages