Stateful Storage Patterns — K8s SRE Reference

TL;DR

For a client MySQL setup, understand the raw Kubernetes pattern before using Helm: StatefulSet for stable identity, headless Service for stable Pod DNS, one PVC per replica, Secret for credentials, probes, backup/restore process, and a StorageClass with the right reclaim policy. Helm is useful later, but this page shows what Helm is creating.

When To Use This Pattern

Use a StatefulSet when MySQL runs inside Kubernetes and needs persistent disk attached to a stable Pod identity. For serious production databases, also consider whether the client already has a managed database service, a database operator, or a platform standard that handles backups, replication, failover, and patching.

Pattern	Use For	Main Risk
StatefulSet + PVC	Lab, small internal DBs, controlled client platform pattern.	You own backup, restore, upgrades, failover behavior.
Helm chart	Standardized deploy with values and templates.	Values can hide risky storage defaults.
Operator	Production database lifecycle automation.	CRD/operator complexity and permissions.
Managed DB	Critical production systems.	Network, IAM, credentials, cost, vendor constraints.

MySQL On Kubernetes Shape

Minimal MySQL StatefulSet storage pattern.

MySQL From Scratch

This is a learning and baseline implementation example. Replace storageClassName, credentials handling, resource requests, backup process, and MySQL configuration to match the client standard.

yamlmysql-statefulset.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: data
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-credentials
  namespace: data
type: Opaque
stringData:
  root-password: change-me
  database: appdb
  user: appuser
  password: change-me-too
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: data
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
    - name: mysql
      port: 3306
      targetPort: mysql
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: data
spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: mysql
          image: mysql:8.4
          ports:
            - name: mysql
              containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-credentials
                  key: root-password
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: mysql-credentials
                  key: database
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: mysql-credentials
                  key: user
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-credentials
                  key: password
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
          readinessProbe:
            exec:
              command: ["mysqladmin", "ping", "-h", "127.0.0.1"]
            initialDelaySeconds: 20
            periodSeconds: 10
          livenessProbe:
            exec:
              command: ["mysqladmin", "ping", "-h", "127.0.0.1"]
            initialDelaySeconds: 60
            periodSeconds: 20
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 2Gi
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-retain
        resources:
          requests:
            storage: 20Gi

Verify The Setup

bashmysql-verify.sh

kubectl apply -f mysql-statefulset.yaml
kubectl rollout status statefulset/mysql -n data

kubectl get sts,pod,svc,pvc -n data -o wide
kubectl get pv
kubectl describe pvc data-mysql-0 -n data

# Stable DNS for the first replica.
kubectl run dns-test -n data --rm -it --image=busybox:1.36 -- \
  nslookup mysql-0.mysql.data.svc.cluster.local

# Connect from a temporary MySQL client Pod.
kubectl run mysql-client -n data --rm -it --image=mysql:8.4 -- \
  mysql -h mysql-0.mysql.data.svc.cluster.local -u appuser -p

Client Setup Checklist

1StorageClass: confirm backend, zone behavior, reclaim policy, expansion support, encryption, and snapshots.
2Credentials: use External Secrets, Vault, SOPS, or client-approved secret process; do not keep plaintext values in Git.
3Backups: define backup target, schedule, retention, restore test, and owner before production data lands.
4Resources: set CPU/memory requests and limits appropriate for workload size.
5Access: expose MySQL internally with ClusterIP/headless DNS, not public LoadBalancer, unless explicitly required.

Simple Backup CronJob Shape

This shows the Kubernetes shape only. In real client work, prefer a tested backup tool, object storage target, encryption, retention policy, monitoring, and restore runbook.

yamlmysql-backup-cronjob.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mysql-backup
  namespace: data
spec:
  schedule: "15 2 * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: mysql:8.4
              command:
                - sh
                - -c
                - mysqldump -h mysql-0.mysql.data.svc.cluster.local -u root -p"$MYSQL_PWD" --all-databases > /backup/mysql-$(date +%F).sql
              env:
                - name: MYSQL_PWD
                  valueFrom:
                    secretKeyRef:
                      name: mysql-credentials
                      key: root-password
              volumeMounts:
                - name: backup
                  mountPath: /backup
          volumes:
            - name: backup
              persistentVolumeClaim:
                claimName: mysql-backup-target

PVC Retention And Deletion

StatefulSet PVCs are intentionally conservative. Deleting the StatefulSet does not delete PVCs by default. That protects data, but it also means old PVCs can remain after scale-down or rebuilds.

yamlpvc-retention.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: data
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  # Other StatefulSet fields omitted.

Where Helm Fits

Helm is usually how clients deploy packaged MySQL. This page teaches the underlying objects so you can read chart output and values confidently. In Helm, look for persistence settings such as persistence.enabled, persistence.storageClass, persistence.size, credentials, primary/replica settings, backup settings, and pod security contexts.

yamlvalues-mysql-persistence.yaml

primary:
  persistence:
    enabled: true
    storageClass: fast-retain
    size: 20Gi
auth:
  existingSecret: mysql-credentials
metrics:
  enabled: true

Troubleshooting

Symptom	Likely Cause	Check First
MySQL Pod Pending	PVC pending, StorageClass missing, zone conflict.	`describe pvc data-mysql-0`, events.
Pod ContainerCreating	Volume attach/mount problem.	Pod events, VolumeAttachment, CSI logs.
CrashLoopBackOff	Bad credentials/env, corrupt data dir, permission issue, insufficient memory.	Pod logs, previous logs, events.
Cannot connect by DNS	Headless Service missing/mislabelled, CoreDNS issue.	Service selector, EndpointSlice, DNS lookup.
Data lost after cleanup	PVC/PV deleted with Delete reclaim policy or wrong volume reused.	PV reclaim policy, backups, audit trail.
Scale replicas does not create HA	Raw MySQL StatefulSet does not configure replication automatically.	Use chart/operator/replication runbook.

Data Safety Checklist

bashbefore-mysql-storage-change.sh

kubectl get statefulset,pod,pvc,pv -n data -o wide
kubectl describe pvc -n data data-mysql-0
kubectl get storageclass fast-retain -o yaml | grep -E 'reclaimPolicy|allowVolumeExpansion|volumeBindingMode'

# Confirm backup evidence before risky work.
kubectl get cronjob,job -n data | grep -i backup
kubectl logs -n data job/<latest-backup-job> --tail=100

# Avoid deleting PVCs unless restore and data-retention plan is explicit.
kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,RECLAIM:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phase