PV, PVC & StorageClasses — K8s SRE Reference

TL;DR

A PersistentVolumeClaim asks for storage, a StorageClass defines how dynamic storage is provisioned, and a PersistentVolume is the actual Kubernetes object bound to the claim. Most storage incidents come from wrong StorageClass, unavailable CSI driver, access mode mismatch, zone binding, attach/mount failures, or misunderstood reclaim policy.

Mental Model

Kubernetes separates what the app asks for from how the platform provides it. The app uses a PVC. The platform defines StorageClasses. A CSI driver talks to the cloud, SAN, NAS, or software-defined storage backend. Once bound, the Pod mounts the PVC and sees a filesystem or block device.

Storage is one of the places where careless deletion can become data loss. Before deleting PVCs, PVs, StatefulSets, or StorageClasses, understand reclaim policy, snapshots/backups, and whether the backend volume is still needed.

Kubernetes storage binding model.

Core Objects

Object	Scope	Purpose	SRE Question
`StorageClass`	Cluster	Defines provisioner, parameters, reclaim policy, binding mode, expansion.	Which backend and behavior will be used?
`PersistentVolumeClaim`	Namespace	App request for capacity, access mode, and class.	Is the request satisfiable?
`PersistentVolume`	Cluster	Concrete volume object bound to a PVC.	What backend asset is bound and what is its reclaim policy?
`VolumeAttachment`	Cluster	Tracks CSI attach state for a node.	Is the disk stuck attached to another node?
CSI controller/node Pods	Cluster add-on	Provision, attach, mount, resize, snapshot, delete.	Is the storage driver healthy?

Dynamic Vs Static Provisioning

Dynamic provisioning is the common cloud-native path: a PVC names a StorageClass and the CSI driver creates the backend volume. Static provisioning is when an admin creates a PV for an existing disk, NFS export, or special backend and lets a PVC bind to it.

Pattern	How It Works	Use When
Dynamic	PVC + StorageClass triggers CSI provisioning.	Most app volumes, cloud block storage, managed CSI.
Static	Admin creates PV that points to existing storage.	Existing NFS share, retained disk recovery, special enterprise storage.
Pre-bound	PVC and PV use `volumeName` or claimRef to force binding.	Controlled recovery or migration cases.

StorageClass

yamlstorageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-retain
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Field	Meaning	Operational Impact
`provisioner`	CSI driver name.	Must match installed CSI driver.
`reclaimPolicy`	What happens to backend asset after PV release.	`Delete` is convenient; `Retain` is safer for critical data.
`allowVolumeExpansion`	Whether PVC size can grow.	Shrinking is generally not supported.
`volumeBindingMode`	When PV is provisioned/bound.	`WaitForFirstConsumer` helps zone-aware scheduling.
`parameters`	Driver-specific backend options.	Class, performance, encryption, filesystem, type, IOPS.

PVC And Deployment From Scratch

yamlpvc-deployment.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: app
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: app
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-retain
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-volume
  namespace: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-with-volume
  template:
    metadata:
      labels:
        app: app-with-volume
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "date >> /data/heartbeat.txt; sleep 3600"]
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: app-data

Access Modes

Mode	Meaning	Common Backend	Gotcha
`ReadWriteOnce` / RWO	Mounted read-write by one node.	EBS, Azure Disk, GCE PD.	Multiple Pods can use it only if scheduled on same node and backend permits.
`ReadWriteMany` / RWX	Mounted read-write by many nodes.	NFS, EFS, CephFS, Azure Files.	Not all storage supports it; performance semantics differ.
`ReadOnlyMany` / ROX	Mounted read-only by many nodes.	Shared content volumes.	App must not need writes.
`ReadWriteOncePod` / RWOP	Mounted read-write by only one Pod.	CSI-supported block storage.	Useful for stronger single-writer guarantees.

Binding Modes And Reclaim Policy

Setting	What It Means	Risk/Benefit
`Immediate`	Provision/bind PVC as soon as it is created.	Can choose a zone before scheduler knows where Pod should run.
`WaitForFirstConsumer`	Wait until a Pod uses the PVC, then provision in scheduled topology.	Better for zonal disks and topology-aware scheduling.
`Delete`	Backend volume is deleted when PV is released.	Good for ephemeral/non-critical data; dangerous if misunderstood.
`Retain`	Backend volume remains after PVC deletion.	Safer for important data; requires manual cleanup/rebind.

StatefulSet Storage Pattern

StatefulSets usually use volumeClaimTemplates. Each replica gets its own PVC, such as data-postgres-0, data-postgres-1. Deleting the StatefulSet does not automatically delete PVCs, which is usually what you want for data safety.

yamlstatefulset-storage.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: data
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-retain
        resources:
          requests:
            storage: 100Gi

Static PersistentVolume

yamlstatic-nfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-shared-pv
spec:
  capacity:
    storage: 100Gi
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-static
  nfs:
    server: 10.0.10.25
    path: /exports/app

Daily Commands

bashstorage-ops.sh

kubectl get storageclass
kubectl describe storageclass <storageclass>
kubectl get pvc -n <namespace> -o wide
kubectl describe pvc <pvc> -n <namespace>
kubectl get pv
kubectl describe pv <pv>
kubectl get volumeattachment
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -i -E 'volume|mount|attach|pvc|provision'

Expansion

Expanding is usually one-way. Confirm the StorageClass and CSI driver support expansion before patching. Filesystem resize may happen online or on next mount depending on driver and filesystem.

bashexpand-pvc.sh

kubectl get storageclass <storageclass> -o yaml | grep allowVolumeExpansion
kubectl patch pvc app-data -n app -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
kubectl describe pvc app-data -n app

Troubleshooting Workflow

Start with the PVC: status, events, StorageClass, requested size, and access mode.
Check the bound PV: reclaim policy, node affinity, capacity, backend handle.
Check Pod events for FailedAttachVolume, FailedMount, or permissions errors.
Inspect CSI controller and node plugin Pods in the storage namespace or kube-system.
For zonal disks, check node zone, PV node affinity, and Pod scheduling constraints.

Symptom To Cause

Symptom	Likely Cause	Check First
PVC Pending	Wrong StorageClass, no default class, CSI provisioner down, quota, WaitForFirstConsumer waiting for Pod.	`describe pvc`, StorageClass, CSI controller logs.
Pod stuck ContainerCreating	Attach or mount failure.	Pod events and VolumeAttachment.
Multi-Attach error	RWO disk still attached to another node.	Old Pod/node, VolumeAttachment, cloud disk attachment.
Permission denied inside container	Filesystem ownership, securityContext, NFS export permissions.	Mount path ownership, `fsGroup`, backend export.
Data disappeared after PVC delete	Reclaim policy was Delete.	PV reclaim policy and backup/snapshot availability.
Expansion stuck	Driver does not support resize, filesystem resize pending, quota.	PVC conditions and CSI logs.
Pod unschedulable with PVC	Zone/topology conflict.	PV node affinity, node labels, volume binding mode.

Safe Change Pattern

1Do not delete first: inspect PVC, PV, reclaim policy, snapshots, and owner before deleting anything.
2Use Retain for important data: especially databases and client-owned persistent volumes.
3Snapshot before risky changes: expansion, migration, reclaim-policy changes, or StatefulSet storage work.
4Know the driver: CSI behavior differs by cloud, NAS, SAN, and on-prem platform.
5Change through source of truth: Helm, GitOps, Terraform, or storage platform process.