Linux Storage & Filesystems — K8s SRE Reference

TL;DR

Use this page when a node has a full disk, a PVC won't mount, or you need to resize a volume. Start read-only (df, lsblk, findmnt), then check the Kubernetes storage troubleshooting page before touching block devices on a running node.

Disk Usage

Always run df and du first when a pod is evicted or kubelet emits DiskPressure — they show filesystem-level fullness and which directories are consuming space.

bashdisk-usage.sh

# Filesystem usage overview
df -hT                           # human sizes with filesystem type
df -hT | grep -v tmpfs           # exclude tmpfs/cgroup entries

# Find which directories are large
du -sh /var/*                    # size of each directory under /var
du -sh --max-depth=2 /           # 2-level breakdown from root
du -sh /var/lib/docker           # container layer storage
du -sh /var/lib/kubelet          # kubelet working directory (pod volumes, etc.)

# Find large files quickly
find / -type f -size +500M -printf "%s\t%p\n" 2>/dev/null | sort -rn | head -20

# inode usage (can be full even when disk space remains)
df -ih                           # inode usage per filesystem
find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20
# Lots of tiny files in a single directory exhausts inodes

Block Devices

lsblk gives a clear tree of disks, partitions, and their mount points; use it on a node to understand the storage topology before resizing or partitioning.

bashblock-devices.sh

# Block device overview
lsblk                            # tree: disk → partition → mount point
lsblk -f                         # include filesystem type and UUID
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,UUID

# Detailed device info
blkid /dev/sdb1                  # UUID, type
fdisk -l /dev/sda                # partition table (requires root)
parted /dev/sda print            # GPT-friendly partition info

# Device health (SMART) — useful for bare metal
smartctl -a /dev/sda             # full SMART report (requires smartmontools)
smartctl -H /dev/sda             # health summary only

Mounts

Use findmnt for a structured view of the mount tree; use mount and umount when you need to manually attach or detach a filesystem during troubleshooting.

bashmounts.sh

# List mounts
findmnt                          # tree view of all mounts
findmnt /mnt/data                # show specific mount
findmnt --type nfs,nfs4          # show only NFS mounts
cat /proc/mounts                 # raw kernel mount table

# Manual mount / unmount
sudo mount /dev/sdb1 /mnt/data
sudo mount -t nfs4 server:/export /mnt/nfs
sudo mount -o remount,ro /mnt/data   # remount read-only

sudo umount /mnt/data
sudo umount -l /mnt/data         # lazy unmount (when "device busy")

# Persistent mounts in /etc/fstab
# UUID=abc123 /mnt/data ext4 defaults,nofail 0 2
#              ^mount   ^fs  ^options
# nofail: don't block boot if mount fails
# Test before rebooting:
sudo mount -a                    # try all fstab entries

# Check what is keeping a mount busy
fuser -m /mnt/data               # PIDs using the mount
lsof | grep /mnt/data

Filesystem Operations

Format a new disk, check filesystem health, or resize online — always unmount first before fsck, and take a snapshot before resizing in production.

bashfilesystem-ops.sh

# Format a new disk (DESTRUCTIVE — double-check device path)
sudo mkfs.ext4 -L data /dev/sdb1
sudo mkfs.xfs  /dev/sdb1         # XFS: default on RHEL/CentOS, good for large files

# Check and repair filesystem (must be unmounted)
sudo fsck -n /dev/sdb1           # dry-run check only
sudo fsck -y /dev/sdb1           # auto-repair (use carefully)
sudo e2fsck -f /dev/sdb1         # extended check for ext4

# Online resize (after expanding the underlying disk/partition)
sudo resize2fs /dev/sdb1         # ext4: grow to fill partition
sudo xfs_growfs /mnt/data        # XFS: specify mount point, not device

# Check ext4 reserved space (5% default can be a surprise)
tune2fs -l /dev/sdb1 | grep "Reserved block"
sudo tune2fs -m 1 /dev/sdb1      # reduce reserved to 1% on data volumes

LVM Quick Reference

LVM layers physical volumes (PV) → volume groups (VG) → logical volumes (LV); it allows online resizing, snapshots, and flexible layout across multiple disks.

bashlvm.sh

# Inspect current LVM layout
pvs                              # physical volumes
vgs                              # volume groups
lvs                              # logical volumes
lvdisplay /dev/vg0/data          # detailed LV info

# Extend a logical volume and filesystem (online for ext4/xfs)
sudo lvextend -L +20G /dev/vg0/data          # add 20 GB
sudo lvextend -l +100%FREE /dev/vg0/data     # use all free space in VG
sudo resize2fs /dev/vg0/data                 # ext4: grow FS
sudo xfs_growfs /mnt/data                    # xfs: grow FS

# Create a snapshot (useful for pre-maintenance backup)
sudo lvcreate -L 5G -s -n data-snap /dev/vg0/data
# Mount snapshot read-only for inspection
sudo mount -o ro /dev/vg0/data-snap /mnt/snap

Troubleshooting Map

Symptom	Command	Likely cause
Pod evicted / DiskPressure	`df -hT; du -sh /var/lib/kubelet/*`	Node rootfs or ephemeral storage full; container logs or layers bloated
PVC stuck Pending	`kubectl describe pvc; kubectl get events -n <ns>`	No matching StorageClass, zone mismatch, quota exceeded
Mount permission denied	`ls -la <mount-point>; id`	Wrong UID/GID in container vs volume; runAsUser mismatch
NFS mount hangs	`showmount -e <server>; mount -v`	Firewall, export missing, nfs-utils not installed on node
Inode exhausted	`df -ih`	Millions of tiny files; clean up or choose larger inode ratio at format time