Worker Nodes — K8s SRE Reference

TL;DR

A worker node is where application containers run. The kubelet watches assigned Pods, asks the container runtime to start containers, uses CNI for networking, mounts volumes through CSI, reports node/pod status, and enforces resource pressure eviction rules.

Mental Model

The control plane decides what should run; the worker node makes it real. When a Pod is scheduled to a node, kubelet pulls the PodSpec from the API server, prepares networking and volumes, starts containers through the runtime, runs probes, and continuously reports status back.

Worker node components and responsibility boundaries.

Node Components

Component	Job	Common Failure Signal
`kubelet`	Node agent. Starts Pods, runs probes, mounts volumes, reports status.	Node NotReady, Pod stuck ContainerCreating, probe failures, status not updating.
Container runtime	Pulls images and runs containers through CRI.	Image pull errors, container start failures, runtime socket errors.
CNI plugin	Gives Pods IPs, routes, and network policy enforcement.	Pod sandbox create failures, missing Pod IPs, cross-node traffic failures.
`kube-proxy`	Implements Service routing using iptables, IPVS, or nftables mode.	Service ClusterIP fails while Pod IP works.
CSI node plugin	Attaches and mounts volumes for Pods on that node.	Pod stuck waiting for volume attach or mount.

First-Look Node Triage

Start with read-only commands. They tell you if the node is healthy, schedulable, under pressure, or failing because of kubelet/runtime/network/storage.

bash node-first-look.sh

# List nodes with status, roles, age, version, internal IP, OS, and container runtime.
kubectl get nodes -o wide

# Replace <node-name> with the exact node name from kubectl get nodes.
kubectl describe node <node-name>

# Show allocatable CPU/memory/pods for every node.
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,PODS:.status.allocatable.pods'

# Show Pods on a single node across all namespaces.
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name> -o wide

# Show recent events involving this node.
kubectl get events --all-namespaces --field-selector involvedObject.kind=Node,involvedObject.name=<node-name> --sort-by=.lastTimestamp

Node Conditions

kubectl describe node shows conditions that summarize whether Kubernetes trusts the node for scheduling and workload execution.

Condition	Healthy Value	Meaning When Unhealthy	Where To Look
`Ready`	`True`	Kubelet is not posting healthy status or node cannot be reached.	Kubelet logs, runtime status, node network, disk pressure.
`MemoryPressure`	`False`	Node memory is low; kubelet may evict Pods.	Requests/limits, node memory, noisy Pods, eviction events.
`DiskPressure`	`False`	Disk is low, often image layers, logs, or ephemeral storage.	Image garbage collection, container logs, ephemeral-storage usage.
`PIDPressure`	`False`	Too many processes on the node.	Pods spawning processes, host process count, PID limits.
`NetworkUnavailable`	`False`	Node network route setup is incomplete.	CNI daemonset, route tables, node labels/taints.

bash node-conditions.sh

# Print node conditions in a compact table.
kubectl get node <node-name> -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason} message={.message}{"\n"}{end}'

# Find all nodes that are not Ready.
kubectl get nodes --no-headers | awk '$2 != "Ready" {print}'

# Watch node state during a maintenance or recovery operation.
kubectl get nodes --watch

Kubelet

Kubelet is the most important node-local process. It watches the API for Pods assigned to its node, creates Pod sandboxes, asks CRI to run containers, executes liveness/readiness/startup probes, handles volumes, and posts status back to the API server.

Managed cluster noteIn EKS/GKE/AKS you usually debug kubelet through node logs, cloud logging, SSM/SSH if allowed, or managed node group replacement rather than hand-editing kubelet config.

bash kubelet-host-checks.sh

# Run these on the node host only when you have approved shell access.

# Check whether kubelet is running.
sudo systemctl status kubelet

# Follow kubelet logs for scheduling, runtime, CNI, probe, and volume errors.
sudo journalctl -u kubelet -f

# Show recent kubelet logs without following forever.
sudo journalctl -u kubelet --since "30 minutes ago" --no-pager

# Inspect kubelet startup flags. The config file path often appears here.
sudo systemctl cat kubelet

Kubelet Config Shape

This is an example shape, not a universal client standard. Use it to recognize important settings when reading an existing cluster.

yaml kubelet-config-example.yaml

# Example kubelet configuration shape.
# Do not paste this into production without matching the client's cluster version and runbook.
apiVersion: kubelet.config.k8s.io/v1
kind: KubeletConfiguration
clusterDNS:
  - 10.96.0.10 # ClusterIP of CoreDNS/kube-dns Service.
clusterDomain: cluster.local # DNS suffix used for Kubernetes service discovery.
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock # CRI socket kubelet talks to.
evictionHard:
  memory.available: "100Mi" # Start hard evictions when available memory is below this threshold.
  nodefs.available: "10%" # Protect node filesystem from filling up.
  imagefs.available: "15%" # Protect image filesystem from filling up.
maxPods: 110 # Maximum Pods kubelet will run on this node.
rotateCertificates: true # Allows kubelet client certificate rotation.

Container Runtime

Modern Kubernetes uses the Container Runtime Interface (CRI). The common runtimes are containerd and CRI-O. Runtime problems often show up as image pull failures, sandbox creation failures, or containers stuck waiting.

bash runtime-checks.sh

# Kubernetes-level check: shows runtime version for each node.
kubectl get nodes -o custom-columns='NAME:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion,KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage'

# Host-level checks for containerd nodes.
sudo systemctl status containerd
sudo journalctl -u containerd --since "30 minutes ago" --no-pager

# crictl talks to the CRI runtime. It is useful when kubelet is unhealthy.
sudo crictl info
sudo crictl ps -a
sudo crictl images

# Inspect one failed container. Replace <container-id> with an ID from crictl ps -a.
sudo crictl inspect <container-id>
sudo crictl logs <container-id>

Node Networking

Worker node networking has two main pieces: the CNI plugin creates Pod networking, and kube-proxy implements Service routing. If direct Pod IP traffic works but Service IP traffic fails, suspect kube-proxy or Service/Endpoints. If Pod IP assignment fails, suspect CNI.

bash node-network-checks.sh

# Find CNI and kube-proxy pods. Names vary by plugin and provider.
kubectl get pods -n kube-system -o wide | grep -E 'calico|cilium|flannel|weave|aws-node|azure|kube-proxy'

# Check whether kube-proxy is running as a DaemonSet.
kubectl get daemonset -n kube-system kube-proxy
kubectl describe daemonset -n kube-system kube-proxy

# If a Pod is stuck ContainerCreating, describe it and look for CNI errors.
kubectl describe pod <pod-name> -n <namespace>

# Compare Pod IP connectivity versus Service connectivity from a debug Pod.
# Replace values with real IPs and ports from the app.
kubectl run netshoot -n <namespace> --rm -it --image=nicolaka/netshoot -- /bin/bash
curl -vk http://<pod-ip>:<port>
curl -vk http://<service-name>.<namespace>.svc.cluster.local:<port>

Resource Pressure And Evictions

Node pressure means kubelet is protecting the node. It may evict Pods when memory, disk, inode, or PID pressure crosses configured thresholds. This is different from the scheduler deciding where new Pods go.

bash pressure-checks.sh

# Metrics API must be installed for kubectl top.
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=memory
kubectl top pods --all-namespaces --sort-by=cpu

# Look for eviction events.
kubectl get events --all-namespaces --sort-by=.lastTimestamp | grep -i -E 'evict|pressure|disk|memory'

# Show Pods on a pressured node with their QoS class.
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name> \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass,PHASE:.status.phase'

Eviction order mattersPods with BestEffort QoS are evicted before Burstable, and Guaranteed Pods are the most protected. Missing requests and limits make production behavior harder to predict.

Cordon, Drain, And Uncordon

Cordon stops new Pods from scheduling to a node. Drain evicts most existing Pods so the node can be patched, replaced, or repaired. Always check PodDisruptionBudgets and daemonsets before draining production nodes.

bash node-maintenance.sh

# Mark the node unschedulable so new Pods do not land on it.
kubectl cordon <node-name>

# Check workloads currently on the node before evicting anything.
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name> -o wide

# Check PodDisruptionBudgets. Drains can block if disruption budgets are too strict.
kubectl get pdb --all-namespaces

# Drain the node. DaemonSet Pods are ignored because their controller recreates them on every node.
# EmptyDir data is deleted, so use this only when that data is safe to lose.
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# After maintenance, allow scheduling again.
kubectl uncordon <node-name>

Troubleshooting Map

!Node NotReady: check kubelet status/logs, node network reachability, certificate expiration, runtime health, and disk pressure.
!Pod stuck ContainerCreating: check CNI errors, image pulls, volume mounts, runtime logs, and Pod events.
!ImagePullBackOff: check image name/tag, registry auth Secret, network path to registry, and rate limits.
!Evicted Pods: check node pressure events, Pod QoS, requests/limits, ephemeral storage, and container log growth.
!Service traffic broken: test Pod IP first, then Service DNS, Endpoints/EndpointSlices, kube-proxy, and network policy.

Mental Model

Node Components

First-Look Node Triage

Node Conditions

Kubelet

Kubelet Config Shape

Container Runtime

Node Networking

Resource Pressure And Evictions

Cordon, Drain, And Uncordon

Troubleshooting Map

Related Pages