Container Runtime Debugging
On Kubernetes nodes, use crictl to inspect containers at the runtime level when kubectl can't reach the API server. Use kubectl debug to attach an ephemeral debug container to a running pod. Use nsenter to enter a container's namespaces from the host for deep inspection.
crictl — CRI Command Line
crictl is the standard CRI CLI — it works with containerd and CRI-O and is your primary tool on a node when the API server is unreachable or when kubelet hasn't started the container yet.
# Configure crictl (needed once per node)
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF
# List containers (all states including exited)
crictl ps -a
crictl ps --name my-app # filter by name
# Container logs at runtime level
crictl logs <container-id>
crictl logs -f --tail 100 <container-id>
# Inspect a container — shows PID, mounts, config
crictl inspect <container-id>
crictl inspect <container-id> | jq .info.pid # get PID for nsenter
# List pods (sandbox containers)
crictl pods
crictl pods --name my-pod
# Images on this node
crictl images
crictl rmi <image> # remove image
# Stats — CPU/memory per container
crictl stats
crictl stats <container-id>kubectl debug — Ephemeral Containers
Ephemeral containers attach to a running pod sharing its PID and network namespace — the preferred way to debug distroless images that have no shell of their own.
# Attach a debug sidecar to a running pod
kubectl debug -it <pod> -n <ns> \
--image=nicolaka/netshoot \
--target=<container-name> \ # share PID namespace with this container
-- /bin/bash
# Debug a node directly (privileged pod with host namespaces)
kubectl debug node/<node-name> -it --image=ubuntu -- bash
# Useful images for debug containers:
# nicolaka/netshoot — curl, dig, tcpdump, ss, mtr, iperf3
# busybox — minimal shell + utilities
# ubuntu/alpine — for apt/apk installing additional tools
# List ephemeral containers on a pod
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.ephemeralContainers}'nsenter — Enter Container Namespaces
nsenter enters a container's Linux namespaces from the host node — useful for examining network, filesystem, and process state at the kernel level, bypassing the container runtime entirely.
# Get container PID via crictl
CONTAINER_ID=$(crictl ps --name my-app -q)
PID=$(crictl inspect $CONTAINER_ID | jq -r .info.pid)
# Enter all namespaces (network, pid, mount, uts, ipc)
nsenter -t $PID --net --pid --mount --uts --ipc -- bash
# Enter only network namespace (e.g., to run tcpdump from host)
nsenter -t $PID --net -- tcpdump -i eth0 -w /tmp/cap.pcap
# Enter network namespace to run ss inside container's network
nsenter -t $PID --net -- ss -tlnp
# Enter mount namespace to ls the container's filesystem
nsenter -t $PID --mount -- ls /proc/$PID/root/app/
# Read a file from inside the container filesystem
nsenter -t $PID --mount -- cat /proc/$PID/root/etc/config.yamlnerdctl (containerd native CLI)
nerdctl is a Docker-compatible CLI for containerd; useful on nodes where Docker is not installed but you need to build or run containers locally for testing.
nerdctl images # list images in containerd
nerdctl pull nginx:1.25
nerdctl run --rm -it nginx:1.25 bash # interactive container
nerdctl build -t myapp:test . # build image
nerdctl ps -a # list all containers
nerdctl logs <container>
nerdctl inspect <container>Troubleshooting Map
| Symptom | Check | Likely cause |
|---|---|---|
| Container stuck in ContainerCreating | crictl ps -a, journalctl -u containerd | Image pull failure, CNI error, volume mount failure |
| containerd not responding | systemctl status containerd, journalctl -u containerd -n 50 | Disk full on /run, socket permission, OOM |
| Can't exec into pod (no shell) | kubectl debug with netshoot | Distroless image; use ephemeral container |
| Network issues inside pod | nsenter -t <pid> --net -- ss -tlnp | CNI misconfiguration, iptables issue |