systemd & Logs — K8s SRE Reference

TL;DR

Use systemctl to manage services and journalctl to read structured logs. On Kubernetes nodes, kubelet, containerd, and docker all run as systemd services — these are your first stop when a node stops joining the cluster or containers won't start.

systemctl — Service Management

The primary interface to systemd: start, stop, restart, enable, and inspect services and their dependencies.

bashsystemctl.sh

# Status and listing
systemctl status kubelet                     # detailed status + last log lines
systemctl status containerd
systemctl list-units --type=service --state=failed   # all failed services
systemctl list-units --type=service          # all loaded services

# Service lifecycle
sudo systemctl start   <service>
sudo systemctl stop    <service>
sudo systemctl restart <service>             # stop then start
sudo systemctl reload  <service>             # reload config without full restart (if supported)
sudo systemctl try-reload-or-restart <service>

# Enable / disable (persist across reboots)
sudo systemctl enable  <service>
sudo systemctl disable <service>
sudo systemctl enable --now <service>        # enable AND start immediately
sudo systemctl is-enabled <service>          # check: enabled/disabled/static

# Service file inspection
systemctl cat kubelet                        # show the unit file content
systemctl show kubelet                       # all properties (machine-readable)
systemctl show kubelet --property=ExecStart,Restart,RestartSec

# After editing unit files
sudo systemctl daemon-reload                 # always reload before restart after file changes

# Dependencies
systemctl list-dependencies kubelet          # tree of dependencies
systemctl list-dependencies --reverse kubelet  # what depends on kubelet

journalctl — Log Queries

journalctl queries the systemd journal; filter by service, time range, priority, or boot to quickly isolate the log window you care about.

bashjournalctl.sh

# Follow a service (like tail -f)
journalctl -u kubelet -f
journalctl -u containerd -f

# Last N lines
journalctl -u kubelet -n 100
journalctl -u kubelet --no-pager -n 50    # pipe-friendly

# Time ranges — very useful during incident post-mortems
journalctl -u kubelet --since "2026-05-23 14:00:00" --until "2026-05-23 14:30:00"
journalctl -u kubelet --since "1 hour ago"
journalctl --since today

# Priority filtering (-p: emerg alert crit err warning notice info debug)
journalctl -u kubelet -p err             # errors and above
journalctl -p crit                       # critical messages across all services

# Current boot only
journalctl -b 0 -u kubelet               # this boot
journalctl -b -1 -u kubelet              # previous boot (useful post-reboot)
journalctl --list-boots                  # list all recorded boots

# Kernel messages (equivalent to dmesg with timestamps)
journalctl -k                            # kernel messages
journalctl -k --since "10 min ago"

# JSON output for programmatic processing
journalctl -u kubelet -n 50 -o json | jq '.MESSAGE'

# Disk usage
journalctl --disk-usage
sudo journalctl --vacuum-size=500M       # trim journal to 500 MB

Kubelet Service on Nodes

When a node shows NotReady in Kubernetes, the kubelet service is usually the first thing to check; these commands orient you to its configuration and recent errors.

bashkubelet-node.sh

systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager | grep -iE "error|fail|warn|PLEG|OOM"

# kubelet config and flags
systemctl cat kubelet                                  # ExecStart line shows all flags
cat /var/lib/kubelet/config.yaml                       # kubelet config file
cat /etc/kubernetes/kubelet.conf                       # kubeconfig for the kubelet

# container runtime
systemctl status containerd
journalctl -u containerd -n 50 --no-pager | grep -i error

# crictl: container runtime interface (works with containerd and CRI-O)
sudo crictl ps -a                         # all containers including exited
sudo crictl logs <container-id>          # container logs at runtime level
sudo crictl inspect <container-id>       # container details including pid

Writing Unit Files

A minimal unit file covers the most common pattern: a long-running daemon that restarts on failure and is ordered after the network is up.

bash/etc/systemd/system/myapp.service

[Unit]
Description=My Application
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5s
# Capture stdout/stderr to journal automatically
StandardOutput=journal
StandardError=journal
# Resource limits
LimitNOFILE=65536
MemoryMax=512M
CPUQuota=200%

[Install]
WantedBy=multi-user.target

systemd Timers (Cron Replacement)

A systemd timer runs a companion .service unit on a schedule; unlike cron, output goes to the journal, failures trigger systemd alerts, and you can see last/next run times.

bashbackup.timer

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=daily                 # run at midnight
RandomizedDelaySec=30m           # random delay up to 30 min (avoids stampedes)
Persistent=true                  # run missed jobs after boot

[Install]
WantedBy=timers.target

# --- companion service ---
# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup job

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh

# Enable and check
sudo systemctl enable --now backup.timer
systemctl list-timers --all | grep backup
journalctl -u backup.service -n 20

Troubleshooting

!Service fails immediately: systemctl status <service> and journalctl -u <service> -n 30 — look for missing binary path, wrong user, missing dependency.
!daemon-reload required: always run systemctl daemon-reload after editing unit files, or changes are silently ignored.
!Journal full: journalctl --disk-usage; configure SystemMaxUse in /etc/systemd/journald.conf.
!kubelet won't start: check swap (free -h) — kubelet requires swap disabled unless configured with --fail-swap-on=false.