Cilium — K8s SRE Reference

TL;DR

Cilium is an eBPF-based CNI that can provide Pod networking, NetworkPolicy, kube-proxy replacement, observability through Hubble, and Gateway/Ingress features depending on installation choices.

eBPF Dataplane

Pod traffic is handled in-kernel by eBPF programs managed by cilium-agent.

Mental Model

Part	What it does	Why you care
cilium-agent	Runs on every node and programs eBPF datapath.	Most dataplane issues show here.
cilium-operator	Handles cluster-wide tasks like IPAM and cleanup.	Important for IP allocation and node lifecycle.
Hubble	Flow visibility and service map.	Fast answer to "who is dropping traffic?"
kube-proxy replacement	Cilium handles Service load balancing.	Service failures are debugged through Cilium too.

Health Checks

bashcilium-health.sh

kubectl get pods -n kube-system -l k8s-app=cilium -o wide # One Cilium agent Pod per node.
kubectl get pods -n kube-system -l io.cilium/app=operator
kubectl logs -n kube-system -l k8s-app=cilium --tail=100

# If the Cilium CLI is installed:
cilium status --wait # Summarizes agent, operator, Hubble, and datapath health.
cilium connectivity test # Creates test workloads and validates common traffic paths.

# Exec into one Cilium agent for node-local detail.
kubectl exec -n kube-system ds/cilium -- cilium status
kubectl exec -n kube-system ds/cilium -- cilium service list # Service load-balancing table.

Cilium Network Policy

yamlcilium-network-policy.yaml

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-frontend-http
  namespace: app
spec:
  endpointSelector:
    matchLabels:
      app: web-api # Destination endpoints protected by this policy.
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend # Source endpoint label.
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /health # Optional L7 HTTP rule if proxy/L7 policy is enabled.

Hubble Flow Debugging

bashhubble.sh

hubble status # Confirms the Hubble API is reachable.
hubble observe --namespace app --last 20 # Recent flows in the app namespace.
hubble observe --from-pod app/frontend-abc123 --to-pod app/web-api-def456
hubble observe --verdict DROPPED --namespace app # Quickly find policy or datapath drops.

# Without hubble CLI, check Hubble pods and relay logs.
kubectl get pods -n kube-system | grep hubble
kubectl logs -n kube-system deploy/hubble-relay --tail=100

kube-proxy Replacement Checks

bashservice-debug.sh

kubectl -n kube-system get configmap cilium-config -o yaml | grep -E 'kubeProxyReplacement|routingMode'
kubectl exec -n kube-system ds/cilium -- cilium service list
kubectl exec -n kube-system ds/cilium -- cilium bpf lb list # Low-level load-balancer map.

# Compare Kubernetes desired endpoints with Cilium service map.
kubectl get svc,endpointslice -n app -l kubernetes.io/service-name=web-api -o wide

Gotchas

!Kernel capabilities matter; very old kernels can limit Cilium features.
!When kube-proxy replacement is enabled, missing Services may be a Cilium datapath issue rather than kube-proxy.
!L7 policy can break traffic that L3/L4 checks appear to allow.
!Hubble dropped-flow verdicts are often faster than guessing through YAML.