Services & Networking Troubleshooting

Symptom Decision Tree

Service Path Checks

NS=app
SVC=web-api

kubectl get svc "$SVC" -n "$NS" -o wide # Service type, ClusterIP, ports, selector.
kubectl describe svc "$SVC" -n "$NS" # Events and endpoint summary.
kubectl get endpoints "$SVC" -n "$NS" -o wide # Legacy endpoint object.
kubectl get endpointslice -n "$NS" -l kubernetes.io/service-name="$SVC" -o wide # Modern endpoint source.
kubectl get pods -n "$NS" -l app=web-api -o wide # Verify selector finds ready Pods.

DNS Debug Pod

kubectl run netshoot -n app --rm -it --image=nicolaka/netshoot -- bash

# Inside the debug Pod:
dig web-api.app.svc.cluster.local # DNS should return the Service ClusterIP.
curl -sv http://web-api.app.svc.cluster.local:8080/health # Tests DNS + Service + backend app.
curl -sv http://10.42.3.21:8080/health # Tests direct Pod IP; bypasses Service routing.
nc -vz web-api.app.svc.cluster.local 8080 # Quick TCP connectivity test.

kubectl logs -n kube-system deploy/coredns --tail=200
kubectl exec -it deploy/misc-tools -n app -- cat /etc/resolv.conf # confirm nameserver + search domains

# Stub upstream resolver health (clusters vary).
kubectl get cm coredns -n kube-system -o yaml

CNI Dataplane

East-west breakage before Services misbehave hints at encapsulated overlay drops or eBPF attachment failures.

kubectl get pods -n kube-system -o wide | grep -iE 'cilium|calico|weave|cni|kube-proxy'

# Inspect programming lag on dataplane-heavy nodes only if SSH allowed.
kubectl get nodes worker-1 -o jsonpath='{.metadata.labels}'

Kube-Proxy Operating Modes

NetworkPolicy Checks

kubectl get networkpolicy -A # Any policy can change traffic behavior in its namespace.
kubectl describe networkpolicy allow-frontend-to-api -n app
kubectl get pod web-api-abc123 -n app --show-labels # Labels must match policy podSelector.
kubectl get namespace app --show-labels # Namespace labels matter for namespaceSelector.

# Temporarily test with a controlled allow rule only if incident policy allows it.
# Prefer reviewing existing policies before adding broad allow-all rules.

Ingress And LoadBalancer

kubectl get ingress -A
kubectl describe ingress web-api -n app # Rules, TLS secret, class, and events.
kubectl get svc -n ingress-nginx # LoadBalancer Service for the controller.
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100

# Test host routing without relying on public DNS.
curl -sv -H 'Host: api.example.com' http://<load-balancer-ip>/health

kubectl get gateways.gateway.networking.k8s.io -A 2>/dev/null
kubectl describe httproute web -n app 2>/dev/null

Failure Map

Packet Capture Ethics

Prefer Hubble/Skupper-style metadata before resorting to tcpdump on production nodes; scrub customer payloads and obtain legal approval when inspecting payload-bearing flows.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -ni eth0 tcp port 443 -c40

IPv4/IPv6 Dual-Stack Quirks

Headless Services And Stateful Routing

Headless clusters (clusterIP: None) return Pod A records straight from Endpoints enabling peer discovery. When Pods appear “healthy” behind a normal Service but flaky headless lookups race, suspects include readiness gating emitting endpoints too early versus application-level bootstrap ordering.

kubectl get svc kafka -n data -o wide
dig kafka.data.svc.cluster.local +tcp +short # expect multiple Pod IPs behind headless svc

externalTrafficPolicy And Source IP

Service Mesh Edge Cases

Envoy-linked meshes shift failures from iptables DNAT chains to xDS staleness symptoms—401 loops when rotated certs or clusters lose RDS subscription.

WireGuard / Encapsulation Math

Each overlay header reduces effective MSS; document expected overhead per CNI release so MTU playbooks stay accurate after upgrades.

• Geneve header adds ~50B depending on options—validate with vendor tables.
• IP-in-IP double encapsulation can trigger PMTU black holes between regions.
• Userspace tunnels (some dev clusters) cap throughput and appear as “random latency spikes”.
• Kernel GSO/GRO toggles sometimes regress on specific NIC firmware pairings.
• Keep a living doc of jumbo capabilities per rack-ID for bare metal estates.

Managed Load Balancer Checks

Hyperscaler LBs reconcile slower than Pods—always compare controller events with cloud console target health timelines.

kubectl describe svc public-api -n edge | grep -i event -A40
kubectl get endpointslices -n edge -o wide # ensure healthy ports match LB listener

Windows Node Caveats

Windows HNS policies diverge from Linux netstack—NetworkPolicy support depends on platform version; treat cross-OS Service connectivity as a first-class test matrix item.

Session Affinity Debugging

sessionAffinity: ClientIP leverages conntrack but breaks when Pods churn frequently; annotate expectations for testers.

kubectl get svc cart -n shop -o yaml | grep -i session

Corporate Egress And HTTP Proxies

Clusters forced through explicit proxies need HTTP_PROXY variables synchronized across init containers; missing CA bundles surface as TLS verify errors indistinguishable from remote outages.

1. Confirm NO_PROXY excludes in-cluster Kubernetes API hostnames.
2. Validate CONNECT tunnel MTU behaves identically across AZs.
3. Rotate proxy credentials centrally; avoid bespoke Secret copies per Deployment.
4. Watch for Envoy-based sidecars doubling proxy chaining latency.

CDN / WAF Layers

When Ingress returns 200 internally but CDN serves stale 503, capture X-Request-Id correlations across CDN vendor logs versus upstream Ingress logs rather than chasing kube-proxy first.

Choosing Probes For Network Tickets

Blend L3/L4/L7 evidence so you do not mistake ICMP success for working HTTP/2 ALPN handshakes.

eBPF Map Pressure

Cilium and similar CNIs expose map fill levels—sudden policy import storms can exhaust LRU maps and silently drop flows.

• Watch for map entry update failed warnings in agent logs during bulk NetworkPolicy applies.
• Correlate with GitOps reconcilers re-applying identical policies in tight loops.
• Resize maps only with vendor guidance; arbitrary sysctl bumps may destabilize nodes.
• Prefer incremental policy changes over wholesale namespace dumps.
• Validate kernel lockdown settings when loading BPF programs after secure-boot enablement.

Egress SNAT Pools

Centralized NAT gateways exhaust ephemeral ports faster than kube-proxy metrics reveal—track five-tuple reuse patterns when API fan-out spikes.

# On Linux node with privileges (use sparingly):
sudo conntrack -S | head

Latency Budget Checklist

East-west microseconds matter when autoscaled fleets chatter; treat each hop as borrowed time.

• Quantify DNS lookup + TCP handshake + TLS (if applicable) independently.
• subtract queueing delays visible in Ingress controller histograms.
• compare client-side timings with Envoy upstream_rq_time equivalents when meshed.
• ensure synthetic monitors match regional customer populations, not just nearest AWS region.
• document expected additional RTT when traffic hairpins through security inspection VPCs.
• record baseline when cluster healthy to spot slow regression during minor CNI upgrades.

Service Defaults And Headless Edge Cases

Closing Network Incidents

Before resolving the ticket, capture the delta between initial symptoms and final fix so future regressions can bisect faster.

1. Paste exact working curl invocation with redacted tokens.
2. Note whether change was config-only or required node reboots.
3. List monitoring gaps that delayed detection.
4. Confirm canaries still cover the failure class post-fix.

Link follow-up tasks to the originating SLO burn if customer impact occurred; network incidents often require cross-team backlog grooming.

Traceroute Hygiene

Document which hop produced each artifact so future shifts do not re-litigate network vs app ownership.

• Tag PCAP files with cluster, namespace, and directional arrow (client→Ingress vs Pod→Pod).
• Mention clock skew when stitching multi-system logs.
• Redact customer headers before uploading to ticket systems.

One-Line Smoke Tests

kubectl run netcheck --rm -i --restart=Never --image=curlimages/curl -- \
  curl -sS -o /dev/null -w "%{http_code}\n" http://kubernetes.default.svc.cluster.local
# Expect 403/401 from apiserver — proves Service routing + DNS alive.

Related Pages