TL;DR

Use this page when debugging connectivity from a node or pod: DNS resolution, port reachability, HTTP endpoints, TLS certificates, and raw packet capture. Start with read-only checks (dig, curl, ss) before reaching for tcpdump.

Interfaces & Routes

Check IP addresses, interface state, and routing table — always do this first when a node has unexpected connectivity or is missing from a load balancer.

bashinterfaces.sh
# Show all interfaces and IP addresses
ip addr show                    # modern replacement for ifconfig
ip addr show eth0               # specific interface
ip -4 addr                      # IPv4 only
ip -6 addr                      # IPv6 only

# Routing table
ip route show                   # main routing table
ip route get 8.8.8.8            # show which interface/gateway reaches an IP
ip route get 10.96.0.1          # useful for checking pod CIDR routing on nodes

# Interface statistics (packet drops, errors)
ip -s link show eth0
cat /proc/net/dev               # raw counters for all interfaces

# MTU — MTU mismatches cause silent drops for large packets
ip link show eth0 | grep mtu
# For tunneled overlay networks (Cilium, Calico VXLAN), MTU is typically ~50 bytes less

DNS Resolution

dig is the most reliable DNS debug tool; use it with +short for quick answers and without flags for full detail including TTL and authority.

bashdns.sh
# Basic lookups
dig example.com                 # full DNS response with TTL
dig +short example.com          # just the answer IPs
dig example.com MX              # mail records
dig example.com TXT             # TXT records (SPF, DKIM, ACME challenges)
dig -x 1.2.3.4                  # reverse lookup (PTR)

# Query a specific nameserver (bypass /etc/resolv.conf)
dig @8.8.8.8 example.com        # query Google's DNS directly
dig @10.96.0.10 my-svc.default.svc.cluster.local  # query CoreDNS inside cluster

# Check /etc/resolv.conf settings
cat /etc/resolv.conf            # nameserver, search domains, ndots

# resolvectl (systemd-resolved)
resolvectl status               # per-interface DNS config and status
resolvectl query example.com    # resolve with detailed trace

# nslookup (simpler, available everywhere)
nslookup example.com
nslookup my-svc.default.svc.cluster.local 10.96.0.10  # specific server

# host: simple forward/reverse
host example.com
host 1.2.3.4

Ports & Connections

Use ss (the modern netstat) to inspect listening ports and established connections; use lsof to find the process owning a port.

bashports.sh
# ss: socket statistics
ss -tlnp            # TCP listening, numeric ports, owning process
ss -tlnp sport = :443           # filter by port
ss -s               # summary of socket counts by state
ss -tp state established        # all established TCP connections
ss -u -ln           # UDP listening sockets
ss -4tlnp           # IPv4 only

# netstat (older but often present)
netstat -tlnp       # same as ss -tlnp

# lsof: find process using a port or file
lsof -i :8080                   # process listening on port 8080
lsof -i TCP:443                 # established TCP on 443
lsof -p <pid>                  # all files/sockets opened by a process
lsof /var/log/app.log           # which process has this file open

# Test port reachability from shell
nc -zv <host> <port>          # TCP connect test; exits 0 if open
timeout 3 bash -c 'cat /dev/null > /dev/tcp/<host>/<port>' && echo open
# /dev/tcp is a bash builtin — no nc needed

HTTP & API Testing

curl is your primary tool for testing HTTP endpoints, sending headers, posting data, and debugging TLS; use -v to see the full request/response cycle.

bashhttp.sh
# Basic requests
curl https://example.com                        # GET
curl -I https://example.com                     # headers only (HEAD)
curl -v https://example.com                     # verbose: shows TLS handshake, headers
curl -s -o /dev/null -w "%{http_code}" https://example.com  # just status code
curl -L https://example.com                     # follow redirects

# POST and PUT
curl -X POST https://api.example.com/data \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"key":"value"}'

# TLS debugging
curl -v --cacert /path/to/ca.crt https://internal.example.com
curl -k https://self-signed.example.com         # skip TLS verify (testing only)
curl --resolve myapp.example.com:443:10.0.0.5 https://myapp.example.com  # override DNS

# Kubernetes in-cluster API call (from a pod)
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
  https://kubernetes.default.svc/api/v1/namespaces/default/pods

# Timing breakdown (latency investigation)
curl -w "\nDNS:%{time_namelookup}s Connect:%{time_connect}s TLS:%{time_appconnect}s Total:%{time_total}s\n" \
  -o /dev/null -s https://example.com

TLS Certificate Inspection

Use openssl to inspect certificate chains, check expiry, and validate SANs before opening a support ticket or touching cert-manager.

bashtls.sh
# Inspect live server certificate
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>&1 | \
  openssl x509 -noout -text | grep -E "Subject:|DNS:|Not After"

# Quick expiry check
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Inspect a certificate file
openssl x509 -in cert.pem -noout -text        # full details
openssl x509 -in cert.pem -noout -subject -issuer -dates -ext subjectAltName

# Verify cert chain
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt cert.pem

# Inspect a Kubernetes TLS secret
kubectl get secret <tls-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -text | grep -E "Subject:|DNS:|Not After"

Reachability & Path

Use ping for basic ICMP reachability and traceroute / mtr to identify where packets are dropping in a path.

bashreachability.sh
ping -c 4 8.8.8.8               # 4 ICMP echo requests
ping -c 4 -I eth0 10.0.0.1     # force specific interface

# traceroute: show hop-by-hop path
traceroute -n 8.8.8.8           # -n skips reverse DNS (faster)
traceroute -T -p 443 example.com  # TCP traceroute on port 443 (bypasses ICMP filters)

# mtr: combined ping + traceroute, real-time
mtr --report --report-cycles 20 8.8.8.8

# iperf3: bandwidth test between two hosts
# on target host: iperf3 -s
# on source host:
iperf3 -c <target-ip> -t 10    # 10-second TCP bandwidth test

Packet Capture

Use tcpdump for low-level packet inspection; always capture to a file first and open in Wireshark if you need deep analysis. Be aware of compliance rules before capturing on production.

bashtcpdump.sh
# Capture to file for later analysis (avoids scrolling live output)
tcpdump -i eth0 -w /tmp/capture.pcap

# Common filters
tcpdump -i eth0 host 10.0.0.5           # traffic to/from specific IP
tcpdump -i any port 443                 # any interface, port 443
tcpdump -i eth0 'tcp and port 8080 and host 10.0.0.5'
tcpdump -i any 'port 53'               # DNS queries

# Show headers only (avoid seeing payload data)
tcpdump -i eth0 -nn port 80            # -nn: don't resolve hostnames or ports

# Read a saved capture
tcpdump -r /tmp/capture.pcap
tcpdump -r /tmp/capture.pcap 'tcp and port 443'

# Kubernetes node: capture on pod veth interface
# Find veth by looking up pod's sandbox PID
crictl inspect <container-id> | grep pid
nsenter -t <pid> -n tcpdump -i eth0 -w /tmp/pod-capture.pcap

Troubleshooting Map

SymptomFirst commandLikely cause
Can't resolve DNSdig @<nameserver> <name>Wrong nameserver, ndots mismatch, CoreDNS down
Port not reachablenc -zv <host> <port>Service not listening, firewall/NetworkPolicy, wrong VIP
HTTP 502/503curl -v <url>Backend pod not ready, upstream connection refused
TLS handshake failureopenssl s_client -connect <host>:443Expired cert, wrong SNI, missing CA, cipher mismatch
Intermittent dropsmtr --report <host>Packet loss at a hop, MTU mismatch
High latencycurl -w timing... DNS slow, TCP slow-start, TLS renegotiation