Services & Load Balancing

Mental Model

Pods are disposable. Every restart can create a new Pod name and IP address, so other workloads should not call Pod IPs directly. A Service is the stable contract in front of those Pods: one name, one virtual IP, one port mapping, and a selector that decides which Pods are eligible backends.

For an SRE, the important distinction is this: the Service object does not run your application. It only points traffic at Pods. If the Service has no endpoints, traffic has nowhere useful to go.

How It Works

1. You create Pods, usually through a Deployment or StatefulSet.
2. You add labels to those Pods, such as app: web-api.
3. You create a Service with a selector that matches those labels.
4. Kubernetes creates EndpointSlices containing the IPs and ports of matching ready Pods.
5. CoreDNS creates DNS records such as web-api.app.svc.cluster.local.
6. The node dataplane, commonly kube-proxy iptables/IPVS or an eBPF CNI, routes Service traffic to backend Pods.

Service Types

Build From Scratch

This lab creates a namespace, a simple HTTP Deployment, and a ClusterIP Service. The point is to learn the relationship between labels, selectors, ports, and endpoints.

apiVersion: v1
kind: Namespace
metadata:
  name: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-api
  namespace: app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: web
          image: nginxdemos/hello:plain-text
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web-api
  namespace: app
spec:
  type: ClusterIP
  selector:
    app: web-api
  ports:
    - name: http
      port: 80
      targetPort: http

kubectl apply -f web-api-service-lab.yaml
kubectl rollout status deploy/web-api -n app

# See the Service and the Pods it should select.
kubectl get svc web-api -n app -o wide
kubectl get pods -n app -l app=web-api -o wide --show-labels

# EndpointSlice is the real backend list used by modern Kubernetes.
kubectl get endpointslice -n app -l kubernetes.io/service-name=web-api -o wide

# Test from inside the cluster.
kubectl run curl -n app --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sS http://web-api.app.svc.cluster.local

Ports And Selectors

Most Service mistakes happen in two tiny fields: selector and targetPort. The selector must match Pod labels. The Service port is what clients call. The targetPort is where the container is listening.

Service DNS

Inside the cluster, CoreDNS gives Services predictable names. Pods in the same namespace can usually call http://web-api. Pods in another namespace should use web-api.app or the full name web-api.app.svc.cluster.local.

kubectl run netshoot -n app --rm -it --image=nicolaka/netshoot -- /bin/bash

nslookup web-api
nslookup web-api.app
nslookup web-api.app.svc.cluster.local

curl -v http://web-api
curl -v http://web-api.app.svc.cluster.local

LoadBalancer Services

A LoadBalancer Service asks the infrastructure provider to create an external or internal load balancer. In AWS EKS, Azure AKS, GKE, OpenStack, or on-prem MetalLB, a controller watches the Service and provisions provider-specific resources.

apiVersion: v1
kind: Service
metadata:
  name: web-api-public
  namespace: app
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app: web-api
  ports:
    - name: http
      port: 80
      targetPort: http

Headless Services

A headless Service sets clusterIP: None. Instead of returning one virtual IP, DNS returns individual Pod records. This is common for StatefulSets where clients need stable identities such as mysql-0.mysql.data.svc.cluster.local.

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: data
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
    - name: mysql
      port: 3306
      targetPort: 3306

Production Defaults

• 1Name ports: use names like http, grpc, and metrics so probes, Services, and NetworkPolicies stay readable.
• 2Use readiness probes: only ready Pods should receive Service traffic.
• 3Keep internal services internal: prefer ClusterIP unless external access is truly required.
• 4Track ownership: check whether the Service is managed by Helm, ArgoCD, Flux, Terraform, or a cloud controller before patching.
• 5Document cloud annotations: provider-specific annotations are operational behavior, not decoration.

Debugging Checklist

Debug Services from inside out. First prove the Pods are healthy, then prove the Service points at them, then test DNS, dataplane, network policy, and external load balancer behavior.

NS=<namespace>
SVC=<service>
APP=<app-label-value>

# 1. Inspect the Service contract.
kubectl get svc "$SVC" -n "$NS" -o wide
kubectl describe svc "$SVC" -n "$NS"
kubectl get svc "$SVC" -n "$NS" -o yaml

# 2. Check selected Pods and readiness.
kubectl get pods -n "$NS" -l app="$APP" -o wide --show-labels
kubectl describe pods -n "$NS" -l app="$APP"

# 3. EndpointSlices should contain ready backend addresses.
kubectl get endpointslice -n "$NS" -l kubernetes.io/service-name="$SVC" -o wide
kubectl get endpointslice -n "$NS" -l kubernetes.io/service-name="$SVC" -o yaml

# 4. Test from inside the cluster.
kubectl run netshoot -n "$NS" --rm -it --image=nicolaka/netshoot -- /bin/bash
nslookup "$SVC.$NS.svc.cluster.local"
curl -vk "http://$SVC.$NS.svc.cluster.local:80"

kube-proxy And CNI Dataplanes

Traditional clusters use kube-proxy to program iptables or IPVS rules. Some CNIs, especially eBPF-based dataplanes such as Cilium, can replace kube-proxy. You do not need to memorize every implementation to troubleshoot well; prove the Kubernetes objects first, then inspect the active dataplane.

kubectl get pods -n kube-system -o wide | grep -E 'kube-proxy|cilium|calico'

# kube-proxy clusters.
kubectl get daemonset -n kube-system kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=100

# CNI-specific checks vary by environment.
kubectl get pods -n kube-system -l k8s-app=cilium -o wide
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide

Symptom To Cause

Safe Change Pattern

1. Confirm ownership with labels and annotations such as app.kubernetes.io/managed-by, Helm release metadata, or ArgoCD labels.
2. Fix the source of truth: Helm values, GitOps manifests, Terraform, or platform module.
3. Use kubectl diff, Helm template output, ArgoCD diff, or Terraform plan before applying.
4. After rollout, verify Service endpoints, DNS, in-cluster curl, and any external health checks.

Related Pages