Amazon EKS Deep Dive

Architecture & Trust Boundaries

Unlike kubeadm, you never SSH to Kubernetes masters: AWS scales and patches the apiserver plane. Your responsibility is subnets, IAM, addons, workloads, and change windows during EKS platform version upgrades.

Creating & Accessing Clusters

# Typical flow after Terraform or eksctl provisioning.
aws eks update-kubeconfig --name prod-platform --region us-east-1

# Inspect platform version vs Kubernetes minor (they differ — check AWS docs).
kubectl version -o yaml
kubectl get nodes -o wide

# STS caller identity confirms which IAM principal your kubeconfig wrapper uses.
aws sts get-caller-identity

Compute: Node Groups & Alternatives

# Terraform / eksctl equivalents set this; illustrative node labels/taints shape.
labels:
  workload: general
  topology.kubernetes.io/zone: "${AZ}" # Often set automatically from subnet.
taints:
  - key: "nvidia.com/gpu"
    value: "shared"
    effect: "NoSchedule"

IRSA — Pod IAM Without Static Keys

Map a Kubernetes ServiceAccount to an IAM role backed by your cluster OIDC issuer. Provision the IAM role and trust policy via Terraform IRSA pattern; annotate the SA in manifests or Helm.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sqs-consumer
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/prod-app-sqs
    eks.amazonaws.com/sts-regional-endpoints: "true"  # Helps when STS global endpoint is flaky.

Detailed trust boundaries and SG patterns: AWS IAM & Security Groups.

EKS Add-ons & Versioning

AWS distributes tested versions of VPC CNI, CoreDNS, kube-proxy, CSI drivers, Pod Identity Agent, etc. Decide who owns Helm vs EKS-managed add-ons to avoid duplication.

aws eks list-addons --cluster-name prod-platform --region us-east-1
aws eks describe-addon --cluster-name prod-platform --addon-name vpc-cni --region us-east-1

AWS Load Balancer Controller

Implements Ingress (and Gateway API progress) against AWS elastic load balancing. Depends on subnets tagged per scheme (public/private internal), IRSA IAM policy, optional WAF integrations, target-type IP vs instance. See Kubernetes Service nuances in Services & Load Balancers.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: web.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

Cluster Autoscaler Basics

Cluster Autoscaler inspects Pods stuck in Pending, consults Scheduler constraints, scales ASGs/MNGs within min/max, and drains nodes gracefully. Separate from Horizontal Pod Autoscaler (pods) and complements Karpenter for different org standards.

# RBAC-heavy component — confirm deployment args match ASG/MNG discovery tags/cloud provider.
kubectl -n kube-system logs deploy/cluster-autoscaler --tail=200

# Pods waiting for topology / resources — CA reacts only when scheduling truly fails scale-out.
kubectl get events -A --sort-by=.lastTimestamp | tail -40

• IAM: controller needs ec2/describe/terminate plus autoscaling per AWS docs (often IRSA).
• Each node group exposes min/max/desired caps — CA cannot exceed AWS ASG boundaries.
• Cluster-wide upgrades happen control-plane-first; cordon/drain node groups thoughtfully.

Helm Shape: AWS Load Balancer Controller

Below is a representative values.yaml fragment—pin chart versions in your pipeline the same way you pin Terraform providers. IRSA role ARNs must exist before helm upgrade applies.

clusterName: prod-platform
region: us-east-1
vpcId: vpc-0123456789abcdef0
serviceAccount:
  create: true
  name: aws-load-balancer-controller
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/prod-alb-controller
enableServiceMutatorWebhook: true
ingressClassConfig:
  default: true
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    memory: 512Mi
nodeSelector: {}
tolerations: []
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - aws-load-balancer-controller
          topologyKey: kubernetes.io/hostname
defaultTags:
  Environment: prod
  Cluster: prod-platform
logLevel: info
enableShield: false
enableWaf: false
enableWafv2: false

VPC CNI & Prefix Delegation

Prefix delegation increases IP density per ENI—critical for dense pod counts before hitting ENI quotas. Coordinate warm pool settings with application burst patterns; mis-tuned settings show up as FailedCreatePodSandBox events while Services still appear healthy at the control plane.

Fargate & Windows Footnotes

• Fargate: DaemonSets (CNI logging, node-exporter patterns) do not exist—shift observability sidecars into Deployments or adopt Fargate-aware agents only.
• Windows nodes: Separate MNG pools, distinct tolerations on workloads, patch cycles differ from Linux AMIs.
• Cluster Autoscaler still needs IAM awareness for each ASG—even if workloads are ephemeral Fargate, static MNG pools may coexist.

Operational Checklists

VPC, Subnets & Routing

Worker nodes commonly live on private subnets with NAT gateways for egress. Elastic load balancers for public Ingress may materialize either in subnets tagged kubernetes.io/role/elb or internal-only subnets tagged kubernetes.io/role/internal-elb (Terraform sample tags). Cross-AZ SG rules plus NACL pitfalls still apply—when NodePort or hostNetwork patterns appear during incidents, correlate with our Services guidance before blindly editing SG ingress.

# Compare AWS subnet tags consumed by CCM / LB controller automation.
aws ec2 describe-subnets \
  --filters "Name=tag:kubernetes.io/cluster/prod-platform,Values=owned" \
  --query 'Subnets[*].{ID:SubnetId,AZ:AvailabilityZone,Name:Tags[?Key==`Name`].Value|[0]}'

APIServer Authorization & Access Entries

# Validate effective RBAC independent of IAM wrapper (after kubeconfig merges).
kubectl auth can-i list secrets -n kube-system
kubectl auth can-i create pods --as=system:serviceaccount:default:debugger

Kubernetes & Platform Upgrades

Advance one minor Kubernetes version per maintenance window whenever possible — skip versions only when AWS publishes explicit exemption guidance. Rotate node groups progressively: bootstrap new AMI groups, cordon+d older nodes while honoring PodDisruptionBudgets, shrink old ASGs only after DaemonSets report healthy replacements.

kubectl get apiservice | grep False
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get pods -A | grep -vE 'Running|Completed' || true
kubectl describe nodes | grep -iE 'pressure|Kubelet' || true

NLB Shape For Service Kind LoadBalancer

Some teams prefer Kubernetes Service=LoadBalancer with NLB annotations while others standardize purely on Ingress. Keep annotations consistent cluster-wide (nlb-target-type, health probes, cross-zone).

apiVersion: v1
kind: Service
metadata:
  name: edge-tcp
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
  type: LoadBalancer
  selector:
    app: tcp-proxy
  ports:
    - name: tcp5443
      port: 5443
      targetPort: 5443

Troubleshooting Matrix

Rolling Control Plane Upgrades (Shape)

#!/usr/bin/env bash
set -euo pipefail
CLUSTER="${CLUSTER:-prod-platform}"
REGION="${REGION:-us-east-1}"

# 1) advance control plane version after change window approval
aws eks update-cluster-version \
  --name "$CLUSTER" --region "$REGION" \
  --kubernetes-version "${TARGET_MINOR:-1.30}"

# 2) wait until ACTIVE between dependent steps — poll with backoff externally
until aws eks describe-cluster --name "$CLUSTER" --region "$REGION" \
  --query 'cluster.status' --output text | grep -qx ACTIVE; do
  echo "waiting control plane converge..."
  sleep 30
done

# 3) refresh node AMI / kubelet per nodegroup name from IaC outputs
NODEGROUP=$(aws eks list-nodegroups --cluster-name "$CLUSTER" --region "$REGION" \
  --query 'nodegroups[0]' --output text)
echo "planned rolling update targeting $NODEGROUP"

# 4) reconcile addons after nodes healthy — ensure compatibility matrix consulted
aws eks list-addons --cluster-name "$CLUSTER" --region "$REGION"

# Document manual verification gates (Ingress smoke, STS IRSA workloads) before declaring complete.

AWS Quotas & Limits To Track

• ENI quotas per instance type interplay with Pods when prefix delegation disabled.
• ELB quotas per region—large ingress churn during testing exhausts quotas quickly.
• EC2 Auto Scaling API rate limits amplified by aggressive Cluster Autoscaler loops.
• Security group rule counts including cross-referenced LB + node SG combos.
• IAM roles per account when each micro-service demands unique IRSA role.
• Route53 ChangeResourceRecordSets throttling mirrored by ExternalDNS logs.
• CloudWatch Logs ingestion spikes when apiserver audit verbose.
• EKS addon API throttling surfaced as Terraform apply retries needing backoff.
• STS regional endpoint throughput during massive rollout events.
• EBS BurstBalance alarms when log-heavy nodes share gp2 pools.
• Target group Attachment limits per LB complicate multi-namespace ingress designs.
• API Discovery publish QPS spikes around CRD churn during Helm upgrades.
• WAF ACL association limits pairing with controllers toggling shields.
• Cross-AZ NAT Gateway bandwidth costs mistaken as application latency regressions.
• KMS requests per second when many pods concurrently decrypt envelopes.
• Service Quotas uplift tickets should reference FinOps stakeholder approval paths.

Surface limits early in sizing reviews alongside Terraform-driven IaC manifests so limits become code-reviewed constants.

Gotchas

• !VPC mismatch: wrong subnets → nodes never join or LBs provision in the wrong SG.
• !IRSA annotation typo: subtle namespace/SA mismatch → SDK falls back to node role (least surprise permissions).
• !Security group sprawl: default cluster SG edits can break apiserver/worker signaling — track changes carefully.
• !Add-on duplication: two CoreDNS controllers or vpc-cni versions cause hard-to-debug iptables/IPAM errors.
• !CA vs PDB: aggressive PodDisruptionBudgets can block scale-down for long periods.
• !Ingress LB pending: usually IAM/IRSA/subnet tags on the LB controller pod — correlate with Events.

Related Pages