Operators & CRDs — K8s SRE Reference

TL;DR

A CRD adds a new API type to Kubernetes. A custom resource is an instance of that type. An operator is a controller that watches those custom resources and reconciles real infrastructure or application state.

Mental Model

Item	Meaning
CRD	Extends the Kubernetes API with a new resource type.
Custom Resource	A YAML object using that CRD, such as `Prometheus` or `Certificate`.
Operator	Controller that watches custom resources and takes action.
Reconcile loop	Repeatedly makes actual state match desired state.

Reconciliation Diagram

Operator pattern: declarative CR drives imperative creation of primitives; status summarizes reconciliation health.

Daily Commands

bashcrd-operator-checks.sh

# List CRDs and discover custom API resources.
kubectl get crd
kubectl api-resources | grep -i '<keyword>'

# Inspect a CRD schema, versions, and conversion settings.
kubectl describe crd <crd-name>

# List custom resources after finding their kind/resource name.
kubectl get <custom-resource> -A
kubectl describe <custom-resource> <name> -n <namespace>

# Find operator Pods and logs.
kubectl get pods -A | grep -i operator
kubectl logs -n <operator-namespace> deploy/<operator-deployment> --tail=200

Minimal CRD Shape

yamlsample-crd.yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.platform.example.com # plural.group
spec:
  group: platform.example.com
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    kind: Backup
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule:
                  type: string

Install And Upgrade Notes

Install CRDs before installing custom resources that depend on them.
CRDs are cluster-scoped and often require elevated permissions.
Helm treats CRDs specially when placed in a chart's crds/ directory; upgrades may need separate CRD management.
Operator upgrades should be tested because CRD schema/version changes can break existing custom resources.

Helm Example

Use OCI registries when available so chart + version are immutable; mirror into internal registries for air-gap.

bashhelm-operator-install.sh

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true \
  --version v1.15.4

kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

Debugging CR Status

The root problem is rarely “YAML failed to apply”; inspect subresources (/status), generation vs observedGeneration, managed fields, conditions, events, finalizers.

bashcrd-status.sh

# Replace with your plural.group and CR name once discovered via `kubectl api-resources`.
RESOURCE=postgresclusters.postgres-operator.crunchydata.com

kubectl describe "$RESOURCE" my-db -n data
kubectl get "$RESOURCE" my-db -n data -o jsonpath='{.status.conditions[*].message}{"\n"}'
kubectl get "$RESOURCE" my-db -n data -w # watch status until conditions stabilize

Troubleshooting Operators

!Custom resource stuck: check status conditions, events, finalizers, and operator logs.
!Delete stuck: finalizers usually mean the operator is trying to clean external resources.
!No reconciliation: operator may lack RBAC, be crashlooping, or watch a different namespace.
!API errors: check CRD schema, version served/storage flags, and conversion webhooks.