Certificate Management — K8s SRE Reference

TL;DR

Certificate management is the lifecycle around TLS identity: request, validation, issuance, storage, use, renewal, and revocation. In Kubernetes, cert-manager commonly watches Certificate resources, uses an Issuer or ClusterIssuer such as ACME, CA, Vault, or Venafi, and writes a kubernetes.io/tls Secret consumed by Ingress, Gateway, webhooks, service mesh, or apps.

Mental Model

A certificate proves that a public key belongs to a DNS name or identity. A private key stays secret. A certificate authority signs the certificate after validating that the requester is allowed to use the name. Kubernetes stores the final cert/key pair in a TLS Secret; controllers such as Ingress read that Secret to terminate HTTPS.

Separate storage from lifecycle: Secrets Management covers how sensitive material is stored and delivered. Certificate Management covers how the cert/key is issued, trusted, renewed, and debugged.

cert-manager turns certificate intent into a renewed TLS Secret that serving controllers consume.

TLS Secret Shape

A TLS Secret stores the private key and certificate chain. The Secret must be in the same namespace as the Ingress or application that references it, unless a controller has its own cross-namespace reference model.

yamltls-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: web-api-tls
  namespace: app
type: kubernetes.io/tls
data:
  tls.crt: BASE64_CERT_CHAIN
  tls.key: BASE64_PRIVATE_KEY

Issuer Vs ClusterIssuer

Object	Scope	Use When
`Issuer`	Namespace	One team or namespace owns issuance settings.
`ClusterIssuer`	Cluster	Platform team offers shared issuance for many namespaces.
`Certificate`	Namespace	Requests a cert and names the target TLS Secret.
`CertificateRequest`	Namespace	Generated request; useful for approval/signing troubleshooting.
`Order`/`Challenge`	Namespace	ACME flow details for Let's Encrypt or other ACME CAs.

ACME With cert-manager

ACME is an automated certificate issuance protocol used by Let’s Encrypt and other CAs. cert-manager creates Orders and Challenges to prove control of a DNS name. Use the Let’s Encrypt staging endpoint first in labs to avoid production rate limits.

yamlclusterissuer-letsencrypt-http01.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: platform@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

HTTP-01 Vs DNS-01

Challenge	How It Proves Ownership	Use When	Common Failure
HTTP-01	CA reaches `http://domain/.well-known/acme-challenge/...`.	Public HTTP routing to the cluster works.	DNS/LB/Ingress does not route challenge traffic to solver Pod.
DNS-01	CA sees a TXT record under `_acme-challenge.domain`.	Wildcard certs, private clusters, internal ingress, no public HTTP path.	DNS provider credentials, propagation delay, wrong hosted zone.

yamlclusterissuer-dns01-route53.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    email: platform@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-dns01-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            hostedZoneID: Z1234567890
            accessKeyIDSecretRef:
              name: route53-credentials
              key: access-key-id
            secretAccessKeySecretRef:
              name: route53-credentials
              key: secret-access-key

Certificate Resource

yamlcertificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: web-api
  namespace: app
spec:
  secretName: web-api-tls
  dnsNames:
    - api.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  privateKey:
    algorithm: RSA
    size: 2048
  usages:
    - digital signature
    - key encipherment
    - server auth

Ingress TLS

Ingress can reference a pre-created TLS Secret, or cert-manager can create a Certificate based on Ingress annotations. In GitOps environments, many teams prefer explicit Certificate resources because ownership is clearer.

yamlingress-tls.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-api
  namespace: app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: web-api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-api
                port:
                  number: 80

Venafi Integration

Enterprise clients often use Venafi TLS Protect/Trust Protection Platform or Venafi as a Service to enforce certificate policy, approval, ownership, and audit. cert-manager has a Venafi issuer type that can request certificates from a Venafi policy zone.

Field cueWhen a client says “Venafi,” expect enterprise certificate policy: approved zones, ownership metadata, custom fields, internal CA chains, and change-control around issuance.

yamlvenafi-tpp-clusterissuer.yaml

apiVersion: v1
kind: Secret
metadata:
  name: venafi-tpp-credentials
  namespace: cert-manager
type: Opaque
stringData:
  username: tpp-api-user
  password: replace-me
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: venafi-tpp
spec:
  venafi:
    zone: "\\VED\\Policy\\Kubernetes\\Apps"
    tpp:
      url: https://tpp.example.com/vedsdk
      credentialsRef:
        name: venafi-tpp-credentials

yamlvenafi-certificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-api
  namespace: app
spec:
  secretName: internal-api-tls
  commonName: internal-api.apps.example.corp
  dnsNames:
    - internal-api.apps.example.corp
  issuerRef:
    name: venafi-tpp
    kind: ClusterIssuer

Internal PKI And CA Issuers

Internal-only services may use a private CA instead of public ACME. cert-manager can use CA, Vault, Venafi, or other issuer integrations. For service mesh or webhook certificates, pay attention to trust bundles and CA rotation.

yamlca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: internal-ca
  namespace: app
spec:
  ca:
    secretName: internal-ca-keypair

Inspect And Debug

bashcert-debug.sh

NS=<namespace>
CERT=<certificate-name>
SECRET=<tls-secret-name>
HOST=<hostname>

kubectl get certificate,certificaterequest,order,challenge -n "$NS"
kubectl describe certificate "$CERT" -n "$NS"
kubectl describe certificaterequest -n "$NS"
kubectl describe issuer -n "$NS"
kubectl describe clusterissuer <clusterissuer>
kubectl logs -n cert-manager deploy/cert-manager --tail=200

kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.tls\\.crt}' | base64 -d > /tmp/tls.crt
openssl x509 -in /tmp/tls.crt -noout -subject -issuer -dates -ext subjectAltName

openssl s_client -connect "$HOST:443" -servername "$HOST" -showcerts



    Symptom To Cause
    Symptom Likely Cause Check First
Certificate stuck Pending Issuer not ready, ACME challenge pending, approval/signing problem. describe certificate, CertificateRequest, Order, Challenge.
HTTP-01 challenge fails Public DNS/LB/Ingress cannot reach solver path. Solver Ingress/Pod/Service and curl challenge URL.
DNS-01 challenge fails DNS credentials, wrong zone, slow propagation. Challenge events and TXT record lookup.
Browser shows wrong cert Ingress references wrong Secret, default cert served, SNI mismatch. Ingress TLS block and openssl s_client -servername.
Certificate name mismatch SAN does not include hostname used by clients. Certificate dnsNames and live cert SAN.
Incomplete chain Intermediate cert missing from tls.crt. OpenSSL chain output and CA bundle expectations.
Renewal did not happen cert-manager down, issuer failure, Secret manually changed, clock/time issue. Certificate status, events, cert-manager logs.
Venafi request denied Policy zone, credentials, custom fields, approval workflow. ClusterIssuer status, cert-manager logs, Venafi policy/audit.

    Production Patterns
    
      1Use staging first: validate ACME config with staging before production issuance.
      2Prefer DNS-01 for wildcards: HTTP-01 cannot issue wildcard certificates.
      3Monitor expiry: alert before certificates reach emergency renewal windows.
      4Protect private keys: TLS Secrets need strong RBAC and encryption at rest.
      5Know the owner: app teams may own Certificate resources while platform owns ClusterIssuers.
    

    Safe Change Pattern
    
      Identify what consumes the certificate: Ingress, Gateway, webhook, service mesh, or app.
      Confirm the source of truth: explicit Certificate, Ingress annotation, Helm chart, GitOps repo, Venafi policy, or cloud platform.
      Validate DNS names, issuer reference, Secret namespace, and challenge method before applying.
      After issuance, inspect the Kubernetes Secret and the live endpoint with SNI.
      Document renewal owner, expiry monitoring, and rollback path.
    

    Related Pages
IngressSecrets ManagementRBACOperators & CRDsIstioPrometheus

Symptom	Likely Cause	Check First
Certificate stuck Pending	Issuer not ready, ACME challenge pending, approval/signing problem.	`describe certificate`, CertificateRequest, Order, Challenge.
HTTP-01 challenge fails	Public DNS/LB/Ingress cannot reach solver path.	Solver Ingress/Pod/Service and curl challenge URL.
DNS-01 challenge fails	DNS credentials, wrong zone, slow propagation.	Challenge events and TXT record lookup.
Browser shows wrong cert	Ingress references wrong Secret, default cert served, SNI mismatch.	Ingress TLS block and `openssl s_client -servername`.
Certificate name mismatch	SAN does not include hostname used by clients.	Certificate `dnsNames` and live cert SAN.
Incomplete chain	Intermediate cert missing from `tls.crt`.	OpenSSL chain output and CA bundle expectations.
Renewal did not happen	cert-manager down, issuer failure, Secret manually changed, clock/time issue.	Certificate status, events, cert-manager logs.
Venafi request denied	Policy zone, credentials, custom fields, approval workflow.	ClusterIssuer status, cert-manager logs, Venafi policy/audit.