CircleCI — K8s SRE Reference

TL;DR

CircleCI pipelines live in .circleci/config.yml. Separate build, validate, and promote jobs. When ArgoCD/Flux owns deploys, CI updates the GitOps repo — don't run helm upgrade from CircleCI unless the client explicitly uses push-based deploy.

Concepts

Concept	Meaning
Job	Steps running in an executor (Docker image or machine)
Workflow	Job orchestration — dependencies, filters, approvals
Context	Shared secrets scoped to teams/projects
Workspace	Pass artifacts between jobs in a workflow

GitOps Promotion Job

Preferred pattern when ArgoCD/Flux manages the cluster — CI updates Git, not the cluster directly.

yaml .circleci/config.yml

  promote-gitops:
    docker:
      - image: cimg/base:stable
    steps:
      - run:
          name: Update GitOps repo
          command: |
            git clone https://x-access-token:${GITHUB_TOKEN}@github.com/example/platform-gitops.git
            cd platform-gitops
            yq -i '.image.tag = env(CIRCLE_SHA1)' apps/web-api/overlays/dev/values.yaml
            git config user.email "ci@example.com"
            git config user.name "CircleCI"
            git checkout -b release/web-api-${CIRCLE_SHA1}
            git commit -am "Promote web-api ${CIRCLE_SHA1}"
            git push origin HEAD

Direct Helm Deploy

Use only when the client does not use GitOps for this app. See Helm in CI/CD for full deploy script.

yaml deploy-prod job

  deploy-prod:
    docker:
      - image: alpine/k8s:1.29.2
    steps:
      - checkout
      - run:
          name: Deploy with Helm
          command: |
            echo "$KUBECONFIG_B64" | base64 -d > kubeconfig
            export KUBECONFIG="$PWD/kubeconfig"
            helm upgrade --install web-api charts/web-api \
              --namespace app-prod --create-namespace \
              --values charts/web-api/values-prod.yaml \
              --set image.tag="${CIRCLE_SHA1}" \
              --atomic --wait --timeout 10m

workflows:
  build-test-deploy:
    jobs:
      - build-image
      - validate-manifests:
          requires: [build-image]
      - hold-for-prod:
          type: approval
          requires: [validate-manifests]
      - deploy-prod:
          context: kube-prod
          requires: [hold-for-prod]
          filters:
            branches:
              only: main

Validate Manifests

This CircleCI job renders manifests and checks them before deployment. Use it as a required gate so broken Helm or Kustomize output fails in CI instead of reaching the cluster.

yaml validate-manifests job

  validate-manifests:
    docker:
      - image: alpine/helm:3.14.0
    steps:
      - checkout
      - run:
          name: Lint and render
          command: |
            helm dependency build charts/web-api
            helm lint charts/web-api
            helm template web-api charts/web-api \
              --namespace app-prod \
              --values charts/web-api/values-prod.yaml \
              --set image.tag="${CIRCLE_SHA1}" > rendered.yaml

Debug Failed Jobs

Use these commands when a CircleCI workflow fails and you need to determine whether the problem is credentials, contexts, branch filters, missing artifacts, or the deploy command itself.

bash circleci-debug.sh

circleci config validate
circleci local execute --job validate-manifests

# After deploy failure.
kubectl auth can-i create deploy -n app-prod
kubectl get events -n app-prod --sort-by=.lastTimestamp | tail -n 30
kubectl describe deploy web-api -n app-prod

Gotchas

!Store kube credentials in restricted contexts, not project env vars visible to all jobs.
!Use branch filters — feature branches must not deploy to production.
!Prefer OIDC/cloud federation over long-lived KUBECONFIG_B64 secrets when available.
!Don't deploy via CircleCI if ArgoCD owns the same release — you'll create ownership conflicts.