GitOps Patterns — K8s SRE Reference

TL;DR

Git is the source of truth; a controller applies it. CI builds and pushes artifacts; GitOps promotes by updating manifest repos. Avoid manual kubectl changes unless you're in an incident path — and create a Git follow-up immediately.

Pull vs Push

Model	How deploy works	Tools
Pull (GitOps)	Controller watches Git, applies to cluster	ArgoCD, Flux
Push (CI deploy)	CI pipeline runs kubectl/helm against cluster	GitHub Actions, CircleCI, Jenkins
Hybrid	CI builds image + updates Git; controller syncs	Most enterprise setups

CI + GitOps Split

Hybrid GitOps: CI builds the artifact; the controller deploys from Git.

CI — test, build image, scan, push to registry with immutable tag.
CI or human — open PR updating image tag in the GitOps repo.
GitOps controller — detects merge, syncs to cluster.
SRE — watch Application status, rollout, metrics.

Repository Shapes

Pattern	Use when	Watch out
App repo owns manifests	Small teams; app and deploy move together	Platform policy duplicated across apps
Separate env repo	Many apps; central release control	Needs clear promotion process
Base + overlays	Same app across dev/stage/prod	Overlay patches grow unwieldy
App-of-apps	Bootstrap many ArgoCD apps from one root	Bad root sync affects all child apps
Monorepo	All apps and infra in one repo	Blast radius on bad merge

Kustomize Base & Overlays

This layout keeps shared Kubernetes YAML in a base and applies environment-specific patches from overlays. Use it when the same app needs small differences per environment without duplicating every manifest.

text repo-layout.txt

apps/web-api/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    dev/kustomization.yaml
    prod/kustomization.yaml

This production overlay reuses the base manifests, sets the target namespace, and patches only the fields that differ in prod. Use this style to keep environment changes visible in a small reviewable file.

yaml overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: app
images:
  - name: registry.example.com/web-api
    newTag: abc123f
replicas:
  - name: web-api
    count: 4
patches:
  - path: resources-patch.yaml

Helm Values in Git

This structure stores Helm values per environment in Git while the chart stays reusable. Use it when CI promotes by changing image tags or values, and ArgoCD or Flux performs the actual sync.

text helm-gitops-tree.txt

platform-gitops/
  apps/web-api/
    Chart.yaml              # Wrapper or references upstream chart.
    values-dev.yaml
    values-prod.yaml        # image.tag updated by CI promotion PR.
  clusters/prod/
    apps/
      web-api.yaml          # ArgoCD Application pointing at apps/web-api.

Promotion Workflow

This is the release flow behind most GitOps pipelines: build once, update Git for the target environment, review the diff, then let the controller sync. Use it as a checklist for dev-to-prod promotion.

bash promotion.sh

# 1. Confirm dev is healthy.
argocd app get web-api-dev
kubectl rollout status deploy/web-api -n dev

# 2. Promote via Git PR — update image tag in prod overlay/values.
git checkout -b promote-web-api-abc123f
yq -i '.image.tag = "abc123f"' apps/web-api/overlays/prod/values.yaml
git diff
git commit -am "Promote web-api abc123f to prod"
git push origin HEAD   # Open PR; require review before merge.

# 3. After merge, watch sync.
argocd app diff web-api-prod
kubectl rollout status deploy/web-api -n prod

App Of Apps

The app-of-apps pattern bootstraps many ArgoCD Applications from one root Application. Use it for platform bundles, but keep the root small because a bad root change can affect many child apps.

yaml root-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/platform-gitops.git
    targetRevision: main
    path: clusters/prod/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Secrets in Git

Approach	When to use
External Secrets Operator	Secrets live in Vault/AWS SM; synced to K8s at runtime
Sealed Secrets	Encrypted blobs safe to commit to Git
SOPS + age/PGP	Encrypted values files in Git (Flux native support)
Never plain secrets in Git	Always — even in private repos

Incident Rules

!Manual patch: if you kubectl patch during an incident, create a Git follow-up immediately or selfHeal will revert it.
!Pause sync carefully: disabling auto-sync is temporary incident control, not a deployment process.
!Find ownership first: locate the Application before changing Deployments, Services, or ConfigMaps.