DevOps & CI/CD Scenarios Knowledge

Introduce build caching, parallelize independent test suites, use incremental builds, containerize build steps for consistency, split slow jobs, right-size runners, and offload heavy work to remote build systems when appropriate.

Add bounded retries with backoff, increase timeouts only where justified, use local mirrors or dependency proxies, stabilize artifact registry connectivity, and monitor network failure rate separately from test failures.

Check branch-specific environment variables, protected secrets, pipeline conditions, deployment rules, required approvals, and merge results. Validate whether CI config differs after merge or whether main has stricter gates.

Isolate the flaky step, capture logs and seed values, add retries with backoff as a short-term shield, containerize the environment, remove nondeterministic dependencies, and track flake rate until the root cause is fixed.

Keep approvals for high-risk changes, but automate low-risk approvals with policy-as-code, progressive delivery, canary checks, environment protection rules, and clear ownership. Manual gates should be intentional, not default friction.

Use a centralized secret and variable manager, validate required variables at pipeline start, avoid inline secrets, document environment contracts, and fail fast with a clear error before any deploy step.

Profile the step, identify hot tasks, optimize code or test selection, set runner resource classes deliberately, split work into parallel jobs, and offload heavy tasks to scalable runners or remote build infrastructure.

Pin versions, use lock files, cache dependencies, control base images, run scheduled dependency updates, and make dependency changes explicit through review rather than accidental fresh installs.

Use multi-stage builds, slim or distroless base images, remove build artifacts, order layers for cache reuse, use registry layer caching, and keep build dependencies out of runtime images.

Run both systems in parallel for a while, compare environment variables and secrets, validate runner images and permissions, migrate one workflow at a time, and use feature flags or branch-scoped rollout for high-risk steps.

Add config schema validation, preflight checks, canary releases, automated rollback, feature flags, review rules, and tests that exercise config combinations before rollout.

Check load balancer rules, target health checks, DNS TTL, readiness probes, service selectors, ingress or gateway routing, and whether both environments are serving compatible versions.

Check canary metrics, error budget burn, rollout thresholds, analysis templates, sample size, route weights, logs, and whether the canary is genuinely receiving representative traffic.

Compare secrets, ConfigMaps, feature flags, quotas, node types, resource limits, data shape, external dependencies, network policy, ingress settings, and version skew between environments.

Use backward-compatible schema changes, expand-and-contract migrations, versioned APIs, decoupled database migrations, compatibility tests, and rollback drills that include data and config state.

Use environment protection rules, explicit environment selection, approvals for sensitive targets, separate credentials, GitOps environment repos, clear naming, and guardrails that verify cluster identity before deployment.

Check readiness probes, dependency connectivity, database access, misconfigured ports, config and secrets, resource pressure, startup time, and Pod events. Readiness should represent actual ability to serve traffic.

Inspect logs, liveness probe settings, startup probes, resource limits, dependency failures, configuration changes, and node pressure. Bad probes can amplify a slow startup into a restart loop.

Roll back or stop rollout if impact is growing, gather memory profiles, compare with canary metrics, set safe resource limits, add alerts, and fix the leak before resuming rollout.

Add performance and load tests to CI, simulate peak traffic, test autoscaling behavior, measure saturation and tail latency, and ensure the canary receives enough load to reveal regressions.

Use terraform plan gates, mandatory reviews, state locking, drift detection, deletion protection, least-privilege apply roles, policy checks, and backups for state and critical resources.

Restore from remote backend versioning if available, avoid manual edits unless absolutely necessary, re-import resources, verify state addresses, enable locking, and document the recovery steps.

Split monolithic state into smaller modules or workspaces, reduce provider calls, avoid unnecessary data sources, use targeted plans only with care, and parallelize independent stacks through orchestration.

Use semantic versioning, pin module versions, publish changelogs, test modules in isolation, run integration tests for downstream consumers, and avoid breaking input or output contracts without a major version.

Enable SSH pipelining, use async tasks where safe, reduce fact gathering, cache facts, batch hosts deliberately, avoid shell loops when modules exist, and remove unnecessary serial bottlenecks.

Identify whether drift is emergency, accidental, or unmanaged. Reconcile through Git or IaC, import legitimate changes, alert on future drift, and restrict manual access where possible.

Import it into IaC or recreate it through IaC, add policy-as-code controls, restrict manual permissions, tag ownership, and make future manual creation visible through drift detection.

Batch operations, add retries with backoff, tune provider throttling controls, reduce parallelism if needed, cache lookups, split high-churn stacks, and coordinate deploy windows.

Rotate the secret immediately, revoke old credentials, scan for use, purge history where required, add secret scanners, pre-commit hooks, protected variables, and move secrets to managed storage.

Increase cooldowns, widen thresholds, use longer evaluation windows, inspect load patterns, tune health checks, and avoid scaling decisions on a noisy single metric.

Find the owning GitOps object, commit intended changes to Git, educate teams on source-of-truth behavior, and restrict direct production edits except documented emergency workflows.

Use webhooks, reduce sync interval carefully, optimize manifest generation, split large apps, reduce repo size, and check controller CPU, memory, and API server throttling.

Add YAML linting, schema validation, server-side dry runs, kubeconform or kubeval checks, OPA or Kyverno policy checks, and CI gates before manifests reach the GitOps controller.

Use environment branches or directories consistently, require PR reviews, reduce concurrent edits to the same files, automate image tag updates safely, and use clear ownership for shared overlays.

Batch low-risk changes, use release branches, deployment windows, progressive delivery, and policy rules that separate build frequency from production rollout frequency.

Use sealed secrets, external secret operators, environment-specific secret stores, clear bootstrap ordering, and validation that referenced secret names exist before rollout.

Use cluster selectors, environment labels, separate repos or folders per environment, explicit project permissions, unique cluster names, and checks that verify destination cluster identity.

Split manifests into modules, use Kustomize overlays or Helm values, reduce duplication, avoid committing generated noise, and keep CRDs and app resources organized by lifecycle.

Manifests and images should be versioned, history should be retained, database changes should be backward compatible, and rollback should be tested as a normal release operation.

Upgrade CRDs first, validate API versions, confirm controllers support the new CRDs, then roll out custom resources. Avoid applying resources before their CRDs and controllers are ready.

Autoscale runners, use ephemeral runners, separate heavy and light workloads, distribute queues, cap concurrency for expensive jobs, and measure queue time as a capacity signal.

Enable retention policies, prune old images and artifacts, use lifecycle rules, deduplicate layers, tag releases clearly, and keep only artifacts required for rollback and audit.

Check registry saturation, network path, regional distance, authentication latency, layer size, cache hit rate, and whether regional mirrors or pull-through caches can reduce latency.

Use weighted queues, labels, autoscaling, workload sharding, runner affinity, and separate pools for CPU-heavy, memory-heavy, and privileged jobs.

Remove redundant stages, modularize repeated logic, standardize templates, separate build from deploy, make environment promotion explicit, and keep each stage's purpose clear.

Use versioned artifacts, immutable image tags, GitOps rollbacks, database-compatible changes, automated rollback triggers, canary analysis, and documented manual rollback steps.

Enforce branch protection, PR reviews, required CI checks, CODEOWNERS, signed commits if needed, and protected deploy environments.

Add a regression test for the failure, improve coverage around the class of bug, add contract or integration tests where needed, and ensure the test runs before deploy.

Require unit tests, integration tests, security scans, policy checks, artifact signing, config validation, and environment-specific deploy gates before production rollout.

Use small changes, canary releases, feature flags, automated rollback, strong observability, fast tests, ownership, progressive delivery, and a culture that treats rollback as normal.