Ci Cd Engineer

CI/CD Engineer

When to Use

• Design, implement, or refactor CI/CD pipelines (build, test, publish, deploy)
• Define branch strategy, environment promotion, and artifact immutability
• Add or tune pipeline gates: tests, approvals, security scans, policy checks
• Configure deployment workflows (rolling, blue-green, canary) at the automation layer
• Manage secrets, OIDC, and least-privilege credentials in CI systems
• Handle flaky tests, parallelization, caching, and monorepo/polyrepo pipeline layout
• Frame DORA metrics and delivery dashboards; coordinate releases with SRE/platform
• Document rollback paths and release runbooks tied to pipeline artifacts

When NOT to Use

• Implement application features, APIs, or business logic → senior-software-engineer
• Operate Kubernetes control plane, node pools, or cluster add-ons only → cloud-engineer, cluster-deployment-engineer
• Own SLI/SLO programs, error budgets, and on-call reliability → site-reliability-engineer
• Author enterprise security policy, IdP, KMS, or SIEM programs → information-security-engineer
• Add SAST/SBOM/supply-chain controls as the primary task → devsecops
• Build internal developer portals, golden paths, or paved-road templates → platform-engineer
• Pre-flight architecture or go/no-go review without pipeline work → build-validator
• Broad GitOps, observability stack, and delivery-infra SRE as the main scope → devops

Related skills

Core Workflows

1. Pipeline discovery and baseline

1. Inventory repos, pipeline entry points, and deploy targets per environment 2. Map current stage order: checkout → build → test → publish → deploy 3. Identify manual steps, floating tags, and missing gates 4. Capture DORA baselines (deployment frequency, lead time, change fail %, MTTR) 5. Document owners: app team, platform, security, SRE

See `references/cicd_engineer_scope.md` for role boundaries and handoffs.

2. Pipeline design

1. Choose branching model aligned to release cadence (trunk, GitFlow, release branches) 2. Standardize reusable workflows/templates; pin tool and runner versions 3. Parallelize independent jobs; cache dependencies with lockfile keys 4. Emit immutable artifacts (digest-pinned images, versioned packages) 5. Fail fast on lint/unit; defer expensive suites with policy

See `references/pipeline_design_and_workflow.md` for patterns and anti-patterns.

3. Build, test, and deploy stages

1. Build: reproducible compilers/images; SBOM/provenance hooks where required 2. Test: unit → integration → contract/e2e; quarantine flaky tests with SLA 3. Publish: push to registry/artifact store; sign when policy requires 4. Deploy: environment-specific jobs; smoke tests after each promotion 5. Verify: synthetic checks or canary metrics before full traffic shift

See `references/build_test_deploy_stages.md` for stage contracts and artifacts.

4. Environments, gates, and promotion

1. One promotion path: dev → staging → prod (no skip without exception) 2. Use environment protection rules and required reviewers for production 3. Gate on test results, security scans, policy (OPA/conftest), and change tickets 4. Promote the same artifact digest across environments 5. Record promotion audit trail (who, what digest, when)

See `references/environments_gates_and_promotion.md` for gate catalog and promotion flows.

5. Security and compliance in CI

1. Inject secrets via OIDC/vault—not long-lived PATs in variables 2. Run SAST/SCA/secrets/IaC scans on PR and default branch 3. Block merge on critical findings per SLA; document exceptions 4. Export scan artifacts for devsecops / compliance-engineer evidence 5. Harden fork PR workflows (no secret access, label-gated runs)

See `references/security_and_compliance_in_ci.md` for credential and scan patterns.

6. Release coordination and reliability metrics

1. Align release windows with SRE capacity and error-budget policy 2. Define rollback: redeploy previous digest vs rebuild; test quarterly 3. Post-release: update change log, close tickets, capture DORA data point 4. Blameless review when change fail rate spikes; feed pipeline fixes 5. Coordinate with deployment-strategist for cutover; with site-reliability-engineer for canary SLO gates

See `references/reliability_metrics_and_release_coordination.md` for DORA and handoffs.

When to load references

Build, test, and deploy stages

Table of contents

1. Stage contracts 2. Build stage 3. Test pyramid in CI 4. Flaky test handling 5. Publish and artifact registry 6. Deploy stage 7. Deployment strategies in workflows 8. Smoke and verification 9. Artifacts between jobs

Stage contracts

Each stage should declare:

Document stage contracts in pipeline README or inline comments for template consumers.

Build stage

Goals: reproducible compile or image from locked dependencies.

Checklist:

• [ ] Pin base images and tool versions (compiler, Node, Go, JDK)
• [ ] Use lockfiles (package-lock.json, go.sum, poetry.lock)
• [ ] Multi-stage Docker builds; minimal runtime image
• [ ] Build args only for non-secret metadata (version, git SHA)
• [ ] Label images: org.opencontainers.image.revision, version
• [ ] Fail on warnings only when policy requires (avoid noisy flakiness)

Outputs: OCI image digest, binary tarball, or package version string—never a floating tag alone.

Test pyramid in CI

Test data: ephemeral databases, containers, or mocks—no shared mutable state between parallel shards.

Coverage: publish coverage artifacts; do not use arbitrary % as sole merge gate without team agreement.

Flaky test handling

1. Detect — track flaky rate per test (retry plugins, test analytics). 2. Quarantine — mark flaky tests skipped in default PR path with ticket link. 3. SLA — fix or delete quarantined tests within agreed days. 4. Retry policy — limit automatic retries to known infra flakes, not assertions. 5. Report — weekly flaky leaderboard to engineering managers.

Never silently retry failed assertions in production deploy paths without visibility.

Publish and artifact registry

After build + required tests:

1. Push image to registry with immutable digest 2. Upload SBOM/provenance when devsecops policy requires 3. Store Helm chart, Terraform bundle, or package to artifact store 4. Retention policy: PR artifacts short TTL; release artifacts per compliance

Signing: cosign/notary or platform equivalent before prod promotion gate.

Deploy stage

Deploy jobs should:

• Download exact artifact from publish stage (digest verification)
• Target one environment per job (dev/staging/prod separation)
• Use infrastructure credentials scoped to that environment
• Emit deployment record (version, digest, timestamp, actor)
• Run smoke tests immediately after deploy

Avoid rebuilding application in deploy job unless deployable is infrastructure-only (e.g., terraform apply with same modules).

Deployment strategies in workflows

Implement at workflow/orchestration layer; pair with runtime config from platform/SRE.

Metric gates for canary belong in coordination with site-reliability-engineer (error rate, latency SLO).

Smoke and verification

Minimum post-deploy checks:

• Health endpoint returns 200
• Critical read-only API path
• Database migration status (if applicable)
• Queue consumer heartbeat (if applicable)

Run from CI job or delegated synthetic monitor; failures trigger automatic rollback when policy allows.

Artifacts between jobs

Rule: production promotion references registry digest or versioned package ID, not workflow artifact zip from a different commit.

CI/CD engineer scope

Table of contents

1. Role definition 2. In scope 3. Out of scope 4. Stakeholders and handoffs 5. Deliverables 6. Success criteria

Role definition

The CI/CD engineer owns how software moves from commit to production through automated pipelines: stage design, workflow maintenance, environment promotion, artifact integrity, deployment automation at the pipeline layer, and delivery metrics. The role partners with application teams, platform engineering, security, and SRE—it does not replace them.

Primary systems: GitHub Actions, GitLab CI, Jenkins, CircleCI, Buildkite, Azure Pipelines, and adjacent registries, artifact stores, and deploy hooks (Helm, Argo CD triggers, serverless deploy plugins).

In scope

Out of scope

Stakeholders and handoffs

Escalate production incidents to on-call/SRE; CI/CD engineer supports rollback execution and pipeline fixes, not incident command unless explicitly dual-hatted.

Deliverables

• Pipeline-as-code in application or shared template repositories
• Documented stage diagram and promotion flow per service tier
• Gate catalog: what blocks merge vs production deploy
• Runbook: hotfix, rollback, and manual promotion exception process
• Dashboard or export for DORA and pipeline reliability metrics
• Onboarding guide for new services (required checks, secrets, environments)

Success criteria

• Reproducible: same commit produces same artifact digest
• Fast feedback: median PR pipeline within team target; critical path identified
• Safe promotion: production receives only approved digests; no env drift rebuilds
• Observable: failed stages categorized (infra, test, security, deploy)
• Recoverable: rollback tested; last-known-good digest documented
• Measurable: DORA and change-fail trends visible; regressions investigated

Environments, gates, and promotion

Table of contents

1. Environment model 2. Promotion flow 3. Gate catalog 4. Approvals and change control 5. Configuration per environment 6. Preview and ephemeral environments 7. Exceptions and break-glass 8. Audit trail

Environment model

Standard tiers (adjust names to org):

Rules:

• One account/project per prod where cloud isolation policy requires
• No direct commits to prod config branches without pipeline
• Secrets scoped per environment in secret manager

Promotion flow

build (once) → artifact v1@sha256:abc…
  → deploy dev (auto)
  → deploy staging (auto or RC tag)
  → gates pass
  → deploy production (approved)

Same digest crosses all environments. Config differences come from environment variables, Helm values, or parameter files—not rebuilds.

Document promotion in runbook:

1. Verify staging smoke green 2. Confirm change ticket / release notes 3. Approve production environment in CI 4. Monitor canary or post-deploy synthetics 5. Close release; tag repository

Gate catalog

Order gates to minimize wasted work: cheap checks before image build when possible.

Approvals and change control

• Use platform environment protection (GitHub Environments, GitLab protected environments, Jenkins input steps)
• Require two approvers for tier-1 services when policy demands
• Approvers must not be sole author of the change for regulated environments
• Link deployment to ticket: Jira/ServiceNow ID in workflow input or commit metadata

Freeze windows: pipeline reads calendar or flag file; production job skips or warns during freeze unless break-glass label.

Configuration per environment

Never store prod secrets in staging vault paths accessible to broader groups.

Preview and ephemeral environments

PR previews:

• Deploy from PR head commit; unique URL per PR
• Tear down on PR close or TTL (24–72h)
• No production data; no production credentials
• Label-gated workflows for fork PRs

Cost control: limit concurrent previews per repo; shared preview cluster with quotas.

Exceptions and break-glass

Document when manual promotion is allowed:

1. Pipeline platform outage with documented workaround 2. Critical security patch with abbreviated suite (pre-approved) 3. Rollback to previous digest (preferred over forward fix in incident)

Each exception requires:

• Ticket with approver
• Post-incident pipeline fix task
• Retrospective if customer-impacting

Audit trail

Retain for compliance and debugging:

Export to SIEM or evidence bucket when compliance-engineer requires continuous control monitoring.

Pipeline design and workflow

Table of contents

1. Design principles 2. Standard stage graph 3. Branching models 4. Reusable workflows and templates 5. Monorepo vs polyrepo 6. Performance and cost 7. Anti-patterns 8. Platform comparison notes

Design principles

1. Pipeline as code — versioned with the app or a pinned template repo; no click-ops-only definitions for production paths. 2. Immutable artifacts — build once, promote many; never rebuild for production without explicit exception. 3. Fail fast — lint and unit tests before integration, container builds, and deploy jobs. 4. Least privilege — scoped tokens per job; no shared mega-credentials across org. 5. Auditable — who approved production, which digest deployed, which tests passed. 6. Idempotent deploy — redeploying the same digest is safe; pipelines tolerate retries.

Standard stage graph

checkout
  → static analysis / lint
  → unit tests (parallel shards)
  → build (compile / image)
  → integration & contract tests
  → security scans (parallel where possible)
  → publish artifact (registry / package feed)
  → deploy non-prod
  → e2e / smoke (non-prod)
  → manual or policy gate
  → deploy staging / canary
  → synthetic / metric gate
  → deploy production
  → post-deploy smoke + notify

Optional branches: preview environments per PR, scheduled nightly full suites, performance benchmarks on main only.

Branching models

Hotfix lane: tag or branch from production commit; run abbreviated test suite; promote hotfix digest through same prod gate as normal releases.

Reusable workflows and templates

• Central callable workflows (workflow_call) or shared Jenkins libraries with semver tags.
• Pin template version in consumer repos; changelog for breaking changes.
• Parameters: language, service tier, regions, feature flags for optional stages.
• Document escape hatches when teams must override (requires platform review).

Contract per template:

Monorepo vs polyrepo

Monorepo

• Path filters — run jobs only when relevant directories change.
• Affected detection — Nx/Turborepo/Bazel/graph tools to map dependency closure.
• Shared versioning — coordinate breaking changes across packages in one PR.
• Risk — long main-branch pipelines; mitigate with sharding and remote cache.

Polyrepo

• Standard template — every service repo inherits same base pipeline.
• Cross-repo releases — orchestrate with meta-pipeline or release train manifest.
• Dependency updates — Renovate/bot PRs with required checks before merge.

Performance and cost

Track queue time vs run time separately—queue indicates runner capacity issues.

Anti-patterns

Platform comparison notes

Choose platform capabilities deliberately; migrating later is expensive—abstract only where multi-platform is a real requirement.

Reliability metrics and release coordination

Table of contents

1. DORA metrics 2. Pipeline health metrics 3. Release coordination 4. Rollback 5. Incident and freeze interaction 6. Partnering with SRE and platform 7. Continuous improvement

DORA metrics

Four key metrics (define measurement consistently org-wide):

Implementation tips:

• Use deployment events API (GitHub deployments, GitLab environments, custom webhook)
• Normalize timezone and “what counts as production”
• Exclude docs-only deploys if policy says so—document exclusion
• Review quarterly with leadership; avoid gaming metrics

DORA informs priorities; SLOs remain owned by site-reliability-engineer.

Pipeline health metrics

Dashboard per repo and aggregate platform view for template teams.

Release coordination

Before release:

1. Confirm staging validation and changelog 2. Check error budget / freeze calendar with SRE 3. Verify migrations backward-compatible or expand-contract plan 4. Notify customer-facing teams per comms policy 5. Ensure on-call coverage for deployment window

During release:

• Use progressive exposure (canary) when tier requires
• Monitor golden signals during deploy job + 30–60 min after
• Single release captain coordinates pipeline and comms

After release:

• Mark deployment complete in tracker
• Close feature flags cleanup tickets
• Capture DORA data point
• Schedule retro if change fail

Coordinate cutover details with deployment-strategist when traffic switching is complex.

Rollback

Preferred: redeploy previous known-good digest without rebuild.

Runbook steps:

1. Identify last good digest from deploy history or container registry 2. Trigger rollback workflow (same pipeline, pinned digest input) 3. Run smoke tests 4. Verify metrics return to baseline 5. File incident if customer impact occurred 6. Block forward deploy until root cause fixed

Test rollback quarterly for tier-1 services.

Incident and freeze interaction

When SRE declares incident or freeze:

• Halt optional releases; allow security hotfix path only
• Pipeline reads FREEZE=true variable or environment lock
• Post-incident: prioritize pipeline fixes that caused or prolonged outage

CI/CD engineer attends blameless postmortem when delivery path contributed; action items tracked in same sprint.

Partnering with SRE and platform

Escalate production mitigation execution to on-call; provide rollback pipeline and artifact coordinates quickly.

Continuous improvement

Monthly delivery review agenda:

1. DORA trend vs targets 2. Top pipeline failures by category 3. Flaky test debt 4. Security gate exceptions expiring soon 5. Template version drift across repos

Quarterly: game day deploying and rolling back in staging; update runbooks from findings.

Security and compliance in CI

Table of contents

1. Shared responsibility 2. Credentials and secrets 3. OIDC federation 4. Runner and supply-chain hardening 5. Security scanning gates 6. Fork and untrusted code 7. Compliance evidence hooks 8. Agentic and AI workflows in CI

Shared responsibility

CI/CD implements hooks; security teams define policy.

Credentials and secrets

Do:

• Store secrets in vault/GitHub encrypted secrets/GitLab masked variables with environment scope
• Rotate credentials on schedule; break glass documented
• Use separate credentials per environment (dev ≠ prod)
• Mask secrets in logs; prevent set -x leaking values

Do not:

• Commit secrets or long-lived cloud keys to git
• Share one prod deploy key across all repos
• Pass secrets to fork workflows without controls

Rotation runbook: update secret → verify pipeline → revoke old → audit last use.

OIDC federation

Prefer OIDC over static cloud keys in CI:

1. IdP trusts CI platform (GitHub/GitLab issuer URL) 2. Role trust policy scoped to repo:org/name:ref:refs/heads/main or environment 3. Short-lived tokens per job 4. Least privilege IAM role per repo or environment

Validate: job cannot assume prod role from unprotected branch.

Runner and supply-chain hardening

Container builds: use digest-pinned base images; scan images before prod gate.

Security scanning gates

Typical stages (configure with devsecops):

Triage workflow: finding ID → owner → SLA → exception with expiry → re-scan.

Export SARIF or vendor JSON to artifact store for dashboards and audits.

Fork and untrusted code

• pull_request_target only when absolutely necessary; prefer pull_request with limited permissions
• Require maintainer approval before running workflows on external forks
• No access to org secrets on fork PRs unless label-gated and reviewed
• Sanitize branch names in shell scripts (injection risk)

Compliance evidence hooks

Pipeline outputs for auditors (coordinate with compliance-engineer):

Schedule monthly export of failed gates and mean time to remediate for leadership review.

Agentic and AI workflows in CI

When pipelines invoke AI coding agents (Copilot Actions, Claude Code, Codex):

• Treat PR comments and workflow inputs as untrusted
• Sandbox with minimal permissions; no prod secrets
• Pin action versions; review workflow diffs like application code
• Use agentic-actions-auditor patterns when available for injection review

CI/CD engineer ensures workflow structure is safe; devsecops defines security bar for tools.