Ops Nuke Cicd

NUKE CI/CD

Quick Reference

• Start from existing nuke/Build.cs targets and preserve output contracts before refactoring.
• Keep target graph intent explicit: use DependsOn for hard prerequisites, After for ordering, Triggers for composed flows, and OnlyWhenDynamic for runtime gates.
• Use this skill for pipeline orchestration and build contracts, not for application-service refactors or NUnit fixture internals.
• Separate fast local feedback (build + filtered tests) from full CI validation (unit + api + db + merged coverage).
• Use the iterative loop code -> build -> run tests -> fix -> repeat for both feature work and test hardening.
• Run preflight checks before expensive targets: SDK version, Docker availability (when required), and expected file paths.
• Use wrapper commands (when available): ./build.sh BuildAll, ./build.sh LocalUnitTest, ./build.sh ApiTest, ./build.sh TestAll.
• Use category filters intentionally, including exclusion filter patterns such as TestCategory!=ComponentTests&TestCategory!=DbTests&TestCategory!=ApiTest.
• Keep UnitTest scoped to non-API categories and create a dedicated ApiTest target for TestCategory=ApiTest.
• Keep TestAll composed from UnitTest + ApiTest (+ DbTest) and coverage merge.
• When migrating from legacy docker-compose/SpecFlow orchestration, decommission compose-first test flow and keep pipeline focused on NUnit API categories.
• Avoid running parallel dotnet test invocations against the same project output path in one job to prevent file-lock/MSBuild manifest failures.
• Keep shell commands robust for zsh: avoid unquoted globs in direct shell commands and validate paths before sed/cat/ls.
• Emit coverage and test artifacts deterministically (coverage.cobertura.xml, HTML summary, JUnit XML).
• Build one or more Docker images with traceable tags (build id, commit sha), then publish digest-pinned deploy variables into deploy.env (image count is repository-specific).
• Branch behavior on IsLocalBuild only for performance and environment-output concerns, not correctness.
• If the task shifts into service implementation details, switch to $software-csharp-backend.
• If the task shifts into fixture design, WireMock/Testcontainers setup, or anti-flake test structure, switch to $qa-testing-nunit.

Workflow

1. Model or review target graph sequencing and execution constraints. Load references/nuke-target-graph-design.md. 2. Design the build-test loop for early failures and rapid signal. Load references/build-test-feedback-loop.md. 3. Define and verify test category filters for unit/API/DB/component separation. Load references/test-categories-and-filters.md. 4. Implement coverage and test reporting with merge/publish outputs. Load references/coverage-and-reporting.md. 5. Implement Docker build/push with tag + digest capture and deployment outputs. Load references/docker-build-push-patterns.md. 6. Enforce stable artifact contracts and environment outputs. Load references/artifacts-and-output-contracts.md. 7. Tune local vs CI behavior without hiding pipeline defects. Load references/local-vs-ci-behavior.md. 8. Harden reliability, logs, and diagnostics for CI incident response. Load references/pipeline-reliability-and-observability.md. 9. Run command hygiene and environment preflight checks before final run. Load references/execution-preflight-and-command-hygiene.md. 10. Run anti-pattern review before finalizing. Load references/nuke-pipeline-antipatterns.md.

Decision Tree

• If target ordering is incorrect or unexpected targets run, use references/nuke-target-graph-design.md.
• If feedback loop is too slow or flaky, use references/build-test-feedback-loop.md.
• If test scope is wrong in CI or local runs, use references/test-categories-and-filters.md.
• If coverage or JUnit artifacts are missing/partial, use references/coverage-and-reporting.md.
• If Docker outputs are not traceable or digest pinning is missing, use references/docker-build-push-patterns.md.
• If downstream jobs cannot consume artifacts or env outputs, use references/artifacts-and-output-contracts.md.
• If local and CI behavior diverge unexpectedly, use references/local-vs-ci-behavior.md.
• If failures are hard to debug from logs, use references/pipeline-reliability-and-observability.md.
• If failures come from shell quoting, glob expansion, missing files, or environment prerequisites, use references/execution-preflight-and-command-hygiene.md.
• If pipeline quality regresses during refactors, use references/nuke-pipeline-antipatterns.md.

Do / Avoid

Do

• Keep targets deterministic and side effects explicit.
• Keep test stages isolated by category and risk profile.
• Keep coverage/report merge as a first-class target in the graph.
• Keep Docker outputs traceable through tag, digest, and exported env variables.
• Keep artifact paths and names stable across branches and CI systems.

Avoid

• Mixing orchestration and hidden side effects inside unrelated targets.
• Running expensive integration tests before compile/unit gates.
• Changing output file names without updating consumer jobs.
• Using local-only shortcuts that invalidate CI parity.
• Ignoring digest capture and shipping mutable image references.

Resources

• NUKE Target Graph Design
• Build-Test Feedback Loop
• Test Categories and Filters
• Coverage and Reporting
• Docker Build Push Patterns
• Artifacts and Output Contracts
• Local vs CI Behavior
• Pipeline Reliability and Observability
• Execution Preflight and Command Hygiene
• NUKE Pipeline Antipatterns
• Skill Data: curated Microsoft and .NET CI/CD references for this skill.

Templates

• NUKE Target Template Build and Test
• NUKE Target Template Docker Build Push Digest
• Test Result and Coverage Publishing Checklist
• CI Troubleshooting Checklist
• PR Pipeline Quality Checklist

CI Troubleshooting Checklist

• Identify first failed NUKE target and stop at that boundary.
• Confirm prerequisite targets executed (DependsOn) and in correct order (After).
• Confirm runtime gates (OnlyWhenDynamic) did not skip required targets.
• Inspect test filters for category drift or typos (ApiTest, DbTests, ComponentTests).
• Verify Docker prerequisites (daemon availability, auth, registry connectivity).
• Verify digest extraction from push output is still valid.
• Verify deploy.env path differs correctly for local vs CI.
• Verify artifact directories are persisted before cleanup targets run.
• Re-run failing target with elevated verbosity to isolate root cause.

PR Pipeline Quality Checklist

• Target graph uses DependsOn/After/Triggers intentionally and minimally.
• Local fast path remains fast and still catches compile + core unit regressions.
• CI composed path includes mandatory suites and coverage merge target.
• Category filters are explicit and reviewed for accidental exclusions.
• Coverage outputs remain in Cobertura format and are merged consistently.
• JUnit files are generated with stable naming patterns.
• Docker images are tagged with traceable metadata and exported with digest references.
• deploy.env contract keys are stable and documented.
• Logging is sufficient to diagnose failing targets without rerunning entire pipeline.
• Cleanup targets do not remove artifacts needed by downstream steps.

Test Result and Coverage Publishing Checklist

• Verify each test target writes JUnit XML to artifacts/.
• Verify collector is set to XPlat Code Coverage;Format=cobertura.
• Verify raw coverage files exist under artifacts/coverage-report/**/coverage.cobertura.xml.
• Verify merge target runs after all required test targets.
• Verify ReportGenerator emits Cobertura + HTML summary into artifacts/coverage-report.
• Verify CI artifact collector paths match real output locations.
• Verify CI test-report parser includes all *-test-result.xml files.
• Verify failed tests still emit logs and partial results when possible.

Artifacts and Output Contracts

Purpose

Define stable outputs that downstream CI/CD jobs can consume without repo-specific assumptions.

Contract Principles

• Keep artifact locations deterministic and version-control-friendly.
• Keep filenames stable across local and CI contexts.
• Keep environment variable names backward-compatible.
• Treat output contracts as public interfaces of the pipeline.

Core Outputs

• artifacts/coverage-report/**/coverage.cobertura.xml
• artifacts/coverage-report/index.html (or equivalent HTML summary)
• artifacts/*-unit-test-result.xml
• artifacts/*-api-test-result.xml
• artifacts/*-db-test-result.xml
• deploy.env (path depends on local vs CI mode)

deploy.env Behavior

• Local run: commonly write to repo root ./deploy.env.
• CI run: commonly write to ${CiProjectDirectory}/deploy.env.
• Include only variables with known values; avoid placeholder keys.

Producer/Consumer Mapping

• Build targets produce binaries and container images.
• Test targets produce JUnit and raw coverage files.
• Merge target produces merged Cobertura + HTML summary.
• Output target writes deployment env contract for deploy stage.

Change Management Rules

• If output names/locations change, update all CI collectors and deploy consumers in the same change.
• Keep one source of truth for artifact directory variables.
• Add guard checks when contract files are mandatory for downstream targets.

Build-Test Feedback Loop

Goal

Return useful failure signal quickly in local runs while preserving full confidence in CI.

Iterative Quality Loop

Use this cycle continuously: 1. Edit code and tests. 2. Build (BuildAll or equivalent target). 3. Run relevant tests (LocalUnitTest, ApiTest, DbTest, TestAll). 4. Fix failures and repeat.

Loop Design

• Keep compile and lightweight tests first.
• Keep integration-like tests in dedicated targets (ApiTest, DbTest, ComponentTests) after unit success.
• Keep one composed target (TestAll) for CI to avoid missing mandatory stages.

Fast-Feedback Baseline

1. Restore 2. BuildAll 3. UnitTest with filtered categories 4. Optional local-only shortcuts for developer iteration

CI-Confidence Baseline

1. BuildAll 2. UnitTest 3. ApiTest 4. DbTest 5. Coverage/report merge target

Recommended Sequencing

Target LocalUnitTest => _ => _
    .Triggers(BuildAll, UnitTest);

Target TestAll => _ => _
    .Triggers(BuildAll, UnitTest, ApiTest, DbTest, MergeCodeCoverageReports);

Common Command Examples

Use repo wrappers when present:

./build.sh BuildAll
./build.sh LocalUnitTest
./build.sh ApiTest
./build.sh TestAll

Direct NUKE invocation:

nuke BuildAll
nuke LocalUnitTest
nuke ApiTest
nuke TestAll

Performance Controls

• Use EnableNoRestore() in test targets when restore was already completed.
• Use EnableNoBuild() for tests when binaries are already produced by earlier targets.
• Use minimal verbosity by default and elevate only for diagnostics.
• Run integration tests only where required by scope.
• For one project, do not run multiple dotnet test commands in parallel in the same job.

Pipeline Wiring as Feature Completion

• New test suites (especially Docker-backed component tests) must be wired into the canonical pipeline path as part of feature completion, not as follow-up cleanup.
• Docker-backed suites need an explicit validation lane and must not accidentally create circular NUKE target graphs.
• Treat pipeline wiring failures as blocking — an unwired suite gives false confidence that the feature is tested.

Failure Diagnostics

• Slow local loop: confirm local target is not triggering API/DB/component suites.
• Repeated restore/build overhead: confirm NoRestore and NoBuild settings.
• CI timeout risk: split large suites by category and review container startup readiness.
• Intermittent MSBuild lock failures: sequence test invocations per project and avoid output-path contention.
• New test suite not running in CI: verify the suite's target is reachable from the canonical pipeline entry point without circular dependencies.

Coverage and Reporting

Purpose

Produce machine-consumable and human-readable outputs from all test stages with stable paths and names.

Required Outputs

• Per-run coverage collector output in Cobertura format.
• Merged coverage artifact for CI quality gates.
• HTML summary for quick inspection.
• JUnit XML files per suite for CI test reports.

DotNet Test Output Pattern

DotNetTasks.DotNetTest(s => s
    .SetDataCollector("XPlat Code Coverage;Format=cobertura")
    .SetResultsDirectory($"{ArtifactsDirectory}/coverage-report")
    .AddLoggers($"junit;LogFilePath={ArtifactsDirectory}/{{assembly}}-unit-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"));

Coverage Merge Pattern

Use ReportGenerator to merge all coverage.cobertura.xml files:

ReportGenerator(s => s
    .SetReports($"{ArtifactsDirectory}/coverage-report/**/coverage.cobertura.xml")
    .SetTargetDirectory($"{ArtifactsDirectory}/coverage-report")
    .SetAssemblyFilters("-*.Tests", "-*.Tests.*")
    .SetReportTypes(ReportTypes.Cobertura, ReportTypes.HtmlSummary));

Publish Checklist

• Verify Cobertura XML exists after each relevant test target.
• Verify merged Cobertura.xml and HTML summary exist after merge target.
• Verify JUnit files are emitted with stable filename patterns.
• Verify CI collects all paths using wildcard-safe patterns.

Failure Diagnostics

• Empty merged report: check SetReports glob and suite execution.
• Missing JUnit files: check logger string and results path permissions.
• Unexpected coverage drop: confirm all intended targets feed merge stage.

Docker Build Push Patterns

Purpose

Build and publish container images with traceability and immutable deployment references.

Image Count

• Do not hardcode image count assumptions in shared guidance.
• Some repositories publish one image, others multiple images.
• Pricing is only one example and currently publishes two image outputs (private API and migrator).

Build Pattern

• Tag images with CI build identifier for traceability.
• Pass COMMIT_HASH, BUILD_DATE, and BUILD_ID as build args.
• Use SetPull(true) when base-image freshness matters.

Push + Digest Capture Pattern

1. Build image with mutable tag (registry/image:{BuildId}). 2. Push tagged image. 3. Parse push output for digest (sha256:...). 4. Emit immutable deploy reference (registry/image@sha256:...).

Example

var imageTag = $"{DockerRegistry}/{DockerImagePrefix}/privateapi:{BuildId}";
DockerTasks.DockerBuild(s => s.SetTag(imageTag));
var outputs = DockerTasks.DockerImagePush(s => s.SetName(imageTag));
var digest = ReadDigits(outputs); // returns "sha256:..."
var deployRef = $"{DockerRegistry}/{DockerImagePrefix}/privateapi@{digest}";

deploy.env Contract

Write exported deploy variables for downstream jobs (one variable per produced image):

DOCKER_IMAGE_DEPLOY_SERVICE_A=<registry>/<repo>/service-a@sha256:...
DOCKER_IMAGE_DEPLOY_SERVICE_B=<registry>/<repo>/service-b@sha256:...

Reliability Checks

• Verify login target runs before push.
• Verify digest parsing handles push output format changes.
• Verify env file is written even when only one image is produced.
• Verify cleanup targets (docker rmi) run only after all required outputs are captured.

Failure Diagnostics

• Missing digest value: inspect push logs and parsing utility.
• Wrong tag source: verify BuildId and repository metadata at runtime.
• Deployment drift: use digest reference, not mutable tag, in downstream deploy steps.

Execution Preflight and Command Hygiene

Purpose

Reduce avoidable CI/local failures caused by missing prerequisites, bad paths, shell quoting, and glob expansion.

Preflight Checklist

• Confirm repository root and expected working directory before running path-sensitive commands.
• Confirm required SDK/runtime versions before build or test runs.
• Confirm Docker availability before running API/DB/component suites that need containers.
• Confirm target files/directories exist before running sed, cat, or ls against hardcoded paths.

Shell Safety Rules

• Prefer rg --files and explicit file lists over broad shell globs.
• Quote command arguments that include special characters or whitespace.
• Avoid patterns that depend on shell-specific glob behavior.
• For complex commands, test a narrow path first before expanding scope.

Test Scope Guardrails

• If user constraints exclude infra-dependent suites, do not run those suites implicitly.
• Run feasible targets first (BuildAll, LocalUnitTest, scoped API/DB tests when available).
• Report skipped targets with clear reason and exact follow-up command.

Frequent Failure Patterns and Fixes

• no such file or directory: verify path from repo root and discover files with rg --files.
• no matches found (zsh glob): replace raw glob with rg --files <dir> | rg <pattern>.
• shell parse/syntax errors: simplify quoting and split compound commands.
• build log file locked: avoid concurrent NUKE runs that write to the same temp log file.

Verification

• Run one narrow command successfully before batch command execution.
• Re-run failing target in isolation after fixing preflight issues.
• Keep final report explicit about prerequisites and skipped validations.

Local vs CI Behavior

Purpose

Tune execution for developer speed without sacrificing CI correctness.

Branching Rules

• Use IsLocalBuild for performance choices and output paths.
• Keep test semantics equivalent unless scope intentionally differs.
• Keep pipeline correctness independent from local shortcuts.

Typical Controls

• SetNoBuild(IsLocalBuild) in test targets when binaries are already built locally.
• EnableNoRestore() when restore was already executed in graph.
• Local output path for convenience; CI output path for collector compatibility.

Performance Trade-offs

• NoBuild and NoRestore reduce local latency but can hide dependency issues if graph is wrong.
• Reusing prior local outputs speeds iteration but increases stale-artifact risk.
• CI should remain explicit and mostly self-sufficient.

Recommended Pattern

var outputFilePath = IsLocalBuild
    ? Path.Combine(".", "deploy.env")
    : Path.Combine(CiProjectDirectory, "deploy.env");

Safeguards

• Ensure local targets still compile/test enough to catch obvious regressions.
• Ensure CI path always executes full quality gates before deploy outputs.
• Keep parity checks: run full TestAll locally before major CI pipeline changes.

CI Docker Host Divergence

• When a CI runner uses a remote Docker daemon, localhost on the runner is not the Docker host. Tests that hard-code 127.0.0.1 or localhost as the broker/service endpoint will fail silently or connect to nothing.
• Test harnesses must resolve endpoints from DOCKER_HOST or Testcontainers host detection, not hard-coded loopback.
• Code-only fixes can make suites CI-compatible, but the final proof requires a real pipeline run on the target CI infrastructure.

Failure Diagnostics

• Works locally, fails in CI: compare IsLocalBuild condition branches.
• CI-only missing files: inspect CI output path and artifact collector configuration.
• Local stale results: clear artifacts or force build/restore during troubleshooting.
• Tests pass locally but fail in CI with connection errors: check whether Docker containers are running on a remote host and endpoints assume localhost.

NUKE Pipeline Antipatterns

Purpose

Catch common design mistakes before they become recurring CI incidents.

Antipatterns and Corrections

• Antipattern: Use After where hard prerequisite is required.

Correction: Replace with DependsOn for mandatory dependencies.

• Antipattern: Trigger all tests in local fast-feedback targets.

Correction: Keep local fast path focused; run full matrix in TestAll.

• Antipattern: Keep filter expressions duplicated and inconsistent across targets.

Correction: Centralize category intent and verify by target.

• Antipattern: Publish mutable image tags to deployment jobs.

Correction: Export digest-pinned references and consume those.

• Antipattern: Couple cleanup (docker rmi) with critical output generation.

Correction: Generate and persist outputs before cleanup.

• Antipattern: Mix restore/build/test logic in one large target.

Correction: Split into composable targets with explicit graph edges.

• Antipattern: Rely on local-only behavior to pass failing tests.

Correction: Keep CI path as source of truth and re-validate locally with CI-like targets.

• Antipattern: Change artifact names without downstream updates.

Correction: Treat artifacts as versioned contracts.

Pre-Merge Review

• Verify graph edges map to intended execution model.
• Verify category filters include/exclude intended suites.
• Verify coverage merge target runs after all required tests.
• Verify deploy outputs include digest-pinned image references.
• Verify local-vs-CI conditionals are minimal and intentional.

NUKE Target Graph Design

Purpose

Design a readable and deterministic target graph that works for both local developer commands and CI pipelines.

Target Relationship Rules

• Use DependsOn for hard prerequisites that must execute before a target.
• Use After when both targets may run, but you need execution order.
• Use Triggers for high-level orchestration targets that compose lower-level targets.
• Use OnlyWhenDynamic(...) to gate expensive stages using runtime conditions.

Practical Graph Pattern

1. Restore/build foundations. 2. Fast checks (unit tests) early. 3. Slower checks (API/DB/component/integration) after fast gates. 4. Aggregation targets (coverage/report merge) after all required tests. 5. Packaging/publishing stages after quality gates.

Migration Graph Pattern (Legacy Compose to NUnit API)

1. Keep BuildAll as compile gate. 2. Keep UnitTest excluding ApiTest/DbTests/ComponentTests. 3. Keep ApiTest as dedicated category run over API test projects. 4. Keep DbTest separate if present. 5. Keep TestAll as orchestration trigger only. 6. Remove compose orchestration targets and environment plumbing once decommissioned.

Example Structure

Target UnitTest => _ => _
    .DependsOn(BuildAll)
    .OnlyWhenDynamic(IsBuildRequired)
    .Executes(() => { /* dotnet test filter excludes ApiTest */ });

Target ApiTest => _ => _
    .After(UnitTest)
    .OnlyWhenDynamic(IsBuildRequired)
    .Executes(() => { /* dotnet test filter=ApiTest */ });

Target TestAll => _ => _
    .Triggers(BuildAll, UnitTest, ApiTest, DbTest, MergeCodeCoverageReports);

Design Checks

• Verify each target has a single clear responsibility.
• Verify high-level orchestration targets avoid direct implementation logic.
• Verify graph ordering prevents expensive work before fast failures are known.
• Verify graph names communicate intent (UnitTest, ApiTest, TestAll, BuildAndPushImagesAll).

Failure Diagnostics

• If expected prerequisites do not run, check DependsOn.
• If order is wrong despite execution, check After.
• If targets are unexpectedly skipped, inspect OnlyWhenDynamic conditions.
• If orchestration misses steps, inspect Triggers definitions.

Pipeline Reliability and Observability

Purpose

Make failures obvious, reproducible, and fast to diagnose.

Reliability Practices

• Fail fast on compile and unit-test errors before expensive stages.
• Keep target side effects deterministic and idempotent where possible.
• Keep container prerequisites explicit for API/DB test stages.
• Keep cleanup steps separated from required deploy-output generation.
• Keep project-level dotnet test invocations sequential when they target the same build output location.

Logging Practices

• Log key runtime identifiers: build id, commit, image tag, and digest.
• Log start/end of expensive stages with clear target names.
• Use minimal default verbosity and targeted debug verbosity during incidents.
• Print Docker/environment diagnostics before integration-style tests.

Observability Signals

• Stage duration and queue time trends.
• Pass/fail rates by test category.
• Coverage merge success/failure and report size deltas.
• Docker push digest extraction reliability.

Incident Playbook

1. Locate first failing target. 2. Verify prerequisite target completion and gate conditions. 3. Inspect generated artifacts in expected locations. 4. Re-run failing target in isolation with elevated verbosity. 5. Patch root cause, then run composed target path to verify graph integrity.

Useful Guardrails

• Enforce stable artifact names in CI config.
• Add explicit checks for required files before publish/deploy steps.
• Keep category filters under source control, not inline in CI YAML only.
• For repeated file-lock incidents, run tests with NoBuild after one upfront build and avoid concurrent test commands against same project output.

Test Categories and Filters

Purpose

Control test scope with predictable category filters so local and CI runs execute the intended suites.

Category Strategy

• Use category names as contract: ApiTest, DbTests, ComponentTests, plus default unit tests.
• Keep category assignment explicit in NUnit attributes.
• Keep NUKE filter expressions centralized per target.
• Keep API migration suites under ApiTest and exclude them from UnitTest.

Filter Patterns

• Unit-only exclusion pattern:

TestCategory!=ComponentTests&TestCategory!=DbTests&TestCategory!=ApiTest

• API-only pattern:

TestCategory=ApiTest

• DB-only pattern:

TestCategory=DbTests

Target Mapping Example

Target UnitTest => _ => _
    .Executes(() => DotNetTasks.DotNetTest(s => s
        .SetFilter("TestCategory!=ComponentTests&TestCategory!=DbTests&TestCategory!=ApiTest")));

Target ApiTest => _ => _
    .Executes(() => DotNetTasks.DotNetTest(s => s
        .SetFilter("TestCategory=ApiTest")));

Target DbTest => _ => _
    .Executes(() => DotNetTasks.DotNetTest(s => s
        .SetFilter("TestCategory=DbTests")));

Migration Refactor Rules

• If migrating from compose/SpecFlow flow, remove legacy orchestration targets from test graph.
• Keep one dedicated ApiTest target over API test projects.
• Keep TestAll composed from BuildAll + UnitTest + ApiTest (+ DbTest) + coverage merge.

Validation Checks

• Verify each category has at least one test.
• Verify unit target excludes integration categories.
• Verify CI composed target includes all mandatory categories.
• Verify filter strings are identical between local and CI paths when scope should match.

Common Failure Modes

• Category typo causes tests to silently skip.
• Broad exclusion filters hide suites that should run.
• Project-level test target mismatch (wrong .csproj) drops expected tests.
• Running parallel dotnet test on the same project output causes intermittent file-lock/MSBuild failures.