Wio

WIO

WIO is one testing workflow skill with five command modes:

• scan: find the highest-value test candidates for a codebase, change, or scope.
• test: write one focused high-value test for a selected behavior, code path, or regression risk.
• workload: generate a realistic, adversarial workload that adds a new failure surface, oracle, sequence, or coverage dimension with controlled variance, replay, and correctness invariants.
• review: review a newly written or existing test for customer value, developer value, signal quality, and maintainability.
• doctor: diagnose test-suite health problems in a codebase or scope.

Commands are accessed through $wio:

Use references/index.md to route from code evidence and candidate failure modes to the right strategy references.

Reference Loading

Do not pick a test strategy from memory or from the nearest existing test alone. First inspect the target code, public behavior, existing tests, fixtures, and test commands. Then identify candidate behaviors or workloads and their likely failure mechanisms. Only after candidates exist, load the references needed to choose the strategy.

For every selected candidate, load at least one strategy reference that matches the failure mechanism before recommending or writing a test:

• Load Risk-Based Testing when priorities, customer impact, security/business risk, or limited test capacity decide what comes first.
• Load User Behavior Testing when the behavior is a user journey, product workflow, API consumer flow, or operator task.
• Load Test Level Selection before choosing unit, component, integration, contract, E2E, workload, synthetic, or monitoring coverage.
• Load Test Oracles And Assertions before writing or reviewing assertions, invariants, snapshots, workload checks, or any test whose failure signal is unclear.
• Load Test Data And Fixtures when state setup, seeds, factories, cleanup, permissions, or data realism affect signal.
• Load Mocking And Test Doubles when a mock, fake, stub, emulator, or real dependency decision could change what risk is preserved.
• Load Test Feedback Loops when choosing local, PR CI, nightly, release, canary, synthetic, or production-monitoring placement.
• Load specialized references only when the fault mechanism calls for them: property-based testing for broad deterministic input spaces, fuzzing for parsers/untrusted input, mutation testing for weak assertions, performance testing for latency/saturation, resilience testing for dependency failure, security testing for abuse or tenant/auth risk, static analysis for code/config-shape defects, and regression selection when the full suite is too slow.

For commands and repo signals in a topic, load the sibling tools.md after the matching overview.md shows that topic is relevant. State which reference files informed the chosen strategy.

Command Selection

Use scan when the user asks what to test next, where coverage would matter, how to prioritize testing work, or which tests would reduce user, production, support, or team risk.

Use test when the user asks to add a test, improve a specific test, cover a bug, or validate a change with a meaningful automated test.

Use workload when the user asks for a realistic user-session workload, scenario generator, traffic model, load/performance scenario, browser journey mix, synthetic user flow, or varied workload that still preserves a stable task goal.

If the user asks to generate a workload, treat existing workloads as evidence and reusable infrastructure, not as the deliverable. A generated workload must add at least one new failure surface, adversarial class, oracle/invariant, state model, dependency fault, user/session path, or coverage dimension. A wrapper, runner, seed sweep, parameter expansion, or documentation-only change around an existing workload is not a generated workload unless the user explicitly asked for a runner or the wrapper adds a new oracle or adversarial model.

Use review when the user asks whether a test is worth keeping, asks for test review, or after $wio test writes or changes a test.

Use doctor when the user asks to audit tests, review suite quality, find flaky or low-value tests, inspect CI test health, or explain why a test suite is slow, noisy, or low-signal.

If the user explicitly names a WIO command, follow that mode. If the command is omitted, infer the mode from the request. If no command or target is provided, show the command table and ask what they want to do. If multiple modes apply, start with scan before test or workload, and use doctor only for existing suite health.

Shared Rules

• Protect meaningful behavior, not coverage numbers.
• Ask what could go wrong before asking what already broke; use that answer to choose tests while the design is still cheap to change.
• Establish product, user, production, support, debugging, review, or release risk before recommending or writing tests.
• Prefer targets where bugs usually occur: boundaries, permissions, state transitions, persistence, external dependencies, concurrency/time, validation/parsing, migrations, configuration, caching, retries/idempotency, UI workflow joins, and recent churn.
• A test is valuable only if it would catch a meaningful regression, save developer time, improve release confidence, or expose a real operational/customer failure mode.
• Before keeping a test or workload, name at least one plausible bug it would catch and the assertion or invariant that would fail.
• Prefer assertions and invariants that encode the mental model of the behavior, including valid cases, invalid cases, and boundary transitions between them.
• Prefer repo-native frameworks, helpers, fixtures, commands, and naming.
• Choose the narrowest test level that preserves the real failure mechanism.
• Read code and tests before choosing strategy; load targeted references after candidate failure modes are known.
• State evidence inspected, commands run, commands not run, and residual risk.
• Mark low-value tests REDO or REMOVE, not KEEP.

Gotchas

• Do not write or keep tests just to increase coverage. Covered code with weak assertions is false confidence.
• Do not mock away the boundary, state, permission, timing, data, or dependency behavior that creates the real risk.
• Do not accept broad snapshots unless the reviewed snapshot is the protected contract and the update path is disciplined.
• Do not use a full-suite command when a smaller command validates the changed behavior with the same signal.
• Do not weaken assertions to fix flakes. First look for nondeterminism, shared state, timing, retries, order dependence, or external services.
• Do not treat green CI as proof the test is valuable. The question is whether the test would fail for the meaningful regression.
• Do not accept tests or workloads that only prove completion, truthiness, object existence, status 200, broad snapshot equality, or mock call counts unless that weak signal is explicitly the protected contract.
• Do not present a thin wrapper around an existing workload as $wio workload generate. If the change only reruns, parameterizes, documents, or sweeps an existing workload, call it a runner and explain the missing new failure surface.
• Do not let subagents write tests or make the final value decision; the main agent owns edits and the final KEEP, REDO, or REMOVE.

Available Scripts

• scripts/test-review-reminder.py: hook helper that reminds the active agent to validate and apply $wio review after test files change.
• scripts/check-wio-report.py: optional checker for saved markdown reports. Use python3 scripts/check-wio-report.py review report.md when a WIO output is written to a file and you need a quick structure check.

Subagent Workflow

When the host supports subagents or parallel agents and user/host policy permits them, use the WIO subagent specs from the official host locations to improve quality without duplicating guidance:

• wio-candidate-scout: read-only discovery of high-value test candidates and real risk.
• wio-strategy-critic: read-only challenge of the chosen test level, oracle, doubles, fixtures, and validation loop before implementation.
• wio-test-reviewer: post-implementation review that returns KEEP, REDO, or REMOVE.

Subagents must inspect targeted code and tests before loading targeted WIO references. They return findings to the main agent; they do not write reports or copy reference content. Claude plugin subagents live in plugins/wio/agents/; Claude project subagents can live in .claude/agents/; Codex custom agents live in .codex/agents/. Installing WIO with skills add does not create those runtime files.

For $wio test, use this sequence:

1. Inspect the repo and target code: public behavior, changed paths, existing tests, fixtures, commands, dependencies, state, side effects, and boundaries. 2. Discover valuable candidates from behavior, risk, bug-prone areas, existing coverage gaps, and code evidence. Use wio-candidate-scout if available. 3. Load the references that match the selected candidate's failure mechanism, then pick the strategy: test level, oracle, fixture/data setup, doubles, adversarial edge coverage, and validation command. 4. Use wio-strategy-critic to challenge the proposed strategy before editing when available. 5. Write one focused test in the main agent, using repo-native patterns. 6. Validate with the smallest relevant command. 7. Review the written test. Use wio-test-reviewer if available. 8. Keep the test only if review returns KEEP; otherwise revise (REDO) or remove (REMOVE).

If subagents are unavailable, perform the same stages in the main agent and explicitly label the review stage.

scan

Find the best parts to test next, the right strategy for each, and the ROI of testing them. This mode is read-only: inspect the repo and existing tests before loading strategy references; do not edit files.

Start with code evidence: inspect product surfaces, target implementation, existing tests, fixtures, and commands. Then use Behavior To Test Map to organize candidates, Risk-Based Testing for ranking tradeoffs, and Test Level Selection only after candidates exist.

Workflow:

1. Establish product/customer context from repo evidence. 2. Inventory existing test frameworks, commands, fixtures, CI, and test layers. 3. Inspect target implementation and nearby tests before choosing a strategy. 4. Map high-value behavior before low-level helpers. 5. Identify bug-prone areas in the scope. 6. Load the references that match each candidate's failure mechanism, then choose the narrowest strategy that preserves the real user or production risk. 7. Rank candidates by impact, likelihood, confidence gap, and cost.

Output template:

## Scope And Evidence
[target, files/commands inspected, tests/CI found]

## Ranked Candidates
1. [behavior] - impact: [why it matters], risk: [fault mechanism], references: [files used], strategy: [level/tool], cost: [small/medium/large]

## Best Next Test
[first investment and why it beats the alternatives]

## Avoid
[coverage-padding or low-signal tests to skip]

## Open Questions
[only questions that would materially change the ranking]

test

Write tests only when they protect meaningful behavior. A useful test reduces future user errors, production incidents, support work, debugging time, review time, or release risk. Do not jump straight to implementation or strategy selection.

Start with code evidence: inspect the target behavior, implementation, existing tests, fixtures, and commands. After selecting a candidate, load Test Level Selection before choosing the test layer and Test Oracles And Assertions before writing assertions. Add data, doubles, feedback, or specialized references when those decisions affect signal.

Workflow:

1. Inspect code, public behavior, existing tests, fixtures, CI, and test commands in the target scope. 2. Discover the highest-value candidate in scope from product risk, bug-prone areas, code shape, existing coverage gaps, and user/developer impact. 3. Load the right strategy references for that candidate's failure mechanism. 4. Pick the strategy from code evidence plus references: test level, oracle, data/fixture setup, doubles, specialized approach, and feedback loop. 5. State the protected behavior, plausible regression it catches, assertion or invariant that would fail, why it matters, references used, and validation command before editing. 6. Write one focused test using repo-native style and existing helpers. 7. Validate with the smallest relevant command. If it is unsafe or unclear, state that instead of guessing. 8. Review the test for value, signal, maintainability, and developer flow impact. 9. Apply the value gate before finalizing: KEEP, REDO, or REMOVE.

Output template:

## Candidate
[behavior/failure mode chosen and why it beat alternatives]

## Strategy
[test level, oracle, data/fixtures, doubles, feedback loop, references used, and why this preserves the real risk]

## Changes
[files changed and concise implementation summary]

## Validation
[command run and result, or why it was not run]

## Review
Verdict: KEEP | REDO | REMOVE
Protected behavior: [...]
Value: [...]
Signal strengths: [...]
False-confidence risks: [...]
Falsification check: [plausible bug and assertion/invariant that would fail]

## Remaining Risk
[what this test does not cover]

workload

Generate workloads that exercise meaningful user sessions, not one-off happy-path tests or wrappers around existing runners. A workload should cover important tasks a real user, API client, operator, or background process performs during a session, with adversarial but realistic misuse and controlled variance that changes data, ordering, scale, timing, or optional branches while preserving the same core task.

Start with code evidence: inspect the user/session entry points, implementation, existing tests, fixtures, commands, and workload/E2E tooling. Existing workloads should reveal gaps, not become the default implementation. After identifying the actor, goal, bug-prone interactions, and existing workload coverage, load Workload Modeling and Test Oracles And Assertions. Add performance, resilience, security, user-behavior, property-based, fuzzing, or data references only when the workload's risk depends on those dimensions.

Originality gate: before editing, state what existing workloads already cover and what the new workload adds. The new workload must introduce at least one of: a new failure surface, adversarial class, oracle/invariant, state model, dependency fault, user/session path, data shape, timing/order dimension, or replay artifact. If the best next step is only a wrapper, runner, seed sweep, or parameter expansion, say that plainly and do not call it a generated workload unless the user asked for that.

Workflow:

1. Inspect user/session entry points, implementation, existing tests, fixtures, commands, and workload/E2E tooling. 2. Inventory existing workloads and summarize their actor, failure surface, oracle/invariants, variance, and replay behavior. 3. Identify the user/session goal and the bug-prone interactions the new workload should expose. 4. State the coverage gap: what existing workloads miss and why that gap matters. 5. Load the references that match the workload's failure mechanism. 6. Choose workload type: browser journey, API scenario, CLI/session script, background-job flow, load profile, synthetic monitor, or property/stateful sequence. 7. Define stable invariants and assertions for correctness, not only completion, and decide which checks run after every step, at terminal state, or eventually. 8. Add adversarial classes deliberately: invalid transitions, duplicate/replayed actions, stale state, permission/tenant edges, malformed-but-valid data, boundary sizes, dependency faults, timing/order changes, partial failure or recovery, and explicit error-handling paths. 9. Add controlled variance with seeds, parameter ranges, optional branches, data shape changes, and recorded replay details. 10. Implement with repo-native libraries, fixtures, helpers, or workload tooling when useful, but do not reuse an existing workload unchanged as the generated workload. 11. Validate with the smallest safe command and report seed, coverage of interactions, limits, and residual risk.

Output template:

## Workload
Actor: [...]
Goal: [...]
Shape: [browser/API/CLI/job/load/synthetic/stateful]
References used: [...]

## Existing Coverage
Existing workloads found: [...]
What they cover: [...]
Gap this workload fills: [...]
Why this is not only a wrapper/runner/seed sweep: [...]

## Coverage
Interactions: [...]
Bug-prone areas: [...]
New failure surface/adversarial class/oracle: [...]
Invariants/assertions: [...]

## Adversarial Model
Misuse paths: [...]
Invalid transitions: [...]
Boundary inputs: [...]
Duplicate/replayed actions: [...]
Permission/tenant edges: [...]
Dependency/time/concurrency faults: [...]

## Variance And Replay
Seed: [...]
Variable inputs/branches/timing/scale: [...]
Replay command or notes: [...]

## Implementation And Validation
Tooling: [...]
Files changed: [...]
Command/result: [...]

## Falsification Check
Plausible bug caught: [...]
Assertion/invariant that fails: [...]
Manual mutation or fault tried, if any: [...]

## Limits
[environment, data, dependency, runtime, cleanup, or flake risk]

review

Review a test as a quality gate, not as a rubber stamp. The test must justify its existence through customer value, production value, support/debugging value, review value, or release confidence.

Start with code evidence: inspect the test diff and protected production behavior. Then load Test Oracles And Assertions. Add data, doubles, feedback-loop, or mutation references only when the test's value depends on those concerns.

Workflow:

1. Inspect the test diff and protected production behavior. 2. Identify the behavior or failure mode the test claims to protect. 3. Load the references needed to judge the test's strategy, oracle, setup, doubles, and feedback loop. 4. Check whether that behavior matters to a user, operator, customer, API consumer, release, support/debugging loop, or developer workflow. 5. Check whether the assertion would fail for the meaningful regression and identify a plausible bug that would make it fail. 6. Check whether setup, fixtures, mocks, and data preserve the real failure mechanism. 7. Check whether the validation command is the smallest useful loop and whether CI placement is appropriate. 8. Return KEEP, REDO, or REMOVE with evidence.

Output template:

Verdict: KEEP | REDO | REMOVE

Protected behavior:
[what behavior or failure mode this test claims to protect]

Value:
[customer, operator, production, support, release, review, or developer-flow value]

Signal strengths:
[why it would fail for the meaningful regression]

False-confidence risks:
[weak assertions, unrealistic setup, over-mocking, snapshots, flake risk, wrong feedback loop]

References used:
[files that informed the decision]

Falsification check:
[plausible bug and assertion/invariant that would fail]

Required action:
[none for KEEP; exact redesign for REDO; removal reason for REMOVE]

doctor

Run a read-only test-suite health scan and report likely concerns with evidence. Do not edit, delete, rewrite, quarantine, or disable tests.

Start with: Test Suite Health Diagnostics. Add flake, feedback-loop, pyramid, mutation, data, doubles, or oracle references only after the suite evidence points there.

Workflow:

1. Identify repository root, language/framework stack, test runners, CI systems, naming conventions, and test layers. 2. Inventory suite shape and test commands. 3. Scan for weak assertions, excessive mocking, flaky timing, shared state, broad snapshots, slow tests, skipped/quarantined tests, and missing critical-risk coverage. 4. Inspect CI and monitoring signals when available. 5. Grade reliability, speed, signal, diagnostic value, maintainability, risk coverage, and monitoring.

Output template:

## Scope And Evidence
[stack, frameworks, CI, test commands, files inspected, tests run/not run]

## Overall
Grade: [A-F or Low/Medium/High trust]
Confidence: [low/medium/high with reason]

## Top Concerns
1. Severity: [P0-P3]
   Concern: [...]
   Evidence: [...]
   Why it matters: [...]
   Suggested action: [...]

## Rubric
Reliability: [...]
Speed: [...]
Signal: [...]
Diagnostic value: [...]
Maintainability: [...]
Risk coverage: [...]
Monitoring/CI fit: [...]

## Quick Wins And Questions
[small actions and only material follow-up questions]

Behavior To Test Map

Purpose

Map product behavior, customer value, code surfaces, dependencies, and risk signals to concrete testing opportunities. Use this when scanning an unfamiliar repo or deciding where an agent should add the next high-value test.

Use When / Avoid When

Core Principles

• Start with customer job, business promise, behavior, and risk; then pick files and test levels.
• Combine four maps: customer/user surface, business-critical workflows, code/dependency graph, and existing test coverage.
• Prefer candidates with clear oracle, stable setup, meaningful failure mode, and local command.
• Do not chase uncovered lines before identifying behavior worth protecting.
• Rank opportunities by customer/business impact, likelihood, confidence gap, and cost to test.
• Recommend one focused test when acting; keep larger plans as candidates.

Decision Rules

Common Failure Modes

• Mapping files to tests without naming the behavior.
• Selecting easy private helpers while ignoring public risk.
• Recommending broad E2E tests for every workflow branch.
• Counting coverage as confidence without inspecting assertions.
• Ranking candidates by developer convenience while ignoring customer, revenue, trust, data, or support impact.
• Ignoring existing commands, fixtures, and local test style.

Output Guidance For Agents

• For each candidate, include part to test, strategy, ROI, evidence, and confidence.
• Explain ROI with customer/business impact, likelihood, confidence gap, and test cost.
• Separate "high confidence from repo evidence" from inference.
• Pick one best next test unless the user asked for a plan.
• If writing a test, cite the selected candidate and keep the implementation focused.

Agent Checklist

• Infer customer profile and product value from README, docs, routes, UI copy, pricing, examples, sales language, support/incident notes, and domain terms.
• Inspect public surfaces: routes, commands, APIs, UI screens, jobs, events.
• Inspect changed/important code: branches, boundaries, side effects, dependencies.
• Inspect existing tests and CI commands.
• Score candidates by impact, likelihood, confidence gap, and cost.
• Choose the lowest level that preserves the risk.

Source Anchors

• Google Testing Blog, Code Coverage Best Practices
• Inozemtseva and Holmes, Coverage Is Not Strongly Correlated with Test Suite Effectiveness
• Martin Fowler, The Practical Test Pyramid
• Google Testing Blog, How Much Testing is Enough?
• Google Testing Blog, Risk-Driven Testing
• Strategyzer, Value Proposition Canvas

Behavior To Test Map Tools

Use these signals to discover test candidates in an unfamiliar repo. The output should be a ranked map, not a dump of every uncovered file.

Tool Categories

Repo Signals To Inspect

Common Commands And Patterns

Candidate Evidence Template

Evidence Rules

• Rank candidates with explicit confidence: confirmed by tests/code/docs/product evidence, inferred from code shape, or speculative.
• If business context is inferred, say so; do not overstate ROI as fact.
• Do not recommend a test whose expected behavior cannot be stated.
• Prefer candidates that can be implemented with existing test patterns and commands.
• Flag missing tooling as a separate prerequisite, not as the test candidate itself.

Source Anchors

• Google Testing Blog, Code Coverage Best Practices
• Google Testing Blog, Test Sizes
• GitLab, JUnit report examples by tool
• Playwright, best practices
• Harvard Business School Working Knowledge, What Customers Want from Your Products

Flaky Test Detection and Management

Strategy Map

Purpose

Detect, classify, debug, and reduce nondeterministic test outcomes without normalizing unreliable CI.

Reliability Goal

Protect the delivery signal by distinguishing product regressions from unreliable tests and by fixing root causes such as time, concurrency, shared state, external services, or infrastructure variance.

When This Strategy Applies

• A test has both pass and fail outcomes on the same commit.
• Failures disappear after rerun, isolation, changed order, or changed parallelism.
• The code or tests depend on time, randomness, async work, browsers, mobile devices, threads, shared fixtures, external services, or CI resources.
• CI uses retries, quarantine, sharding, or merge gates where hidden flakiness can distort release decisions.

When This Strategy Does Not Apply

• A failure is deterministic and explained by the diff.
• The only proposed action is increasing retries.
• A test is obsolete and should be rewritten or removed after risk review.
• The task is a small deterministic behavior change with no flake evidence.

Signals To Inspect First

• Same-commit pass/fail history, retry metadata, seed values, test order, parallelism, logs, screenshots, traces, fixture cleanup, fixed sleeps, wall-clock reads, unseeded randomness, shared state, external network calls, CI image changes, browser/device versions, and resource limits.

Test Design Principles

• Retries are a detection and productivity tool only when retry outcomes remain visible.
• Quarantine reduces gate damage but must keep execution, ownership, evidence, and a return path.
• Bounded condition waiting is usually better than fixed sleeps.
• Some flaky-test fixes belong in production code because nondeterministic tests can expose real races or state leaks.
• Repetition samples behavior; it cannot prove absence of low-probability flakes.

Good Test Characteristics

• Tests wait for observable conditions, not arbitrary time.
• State is isolated with unique data, transactions, temporary resources, or clean teardown.
• Random and generated tests record seeds and shrink failures.
• Retry classifications preserve first-attempt failures.
• Failure artifacts are sufficient to reproduce or narrow the cause.

Poor Test Characteristics

• Global retries make red CI green without recording flake status.
• Permanent skips or quarantines remove coverage silently.
• Tests assert exact timing, unordered output order, animation timing, or log wording when not required.
• Live third-party services are used in ordinary unit tests.
• Background work, ports, files, users, or databases leak across tests.

Execution Pattern

• Confirm whether the failure is deterministic or same-commit nondeterministic.
• Collect artifacts, environment metadata, order, seed, and retry attempts.
• Reproduce with targeted repeats, isolation, order shuffling, and parallelism changes.
• Identify root cause category: test bug, production race, shared state, async wait, environment, dependency, or resource limit.
• Fix the root cause or quarantine with owner and expiry.
• Run repeated targeted validation and the smallest impacted suite.
• Report whether confidence is sampled and what flake risk remains.

Examples

• Weak: await sleep(5000) after clicking submit. Stronger: wait until the order row exists or the success region is visible, with a bounded timeout and useful failure artifact.
• Weak: enabling retries on every unit test. Stronger: classify pass-after-fail in CI, record the attempt, and fix shared state or timing in the specific flaky test.

Validation

• Run the suspected test repeatedly under the same commit.
• Run isolated and in-suite forms when order dependency is suspected.
• Vary seed, order, and parallelism only as far as the test framework supports deterministically.
• Confirm retry results remain visible in reports.
• For fixes, compare failure rate before and after using enough repetitions to be meaningful, while acknowledging sampling limits.

Failure Modes

• Retry suppression hides real defects.
• Quarantines become permanent and reduce gate coverage.
• Reproduction loops are too broad and waste CI.
• Flake fixes weaken assertions instead of controlling nondeterminism.
• Tests pass locally but still depend on CI-only resources or timing.

Overview

Flaky test management protects the trustworthiness of CI by identifying tests that pass and fail under the same code revision, preserving evidence, and driving root-cause fixes. It is a control layer around unit, integration, browser, mobile, and system tests; it is not a reason to normalize unreliable gates.

Retries and quarantine can reduce immediate disruption, but only if first-attempt failures, owners, artifacts, and expiry are visible. A pass-after-retry is still a flake signal.

Best Fit

Use this strategy when CI gates merges or releases, retries are common, failures vanish after rerun, or tests depend on time, randomness, async work, concurrency, browsers, mobile devices, external services, shared fixtures, or constrained CI resources.

It is highest value in large suites, monorepos, browser/mobile automation, distributed systems, queues, scheduled jobs, event streams, caches, database-heavy tests, and teams where rerun culture is eroding trust.

Candidate Matrix

Root-Cause Map

Signals

Management Rules

• Prefer root-cause fixes over weaker assertions.
• Use quarantine only with owner, reason, evidence, expiry, and continued non-blocking execution.
• Keep retries visible; never collapse pass-after-fail into ordinary green.
• Reproduce with targeted repeats, isolation, order shuffle, parallelism changes, and seed replay.
• Validate fixes with repeated targeted runs plus the smallest impacted suite.
• State that repeated passes are sampled confidence, not proof of stability.

Limits And Tradeoffs

Examples

Packages And Libraries

Source Anchors

• Google and Microsoft engineering material both treat same-code inconsistent outcomes as a CI trust problem, not a normal test result.
• Playwright, Bazel, Develocity, Datadog, and Microsoft-style systems preserve retry/flaky status as distinct signal.
• pytest and framework docs repeatedly call out overly strict timing, floating-point, ordering, and external dependency assertions as flake sources.
• Flaky-test research identifies async waits, concurrency, order dependency, isolation problems, remote services, and resource leaks as common causes.

Quality Bar

• Flake claims cite same-commit pass/fail evidence when possible.
• First-attempt and retry outcomes remain visible.
• Fixes control nondeterminism rather than deleting meaningful assertions.
• Quarantined tests keep ownership, execution, artifacts, and a return path.
• Residual risk names sample size, commands run, and commands not run.

Tools

For flaky test detection and management, the practical tools are usually test runners or runner plugins that can re-run only failures, label pass-after-retry tests as flaky, emit CI-readable reports, and optionally fail the build when flakes appear. Choose the tool that matches the project’s runtime first, then prefer configurations that make flakes visible rather than silently hiding them.

Playwright Test

• Use for: Browser/E2E suites where retries, trace capture, and flaky classification should be built into the test runner.
• Languages/ecosystem: JavaScript/TypeScript, with Playwright ecosystems for web testing.
• Why it is trusted: Playwright Test officially categorizes results as passed, flaky, or failed when retries are enabled, and supports first-retry traces for debugging.
• Official docs: https://playwright.dev/docs/test-retries
• Good usage pattern:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [['list'], ['junit', { outputFile: 'test-results/junit.xml' }]],
  use: {
    trace: 'on-first-retry',
  },
});

pytest-rerunfailures

• Use for: Python test suites that need pytest-native retries, per-test flaky marks, and CI failure on detected flakes.
• Languages/ecosystem: Python / pytest.
• Why it is trusted: The pytest-rerunfailures docs cover global retries, per-test @pytest.mark.flaky, exception filtering, and --fail-on-flaky exit behavior.
• Official docs: https://pytest-rerunfailures.readthedocs.io/latest/
• Good usage pattern:

# pytest.ini
[pytest]
reruns = 2
reruns_delay = 1
addopts =
    --fail-on-flaky
    --only-rerun TimeoutError
    --only-rerun ConnectionError

Apache Maven Surefire / Failsafe

• Use for: JVM projects using Maven that need retry-based flake detection for unit tests and integration tests.
• Languages/ecosystem: Java/JVM; JUnit 4, JUnit 5, TestNG-adjacent Maven test workflows.
• Why it is trusted: Maven Surefire/Failsafe are Apache Maven’s standard test plugins; their docs support rerunFailingTestsCount, flaky XML report elements, and failOnFlakeCount gates.
• Official docs: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
• Good usage pattern:

<!-- pom.xml: fail CI when a unit test passes only after retry -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <rerunFailingTestsCount>2</rerunFailingTestsCount>
    <failOnFlakeCount>1</failOnFlakeCount>
  </configuration>
</plugin>

Use the Failsafe equivalent for integration tests run in verify.

gotestsum

• Use for: Go projects that want go test output, JUnit reports, JSON logs, and failed-test reruns in CI.
• Languages/ecosystem: Go.
• Why it is trusted: gotestsum wraps go test, writes JUnit XML and JSON logs, re-runs failed tests, and documents guardrails such as max failure limits before reruns.
• Official docs: https://github.com/gotestyourself/gotestsum
• Good usage pattern:

gotestsum \
  --format=pkgname \
  --junitfile=reports/go-tests.xml \
  --jsonfile=reports/go-tests.jsonl \
  --rerun-fails=2 \
  --rerun-fails-max-failures=5 \
  --packages="./..." \
  -- -count=1 -race

cargo-nextest

• Use for: Rust workspaces that need fast CI test runs with retries, flaky marking, per-profile policy, and JUnit output.
• Languages/ecosystem: Rust / Cargo.
• Why it is trusted: cargo-nextest officially marks pass-after-retry tests as flaky, supports flaky-result = "fail", per-test overrides, delay/backoff, and JUnit flaky tags.
• Official docs: https://nexte.st/docs/features/retries/
• Good usage pattern:

# .config/nextest.toml
# CI command: cargo nextest run --profile ci
[profile.ci]
fail-fast = false
retries = { backoff = "fixed", count = 2, delay = "1s" }
flaky-result = "fail"
[[profile.ci.overrides]]
filter = 'test(remote_api)'
retries = { backoff = "exponential", count = 2, delay = "5s", jitter = true }

Fuzz Testing / Continuous Fuzzing

Strategy Map

Purpose

Exercise input-handling boundaries with generated or mutated inputs to find crashes, hangs, memory errors, parser failures, and invariant violations.

Reliability Goal

Reduce robustness and security risk in code that processes untrusted, malformed, high-volume, or complex structured input.

When This Strategy Applies

• The change touches parsers, decoders, serializers, protocol handlers, file formats, compression, regexes, normalization, URL/path handling, image/media processing, or unsafe/native code.
• A deterministic harness can call the behavior quickly with generated input.
• A crash, timeout, sanitizer finding, or invariant violation is a meaningful failure.
• Continuous fuzzing infrastructure or seed corpus already exists, or the target is high-risk enough to justify adding it.

When This Strategy Does Not Apply

• No clear input boundary or oracle exists.
• The behavior depends on live services, real time, global state, or irreversible side effects.
• The only assertion is vague and expensive.
• The target is ordinary CRUD logic better covered by examples, properties, contract, or integration tests.
• The team cannot triage minimized crashes or maintain seed corpora.

Signals To Inspect First

• Input parsers, public deserialization APIs, untrusted uploads, protocol messages, seeds/corpora, sanitizer builds, OSS-Fuzz/ClusterFuzz config, fuzz target naming, crash artifacts, timeout thresholds, memory limits, and previous malformed-input bugs.

Test Design Principles

• A fuzz target should be small, deterministic, fast, and side-effect controlled.
• The oracle can be crash-free execution, sanitizer cleanliness, parse/serialize invariant, resource bound, or business invariant.
• Seed corpora should include real and boundary examples, not only random bytes.
• Prefer targeted subsystem fuzzers in addition to whole-system fuzzing; broad simulations can miss deep states or make precise assertions hard.
• Design minimal fuzzable interfaces early. Remove accidental dependencies and expose the essential input/output transformation so the fuzzer reaches meaningful states quickly.
• Cover positive and negative space separately: valid inputs should round trip or preserve semantics; invalid encodings should reject loudly and never silently misinterpret data.
• Crashes become regression tests by adding minimized inputs.
• Continuous fuzzing needs ownership, deduplication, minimization, and triage.

Good Test Characteristics

• Harnesses isolate one input boundary and reset state each run.
• Sanitizers or runtime checks are enabled where supported.
• Timeouts and resource limits catch hangs and pathological inputs.
• Generated-input distribution is inspected so structured generators do not create blind spots.
• Exact or model-based checks are used when count-only or subset checks can miss missing or extra results.
• Crashes are minimized, stored, and replayed in deterministic tests.
• Short PR fuzz runs are paired with longer scheduled campaigns for high-risk code.

Poor Test Characteristics

• A broad application boot fuzzes everything and diagnoses nothing.
• The target writes to shared databases or external services.
• Crashes are ignored because they are “just fuzz input.”
• Generated inputs never reach parser states because no seed corpus exists.
• Structured generation accidentally avoids the hard case, such as always producing consecutive records, aligned ranges, or easy orderings.
• Fuzzing is claimed as security coverage for authorization or business logic without a relevant oracle.

Execution Pattern

• Identify the input boundary and failure oracle.
• Build a minimal deterministic harness around public behavior.
• Seed with valid, invalid, boundary, and regression inputs.
• Run a short local/CI fuzz pass and replay any failures.
• Minimize and deduplicate crashes.
• Add minimized inputs to regression corpus or ordinary tests.
• Schedule longer continuous fuzzing for high-risk targets.

Examples

• Weak: fuzz an HTTP server through a live port with random bytes and no artifact capture. Stronger: fuzz the request parser or route-normalizer function directly, assert controlled errors, enable sanitizers, and store minimized crashing inputs.
• Weak: fuzz authorization decisions with random users but no expected policy. Stronger: use property tests or model-based tests for policy invariants; reserve fuzzing for malformed token or parser robustness.

Validation

• Run the fuzz target for the repository’s standard short duration.
• Replay minimized failures outside the fuzzer.
• Confirm regression corpus cases fail before the fix when applicable and pass after.
• Check the target is deterministic and free of external side effects.
• Inspect coverage/corpus growth as a diagnostic, not a proof of completeness.

Failure Modes

• Harnesses are too slow or broad for CI.
• Nondeterminism creates irreproducible crashes.
• No oracle means bugs are missed except crashes.
• Corpus and artifacts are not retained.
• Fuzzing is used where risk is authorization, integration, or performance rather than input robustness.

Overview

Fuzz testing feeds many generated inputs into a deterministic harness to find crashes, hangs, memory errors, parser bugs, invariant violations, and unsafe handling of untrusted data. Continuous fuzzing keeps that search running over time and turns minimized crashes into regressions.

Fuzzing is strongest when the input boundary is clear, the harness is fast, failures are reproducible, and the oracle is at least “no crash, no timeout, controlled error,” preferably with semantic invariants.

Best Fit

Use fuzzing for parsers, decoders, serializers, file formats, URL/path handling, protocol handlers, decompression, cryptography wrappers, query languages, config loaders, API request validators, and any boundary that accepts untrusted or structured input.

Use continuous fuzzing when the code changes often, has security exposure, or has a history of input-handling defects. Seed corpora and minimized crash artifacts are part of the product; keep them reviewed and versioned.

Candidate Matrix

When Not To Use

Avoid fuzzing when there is no deterministic harness, no meaningful input boundary, no reproducible failure path, or no owner to triage findings. Do not replace semantic tests with fuzzing; fuzzers are excellent at finding surprising cases but still need clear oracles.

For deterministic business invariants over valid data, property-based testing may be a better first tool. For production abuse cases involving auth, rate limits, and tenant boundaries, pair fuzzing with security tests.

Harness Design Notes

Signals

Workflow

1. Choose one input boundary and build a small deterministic harness. 2. Add seed corpus from valid examples, regressions, edge cases, and protocol fixtures. 3. Inspect generator or corpus distribution for positive, negative, boundary, and adversarial classes. 4. Run locally with sanitizers or coverage guidance when available. 5. Minimize crashes and commit regression seeds when useful. 6. Add CI or scheduled continuous fuzzing only after local harnesses are stable. 7. Track findings with owner, artifact, minimized input, command, and fix status.

Continuous Fuzzing Shape

Examples

Packages And Libraries

Source Anchors

• libFuzzer frames the core pattern as fast, in-process, coverage-guided fuzzing of a target function.
• Go fuzzing treats seed corpora, minimized failing inputs, and replay through normal tests as part of the workflow.
• OSS-Fuzz, ClusterFuzz, ClusterFuzzLite, and CIFuzz separate short change-focused fuzzing from longer batch fuzzing, corpus pruning, minimization, and triage.
• Sanitizers such as ASan, UBSan, and TSan make fuzzing more valuable for memory, undefined behavior, and race defects, but they add build/runtime cost.

Quality Bar

• Harness is deterministic, fast, isolated, and bounded.
• Seeds include valid examples, malformed inputs, boundaries, and prior failures.
• Generator distribution has been inspected for blind spots.
• Crashes include minimized input, reproducer command, sanitizer output when relevant, and owner.
• Semantic invariants are added where “does not crash” is too weak.
• Continuous fuzzing has triage policy and does not create unowned alert noise.

Tools

For fuzz testing, the key tool types are coverage-guided fuzzers, language-native fuzz test runners, and CI orchestration for corpus/crash retention. Choose the language-native option when it exists, AFL++ for native/binary targets, and ClusterFuzzLite when the main need is continuous fuzzing in CI rather than a local fuzzing campaign.

AFL++

• Use for: High-throughput coverage-guided fuzzing of C/C++ libraries, CLI parsers, file-format handlers, and binary-oriented targets.
• Languages/ecosystem: C, C++, native binaries; also useful for many projects that can expose a command-line harness.
• Why it is trusted: AFL++ has official quick-start guidance for source-available, binary-only, network, and GUI targets, with standard afl-cc / afl-fuzz workflows and crash replay guidance.
• Official docs: https://aflplus.plus/docs/
• Good usage pattern: Build an instrumented target, seed it with small valid inputs, and cap unattended runs with AFL++ environment controls. AFL_EXIT_ON_TIME exits after no new paths are found for the configured seconds.

mkdir -p seeds findings
printf '{"schema":1,"items":[]}\n' > seeds/valid.json
CC=afl-cc CXX=afl-c++ cmake -S . -B build -DENABLE_FUZZING=ON
cmake --build build --target json_parser_fuzz
AFL_NO_UI=1 AFL_EXIT_ON_TIME=300 \
  afl-fuzz -i seeds -o findings -- ./build/json_parser_fuzz @@

ClusterFuzzLite

• Use for: Continuous fuzzing in CI, especially PR fuzzing plus scheduled longer batch fuzzing.
• Languages/ecosystem: CI orchestration for fuzz targets; supports GitHub Actions, GitLab, Google Cloud Build, and Prow workflows.
• Why it is trusted: It provides first-class modes for code-change fuzzing, batch fuzzing, corpus pruning, coverage reports, and continuous builds, with official GitHub Actions examples.
• Official docs: https://google.github.io/clusterfuzzlite/
• Good usage pattern: Use PR fuzzing for fast feedback, then add scheduled batch fuzzing later to build a stronger corpus.

name: ClusterFuzzLite PR fuzzing
on:
  pull_request:
    paths: ['**']
permissions: read-all
jobs:
  fuzz:
    runs-on: ubuntu-latest
    steps:
      - name: Build fuzzers
        uses: google/clusterfuzzlite/actions/build_fuzzers@v1
        with:
          language: c++
          github-token: ${{ secrets.GITHUB_TOKEN }}
          sanitizer: address
      - name: Run fuzzers
        uses: google/clusterfuzzlite/actions/run_fuzzers@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          mode: code-change
          fuzz-seconds: 600
          sanitizer: address
          output-sarif: true

Go Fuzzing

• Use for: Fuzzing Go package APIs directly inside normal go test workflows.
• Languages/ecosystem: Go standard toolchain, testing.F, go test -fuzz.
• Why it is trusted: Go fuzzing is built into the standard toolchain from Go 1.18, uses coverage guidance, and stores failing inputs as regression corpus entries run by future go test executions.
• Official docs: https://go.dev/doc/security/fuzz/
• Good usage pattern: Keep fuzz tests deterministic, seed realistic examples with f.Add, and use -fuzztime for CI-bounded runs.

package query
import (
	"net/url"
	"reflect"
	"testing"
)
func FuzzParseQueryRoundTrip(f *testing.F) {
	f.Add("user=alice&role=admin")
	f.Add("q=a%2Bb&empty=")
	f.Fuzz(func(t *testing.T, raw string) {
		values, err := url.ParseQuery(raw)
		if err != nil {
			return
		}
		reparsed, err := url.ParseQuery(values.Encode())
		if err != nil {
			t.Fatalf("encoded query is not parseable: %v", err)
		}
		if !reflect.DeepEqual(values, reparsed) {
			t.Fatalf("round trip changed values: %#v != %#v", values, reparsed)
		}
	})
}
// CI smoke run:
// go test ./... -run=FuzzParseQueryRoundTrip -fuzz=FuzzParseQueryRoundTrip -fuzztime=60s

cargo-fuzz

• Use for: Coverage-guided fuzzing of Rust crates, especially parsers, codecs, protocol handlers, and unsafe-code boundaries.
• Languages/ecosystem: Rust, Cargo, libFuzzer via libfuzzer_sys.
• Why it is trusted: The Rust Fuzz Book documents cargo fuzz init, checked-in fuzz targets under fuzz/fuzz_targets, and cargo fuzz run <target> using libFuzzer.
• Official docs: https://rust-fuzz.github.io/book/cargo-fuzz.html
• Good usage pattern: Keep the harness narrow, reject invalid inputs cheaply, assert semantic invariants, and run a bounded libFuzzer session in CI.

#![no_main]
#[macro_use]
extern crate libfuzzer_sys;
use serde_json::Value;
fuzz_target!(|data: &[u8]| {
    if let Ok(value) = serde_json::from_slice::<Value>(data) {
        let encoded = serde_json::to_vec(&value).unwrap();
        assert!(serde_json::from_slice::<Value>(&encoded).is_ok());
    }
});
// CI smoke run:
// cargo fuzz run serde_json_roundtrip -- -max_total_time=60

Jazzer

• Use for: Coverage-guided fuzzing of JVM libraries, parsers, deserializers, and security-sensitive Java/Kotlin/Scala code.
• Languages/ecosystem: JVM, JUnit 5, Maven, Gradle, Bazel.
• Why it is trusted: Jazzer is a coverage-guided in-process JVM fuzzer based on libFuzzer and has official JUnit integration with @FuzzTest, regression mode, and fuzzing mode via JAZZER_FUZZ=1.
• Official docs: https://github.com/CodeIntelligenceTesting/jazzer
• Good usage pattern: Write fuzz tests beside unit tests, constrain generated inputs with Jazzer annotations, and let crash inputs become regression cases.

import com.code_intelligence.jazzer.junit.FuzzTest;
import com.code_intelligence.jazzer.mutation.annotation.NotNull;
import com.code_intelligence.jazzer.mutation.annotation.WithUtf8Length;
import java.net.URI;
import java.net.URISyntaxException;
import static org.junit.jupiter.api.Assertions.assertEquals;
class UriFuzzTest {
  @FuzzTest
  void normalizedUriStaysParseable(
      @NotNull @WithUtf8Length(min = 1, max = 2048) String input) {
    try {
      URI normalized = new URI(input).normalize();
      assertEquals(normalized, new URI(normalized.toString()));
    } catch (URISyntaxException ignored) {
      // Invalid URIs are acceptable; crashes and invariant failures are not.
    }
  }
}
// Fuzzing mode:
// JAZZER_FUZZ=1 mvn test

Testing Practice References

Reference selection starts with the reliability risk, not with the easiest test to write. These practices are complementary: a high-quality testing plan often combines a focused behavior test, a boundary or integration check, and an honest statement of residual risk.

WIO Entry Points

WIO exposes one skill with five command modes:

Load only the reference files needed for the current decision, but do not skip references at the strategy step. The intended order is code evidence, candidate discovery, then reference-guided strategy selection.

Exploration Order

1. Infer product/customer context from README, docs, examples, UI copy, routes, API docs, pricing/plans, support/incident notes, and domain terms. 2. Inventory test files, commands, framework config, CI jobs, fixtures, skips, retries, reports, and existing naming/style. 3. Inspect the production code behind the reviewed or candidate test: public behavior, dependencies, state, side effects, data, and boundaries. 4. Identify candidate behaviors or workloads and list likely bug-prone areas before choosing unit, component, integration, contract, E2E, workload, monitoring, or specialized testing. 5. Load the specific reference files that match the selected candidate's failure mechanism; use each tools.md sibling for repo signals and commands when implementation or validation details matter. 6. Choose the strategy from both code evidence and the loaded references. 7. Report concise evidence and references used, not internal exploration notes.

Bug-Prone Areas

Use this list during $wio scan, $wio test, and $wio workload before picking the strategy. It is a targeting aid, not a reason to test every line.

WIO Test Pipeline

$wio test should not jump straight to writing code. The expected pipeline is:

1. Inspect the relevant code, public behavior, existing tests, fixtures, commands, and CI shape. 2. Discover a candidate with real user, production, support, release, review, or developer-flow value in a bug-prone area. 3. Load the references that match the candidate's fault mechanism. 4. Pick the strategy from code evidence plus references: test level, oracle, data/fixture setup, doubles, specialized approach, and feedback loop. 5. Write one focused repo-native test. 6. Validate with the smallest relevant command. 7. Review the written test for value and signal. 8. Return KEEP, REDO, or REMOVE.

When subagents are available, use:

• wio-candidate-scout for discovery.
• wio-strategy-critic before editing.
• wio-test-reviewer after validation.

These subagents are process accelerators, not separate doctrine.

WIO Workload Pipeline

$wio workload should produce a realistic, adversarial, replayable scenario rather than a random script or a thin wrapper around an existing workload. Existing workloads are evidence and infrastructure; the generated workload must add a new failure surface, adversarial class, oracle/invariant, state model, dependency fault, user/session path, data shape, timing/order dimension, or replay artifact.

1. Inspect the code, entry points, existing tests, fixtures, commands, and any workload or E2E tooling. 2. Inventory existing workloads and summarize their actor, failure surface, oracle/invariants, variance, and replay behavior. 3. Identify the actor, session goal, and bug-prone interactions. 4. State the gap the new workload fills beyond existing workload coverage. 5. Load workload, oracle, and any specialized references that match the failure mechanism. 6. Pick workload shape and execution loop. 7. Define correctness assertions, invariants, and failure artifacts. 8. Add adversarial classes deliberately: invalid transitions, duplicate/replayed actions, stale state, boundary data, permission/tenant edges, timing/order changes, and dependency faults. 9. Add bounded variance with seed/replay details. 10. Implement with repo-native helpers or tooling when asked to edit, but do not reuse an existing workload unchanged as the generated workload. 11. Validate safely and report limits.

Quick Selection

Decision Path

1. If the intended property is unclear, start with Testing Core Concepts, then inspect requirements, public APIs, existing tests, incidents, and user workflows before writing tests. 2. If code is hard to exercise, use Testability. 3. If the question is where a test belongs, use Test Level Selection and Test Automation Pyramid. 4. If behavior should come from user workflows, use User Behavior Testing. 5. If a varied user session, traffic model, or operation sequence should expose interaction bugs, use Workload Modeling. 6. If dependency substitution is the decision, use Mocking And Test Doubles. 7. If the suite exists but trust is low, use Test Suite Health Diagnostics. 8. If red CI often turns green after reruns, use Flaky Test Detection and Management. 9. If defects are recognizable from code or config shape, use Static Testing / Static Analysis. 10. If security risk extends beyond code shape, use Security Testing Beyond SAST. 11. If test capacity is constrained, use Risk-Based Testing. 12. If examples keep missing deterministic edge cases, use Property-Based Testing. 13. If untrusted or structured inputs can crash, hang, corrupt, or violate invariants, use Fuzz Testing / Continuous Fuzzing. 14. If coverage is high but assertions feel weak, use Mutation Testing. 15. If full regression is too slow, use Regression Test Selection / Test Impact Analysis. 16. If user-visible reliability depends on traffic, latency, or saturation, use Performance, Load, and Stress Testing. 17. If dependency or infrastructure failure is the risk, use Resilience Testing and Fault Injection.

Cross-Cutting Testing Judgment

• Coverage is not confidence; covered code can still have weak assertions or no useful oracle.
• Passing tests are not proof of reliable software; they are evidence about selected behaviors under selected conditions.
• The best test level is the lowest level that preserves the real risk.
• The best assertion is one that would fail for the regression or failure mode that matters.
• Mocks are useful for boundaries, speed, and determinism, but excessive mocking can test the mock instead of the system.
• Fast tests are valuable when they still exercise the behavior at risk.
• End-to-end tests are valuable for critical workflows but should remain selective, observable, and debuggable.
• Snapshot tests need a clear protected contract; unreviewed snapshot updates create low-signal approval tests.
• Regression tests should prove that a specific failure cannot silently return.
• Candidate selection should maximize test ROI: customer/business impact, likelihood, confidence gap, and cost to test.
• Workloads should model real sessions with bounded, recorded variance and correctness assertions, and generated workloads should clearly state what they add beyond existing workload coverage.
• Validation reports should name commands run, commands not run, and residual risk.

Mocking And Test Doubles

Purpose

Use test doubles to control dependencies while preserving the behavior risk under test. A double is a tool for isolation, determinism, speed, or boundary modeling, not a substitute for understanding the real dependency.

Use When / Avoid When

Core Principles

• Prefer state/output assertions. Use interaction assertions when the externally visible behavior is the interaction.
• Choose the lightest double that preserves risk: dummy, stub, fake, spy, mock, emulator, container, or contract test.
• Keep doubles honest at boundaries with contract tests, schema validation, provider verification, or a smaller number of real-dependency integration tests.
• Do not mock the system under test. Mock collaborators at boundaries.
• Avoid strict call-order assertions unless order is part of the contract.
• Use fakes for meaningful behavior only when their rules are simpler and independently trustworthy.

Decision Rules

Common Failure Modes

• Over-mocking every collaborator, producing fast tests with low product fidelity.
• Verifying implementation call sequences instead of behavior or contract.
• Fakes that drift from production semantics.
• Patching the wrong namespace or leaving patches active across tests.
• Mocking framework, language, or library behavior instead of owning a testable adapter.

Output Guidance For Agents

• Name the dependency replaced, why it is replaced, and what risk remains untested.
• State the double type and the contract it models.
• Keep assertions focused on observable behavior or the externally meaningful interaction.
• Add or point to an integration/contract test when the double stands in for a critical boundary.

Agent Checklist

• Identify whether the dependency is part of the behavior claim.
• Use the real dependency if it is cheap and deterministic.
• Place doubles at ownership boundaries, not deep inside implementation.
• Avoid strict interaction checks unless the protocol is the behavior.
• Run at least one real/contract path for critical external boundaries.

Source Anchors

• Martin Fowler, Mocks Aren't Stubs
• Martin Fowler, TestDouble
• Pact, contract testing introduction
• Google Testing Blog, Increase Test Fidelity By Avoiding Mocks

Mocking And Test Doubles Tools

Prefer the repo's existing double framework. Add a new tool only when the current stack cannot model the boundary cleanly.

Tool Categories

Repo Signals To Inspect

Common Commands And Patterns

Evidence Rules

• A mock is justified when it removes irrelevant cost/nondeterminism or verifies a meaningful outbound protocol.
• A fake is justified when its semantics are smaller than production and still representative for the behavior.
• A container/emulator is justified when real dependency semantics are the risk.
• Flag tests that would pass if production never integrated with the real provider.

Source Anchors

• Python, unittest.mock
• Mockito, documentation
• Pact, consumer tests
• Docker, Testcontainers

Mutation Testing

Strategy Map

Purpose

Audit whether existing tests fail when small artificial behavioral changes are introduced into code under test.

Reliability Goal

Reduce false confidence from tests that execute code but do not assert behavior strongly enough to catch meaningful regressions.

When This Strategy Applies

• Normal tests already pass and are reasonably fast/deterministic.
• The target is high-risk deterministic logic: authorization, billing, eligibility, validation, state transitions, parsers, quotas, or compliance rules.
• Coverage is high but confidence in assertions is low.
• A bug escaped because tests missed a boundary, boolean, comparison, or omitted side effect.

When This Strategy Does Not Apply

• The baseline suite is failing, flaky, or too slow.
• The target is generated, vendored, boilerplate, UI-layout, migration, schema-only, or low-risk glue code.
• The main risk is integration compatibility, performance, accessibility, rollout, or visual behavior.
• The team would chase 100% mutation score without inspecting surviving mutants.

Signals To Inspect First

• Mutation config, fast unit/component tests, code coverage, high-risk modules, escaped defects, weak assertions, surviving mutant reports, equivalent mutants, timeout settings, excluded generated code, and test runtime.

Test Design Principles

• Mutation score is a proxy for test sensitivity, not correctness.
• Surviving mutants are prompts for inspection; some are equivalent, dead code, redundant logic, or unspecified behavior.
• Target small high-risk scopes before broad runs.
• A useful response is a stronger behavior test, not blindly asserting implementation details.
• Mutation amplifies flakiness and runtime cost.

Good Test Characteristics

• Runs are scoped to changed or high-risk code.
• Surviving mutants are reviewed line by line with behavior context.
• New tests assert public behavior, boundary cases, negative cases, and side effects.
• Equivalent or irrelevant mutants are documented or excluded narrowly.
• Mutation reports are used to improve tests, not as vanity scores.

Poor Test Characteristics

• Running mutation against slow E2E or snapshot-heavy suites.
• Adding brittle implementation-detail assertions only to kill mutants.
• Mutating generated code or migrations.
• Treating survived mutants as always real bugs.
• Reporting a score without naming the missing behavior.

Execution Pattern

• Run baseline tests first.
• Select a focused high-risk target.
• Run the repository mutation tool with existing config or a narrow scope.
• Inspect surviving mutants and classify meaningful, equivalent, dead-code, or out-of-scope.
• Add or improve behavior tests for meaningful survivors.
• Rerun targeted tests and the mutation subset.
• Report remaining survivors and rationale.

Examples

• Weak: tests cover withdrawal below and above balance but not exact balance. A <= to < mutant survives. Stronger: add an exact-balance boundary test asserting full withdrawal is allowed.
• Weak: kill a discount authorization mutant by asserting a private helper call. Stronger: test that an unauthorized user cannot apply the discount and persisted pricing remains unchanged.

Validation

• Confirm baseline tests pass before mutation.
• Verify the added test fails for the meaningful mutant or original bug.
• Rerun mutation for the focused target only.
• Check for flaky tests and timeouts before trusting scores.
• Do not use coverage or mutation score alone as proof of correctness.

Failure Modes

• Equivalent mutants waste time.
• Mutation runs become too slow for developer feedback.
• Snapshot tests kill mutants for irrelevant changes.
• High scores hide missing integration or contract tests.
• Agents overfit tests to mutants instead of user-visible behavior.

Overview

Mutation testing checks whether existing tests fail when small, meaningful changes are injected into production code. It measures assertion strength, not coverage volume. A high line-coverage suite can still miss mutants if it executes code without verifying the behavior that matters.

Use mutation results as a diagnostic for important logic with existing tests. Do not introduce it as a broad gate before the suite is deterministic, reasonably fast, and trusted.

Best Fit

Use mutation testing for deterministic, high-value logic where tests already run and correctness matters: calculations, permissions, validation, parsers, state machines, pricing, scheduling, retry/idempotency rules, data transformations, and security-sensitive branches.

It works best when scoped to changed files, critical modules, or post-incident areas. Whole-repo mutation runs are often too slow and noisy unless the project already has mature tooling and baselines.

Score Interpretation

Candidate Matrix

When Not To Use

Avoid mutation testing when the baseline suite is failing, flaky, extremely slow, mostly snapshot-based, or dominated by generated code and low-risk glue. Do not chase 100 percent mutation score; equivalent mutants and low-value code make that target wasteful.

Do not use mutation score as a team ranking metric. Use it to decide which tests to improve, which code needs clearer seams, and where risk is acceptable.

Signals

Workflow

1. Run mutation testing on the smallest meaningful target. 2. Review surviving mutants and discard equivalent or low-risk cases explicitly. 3. Add or strengthen behavior tests for meaningful survivors. 4. Re-run the target and report score only with scope and caveats. 5. Expand to adjacent modules only when the risk justifies the cost.

Test Improvement Rules

• Add tests that express the requirement the mutant violated, not tests that merely kill the mutant.
• Prefer boundary, negative, error-path, and invariant tests over implementation-call assertions.
• Promote incident-related survived mutants into durable regression tests.
• Keep mutation runs deterministic; do not draw conclusions from flaky or failing baseline tests.
• Report excluded files and equivalent mutants so the score remains interpretable.

Examples

Packages And Libraries

Source Anchors

• Mutation-testing research at large scale treats mutation score as a test-effectiveness signal, not a direct quality score.
• Equivalent mutants, generated code, snapshots, and low-risk glue are known sources of noisy mutation results.
• Mutation testing is most useful when paired with a deterministic baseline suite and risk-scoped targets.

Quality Bar

• Mutation scope is tied to risk or changed behavior.
• Surviving mutants are triaged as meaningful, equivalent, or accepted risk.
• New tests would fail for the survived mutant before the fix.
• Reports include scope, command, baseline status, and runtime cost.

Tools

Mutation testing tools deliberately change production code and then rerun tests to find weak assertions. Choose by runtime, start with a narrow target package, and run it in CI or nightly jobs only after the normal test suite is stable and reasonably fast.

PIT / PITest

• Use for: JVM mutation testing for Java and other JVM projects.
• Languages/ecosystem: Java/JVM; Maven, Gradle, JUnit, and TestNG.
• Why it is trusted: PIT is the standard JVM mutation-testing tool and documents scoped targets, reports, history, and build-failing thresholds.
• Official docs: https://pitest.org/quickstart/maven/
• Good usage pattern:

<!-- pom.xml -->
<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <configuration>
    <targetClasses>
      <param>com.acme.billing.service*</param>
    </targetClasses>
    <targetTests>
      <param>com.acme.billing*</param>
    </targetTests>
    <mutationThreshold>75</mutationThreshold>
    <coverageThreshold>80</coverageThreshold>
    <outputFormats>
      <param>HTML</param>
      <param>XML</param>
    </outputFormats>
    <withHistory>true</withHistory>
  </configuration>
</plugin>

mvn test-compile org.pitest:pitest-maven:mutationCoverage

StrykerJS

• Use for: JavaScript and TypeScript mutation testing with Jest, Vitest, Mocha, Karma, and related runners.
• Languages/ecosystem: JavaScript, TypeScript, Node.js, frontend and backend test suites.
• Why it is trusted: StrykerJS documents runner integrations, mutation score thresholds, incremental mode, and HTML/CI reports.
• Official docs: https://stryker-mutator.io/docs/stryker-js/introduction/
• Good usage pattern:

// stryker.conf.json
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "packageManager": "npm",
  "testRunner": "vitest",
  "mutate": ["src/**/*.ts", "!src/**/*.d.ts"],
  "reporters": ["html", "clear-text", "progress"],
  "thresholds": {
    "high": 80,
    "low": 70,
    "break": 70
  }
}

npx stryker run

mutmut

• Use for: Python mutation testing with pytest-centered workflows.
• Languages/ecosystem: Python, pytest.
• Why it is trusted: mutmut documents incremental runs, pytest integration, configuration in pyproject.toml, and an interactive result browser.
• Official docs: https://mutmut.readthedocs.io/en/latest/
• Good usage pattern:

# pyproject.toml
[tool.mutmut]
paths_to_mutate = ["src/"]
pytest_add_cli_args_test_selection = ["tests/"]

mutmut run
mutmut browse

cargo-mutants

• Use for: Rust mutation testing that works with ordinary Cargo projects.
• Languages/ecosystem: Rust / Cargo.
• Why it is trusted: cargo-mutants documents scoped mutation runs, timeouts, workspace support, and JSON/text output for automation.
• Official docs: https://mutants.rs/
• Good usage pattern:

cargo mutants \
  --in-place \
  --package billing-core \
  --timeout 60 \
  --output target/mutants

Stryker.NET

• Use for: .NET mutation testing in C# projects.
• Languages/ecosystem: .NET, C#, test projects using common .NET test runners.
• Why it is trusted: Stryker.NET documents project selection, threshold gates, reporter options, and CI-friendly command-line usage.
• Official docs: https://stryker-mutator.io/docs/stryker-net/introduction/
• Good usage pattern:

dotnet stryker \
  --project src/Billing/Billing.csproj \
  --test-project tests/Billing.Tests/Billing.Tests.csproj \
  --threshold-high 80 \
  --threshold-low 70 \
  --threshold-break 70 \
  --reporter html

Performance, Load, and Stress Testing

Strategy Map

Purpose

Validate latency, throughput, saturation, scalability, and overload behavior under realistic workloads and environments.

Reliability Goal

Reduce production risk from performance regressions, capacity limits, queue buildup, resource exhaustion, and degraded user experience that functional tests cannot expose.

When This Strategy Applies

• A change affects request paths, database queries, caching, concurrency, background jobs, payload size, algorithms, infrastructure, or release capacity.
• SLOs, latency budgets, throughput targets, or launch-readiness thresholds exist.
• The workload can be modeled with representative users, data, traffic mix, and dependency behavior.
• Overload or graceful degradation behavior matters.

When This Strategy Does Not Apply

• No baseline, workload model, or success criteria exists.
• The environment is so unrealistic that results would mislead.
• The risk is functional correctness better covered by unit/integration tests.
• Load would hit systems without authorization or safe isolation.
• The test cannot observe bottlenecks or user-visible outcomes.

Signals To Inspect First

• SLOs, p95/p99 latency, throughput, error budgets, concurrency, traffic mix, payload sizes, cache hit rates, data volume, DB indexes, queue depth, CPU/memory/IO, autoscaling, dependency limits, and production telemetry.

Test Design Principles

• Performance tests are models; realism and baselines determine value.
• Measure user-visible outcomes and saturation signals, not only average latency.
• Small focused benchmarks catch algorithmic regressions; load tests catch system capacity and interactions.
• Stress tests should define acceptable degradation and recovery.
• Do not hide correctness errors inside performance runs.

Good Test Characteristics

• Workloads reflect real request mixes, think times, data sizes, auth paths, cache states, and dependency behavior.
• Assertions include latency percentiles, error rates, resource saturation, queue depth, and recovery.
• Baselines compare before/after under similar conditions.
• Results include enough telemetry to locate bottlenecks.
• Tests are scoped to safe environments with cleanup.

Poor Test Characteristics

• A synthetic benchmark with unrealistic data is used as launch proof.
• Only average latency is reported.
• Load generation overwhelms dependencies unrelated to the change.
• No correctness assertions run during load.
• Performance claims lack environment, version, workload, or baseline details.

Execution Pattern

• Define the performance risk and success criteria.
• Choose benchmark, load, stress, soak, or capacity test level.
• Build representative data and workload.
• Run a baseline if possible.
• Execute with observability enabled.
• Analyze percentiles, errors, resource saturation, and correctness.
• Rerun after fixes and report environment and residual risk.

Examples

• Weak: call one endpoint in a tight loop with one user and average latency. Stronger: replay representative authenticated traffic mix with realistic payloads, assert p95/p99 and error budget, and inspect DB/cache/queue metrics.
• Weak: microbenchmark a cache change only. Stronger: pair a focused benchmark with an integration load test that verifies cache hit behavior and backend saturation.

Validation

• Run the configured performance command or benchmark with documented environment.
• Compare against baseline and thresholds.
• Verify functional correctness during the run.
• Inspect telemetry for saturation and bottlenecks.
• Repeat suspicious results to rule out environmental noise.
• State limits of the workload model.

Failure Modes

• Unrepresentative workloads create false confidence.
• Shared environments add noise or harm other users.
• Averages hide tail latency.
• Missing telemetry prevents diagnosis.
• Optimizations change behavior or weaken tests.

Overview

Performance, load, and stress testing exercise a system under controlled traffic to learn whether it meets latency, throughput, error-rate, and saturation expectations. In practice: performance testing measures behavior against performance goals, load testing validates expected or increasing production-like demand, and stress testing pushes beyond expected demand or with constrained resources to expose breaking points and recovery behavior. These definitions align with ISTQB terminology.

The reliability problem this solves is not “is the code functionally correct?” but “does the whole service remain useful when queues, caches, databases, networks, autoscalers, and dependencies interact under pressure?” Google SRE frames stress testing as a way to quantify confidence in systems at scale, not just individual components.

Best Fit

Highest ROI comes when the system has user-visible latency or availability SLOs, real traffic growth, costly outages, expensive infrastructure, autoscaling behavior, shared downstream dependencies, or business-critical launch events.

Use it before major releases, migrations, pricing or traffic-model changes, infrastructure resizing, large customer onboarding, regional failover work, cache strategy changes, or changes to concurrency limits, queueing, retries, batching, rate limiting, or database indexes.

It is especially useful when paired with production observability, because load-test results without server-side metrics mostly say “it got slower,” not “why.”

Good Candidates

• Public APIs, internal platform APIs, service meshes, gateways, and edge services.
• Search, checkout, payments, authentication, file upload, streaming, messaging, and notification flows.
• Batch pipelines where throughput, queue depth, backlog drain time, or memory growth matters.
• Systems using autoscaling, connection pools, worker pools, queues, caches, or rate limits.
• Multi-tenant systems where noisy-neighbor effects or shared limits can degrade other customers.
• Launch readiness: “Can we handle 3x expected peak for 30 minutes and recover without manual intervention?”
• Regression gates for mature services: “p95 latency under 300 ms and error rate under 1% at 1,000 RPS.”

When Not To Use

Do not start with load testing when the team lacks basic production metrics, clear SLOs, representative traffic shape, or a realistic test environment. The result will usually be misleading.

Avoid expensive full-system tests for small, CPU-bound functions where microbenchmarks or profiling give a faster answer.

Do not use synthetic load as proof that production will be safe if real users have different think times, request mixes, payload sizes, auth paths, cache hit rates, regions, or dependency behavior.

Do not run uncontrolled stress tests against shared production dependencies unless blast radius, rate limits, rollback, and stakeholder communication are explicit.

Limitations

Load tests are models, not reality. The hardest parts are workload realism, dependency realism, data realism, and interpreting bottlenecks. AWS recommends production-like environments and synthetic or sanitized production data for cloud workload load testing; this is often the difference between useful signal and false confidence.

A test can overload the load generator before the service. Always monitor generator CPU, network, open files, connection reuse, DNS behavior, and outbound bandwidth.

Average latency is usually the wrong decision metric. Tail latency matters because distributed services often amplify rare slow events across many subrequests; Google’s “The Tail at Scale” remains authoritative because it describes this behavior from large production systems and is still cited in latency engineering.

Beware coordinated omission: a load generator that waits for slow responses before sending more traffic can under-report user-experienced latency during stalls. wrk2 was created specifically to address this by measuring against intended request timing.

Stress tests can cause cascading failures if retries, queues, autoscaling, or partial dependency failures create positive feedback loops; Google SRE documents overload-driven cascades as a common distributed-systems failure mode.

Signals

Helpful Signals

• Test scenarios map to named SLOs, user journeys, and expected peak/soak/spike profiles.
• Results include p50/p90/p95/p99 latency, throughput, error rate, saturation, queue depth, retry rate, GC pauses, connection-pool use, cache hit rate, database metrics, and dependency latency.
• The team can identify the bottleneck and the next scaling constraint.
• Repeated runs are comparable, versioned, and tied to changes in code, config, infrastructure, or data.
• The system degrades predictably: rate limits, backpressure, shedding, and alerts fire before total failure.

Misuse Signals

• Success is reported as “handled N users” without request rate, request mix, payloads, duration, or error budget.
• Only client-side averages are reviewed.
• The test passes only because caches are warm, auth is bypassed, data is tiny, or dependencies are mocked unrealistically.
• Load tests run rarely, require heroic setup, or are ignored unless a launch fails.
• Stress tests find breaking points but no one changes capacity plans, limits, dashboards, or runbooks.

Examples

A checkout team defines a pre-holiday test: 2x normal peak for 20 minutes, then a 5x spike for 3 minutes. Pass criteria: p95 checkout API latency under 500 ms, payment authorization errors under 0.5%, no queue backlog older than 2 minutes, and no database CPU above 80% for more than 5 minutes.

A platform API team adds a CI performance gate for one critical endpoint: run 10 minutes at the previous release’s p95 traffic, fail if p99 latency regresses by more than 20% or error rate exceeds 1%. k6 thresholds are a common way to express pass/fail criteria on metrics such as request failure rate and percentile latency.

An SRE team runs a controlled stress test in staging with one dependency rate-limited to production quota. The goal is not maximum RPS; it is verifying backpressure, retry budgets, autoscaling behavior, alert timing, and recovery after load returns to normal.

Packages And Libraries

General HTTP/API: Apache JMeter, Grafana k6, Gatling, Locust. JMeter is mature and broad; k6 is code-oriented and strong for threshold-based automation; Gatling is strong for code-defined simulations and workload modeling; Locust is Python-native and supports distributed load generation.

High-throughput benchmarking: wrk2 for constant-throughput HTTP benchmarking and coordinated-omission-aware latency measurement. Use carefully; it is better for focused endpoint benchmarking than complex user journeys.

Cloud/provider services: Azure Load Testing when the team wants managed load generation, JMeter compatibility, and Azure resource-metric integration. AWS Well-Architected guidance is useful even when using open-source tools: run sustained tests, discover breaking points, and model production scale.

Tool choice is ecosystem-specific; the durable practice is representative workload modeling, clear pass/fail thresholds, server-side observability, and repeatable comparison.

Tools

For Performance, Load, and Stress Testing, the useful tool types are: code-driven scenario engines for realistic user flows, protocol-oriented runners for broad backend coverage, and compact fixed-rate generators for quick capacity probes. Choose by workload model, protocol support, scripting language, CI pass/fail thresholds, and whether you need distributed load generation or only repeatable local regression checks.

Grafana k6

• Use for: API and service load tests as code with CI-friendly thresholds.
• Languages/ecosystem: Go engine; JavaScript/TypeScript test scripts; Grafana/Prometheus-friendly.
• Why it is trusted: Official docs describe k6 as open source, Go-based, scriptable in JavaScript/TypeScript, and designed around checks plus thresholds for automation.
• Official docs: https://grafana.com/docs/k6/latest/
• Good usage pattern:

import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
  scenarios: {
    steady_api_load: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: 20 },
        { duration: '2m', target: 20 },
        { duration: '30s', target: 0 },
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<400'],
    checks: ['rate>0.95'],
  },
};
export default function () {
  const baseUrl = __ENV.BASE_URL || 'https://staging.example.com';
  const res = http.get(`${baseUrl}/api/products?limit=20`);
  check(res, {
    'status is 200': (r) => r.status === 200,
    'returns JSON': (r) =>
      String(r.headers['Content-Type']).includes('application/json'),
  });
  sleep(1);
}

Apache JMeter

• Use for: Broad protocol coverage, GUI-assisted test-plan creation, and headless load execution.
• Languages/ecosystem: Java/JVM; JMX test plans; Groovy/JSR223 scripting; Maven/Gradle/Jenkins integrations.
• Why it is trusted: Apache documents JMeter as open-source, pure Java, multi-protocol, extensible, and explicitly recommends CLI mode for load testing.
• Official docs: https://jmeter.apache.org/usermanual/
• Good usage pattern:

# perf/orders.jmx contains:
# Thread Group + HTTP Request Defaults + HTTP Samplers + JSON/Response Assertions.
jmeter -n \
  -t perf/orders.jmx \
  -l build/jmeter/orders.jtl \
  -j build/jmeter/jmeter.log \
  -e -o build/jmeter/html \
  -Jbase_url=https://staging.example.com \
  -Jusers=200 \
  -Jramp_seconds=120 \
  -Jduration_seconds=600

Locust

• Use for: Python-coded user journeys with custom logic, fixtures, and distributed execution.
• Languages/ecosystem: Python; gevent-based concurrency; optional web UI or headless CLI.
• Why it is trusted: Locust docs cover headless execution, Python locustfiles, custom response validation, and distributed master/worker load generation.
• Official docs: https://docs.locust.io/en/stable/
• Good usage pattern:

# CI run:
# locust -f locustfile.py --headless --users 200 --spawn-rate 20 \
#   --run-time 10m --host https://staging.example.com
from locust import HttpUser, task, between
class ApiUser(HttpUser):
    wait_time = between(1, 3)
    @task(3)
    def browse_products(self):
        with self.client.get(
            "/api/products?limit=20",
            name="GET /api/products",
            catch_response=True,
        ) as resp:
            if resp.status_code != 200:
                resp.failure(f"unexpected status {resp.status_code}")
                return
            try:
                items = resp.json().get("items", [])
            except ValueError:
                resp.failure("invalid JSON")
                return
            if not items:
                resp.failure("empty product list")
    @task(1)
    def view_cart(self):
        self.client.get("/api/cart", name="GET /api/cart")

Gatling

• Use for: JVM-centered, code-driven HTTP load tests with precise injection models and post-run assertions.
• Languages/ecosystem: JVM; Java, JavaScript, TypeScript, Scala, and Kotlin SDKs.
• Why it is trusted: Gatling’s main project is Apache-2.0 licensed, and its docs define code SDKs, open/closed workload injection, checks, and assertions that fail simulations when violated.
• Official docs: https://docs.gatling.io/
• Good usage pattern:

import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;
import java.time.Duration;
import io.gatling.javaapi.core.*;
import io.gatling.javaapi.http.*;
public class OrdersSimulation extends Simulation {
  HttpProtocolBuilder httpProtocol = http
      .baseUrl(System.getProperty("baseUrl", "https://staging.example.com"))
      .acceptHeader("application/json");
  ScenarioBuilder scn = scenario("orders-api")
      .exec(http("list orders")
          .get("/api/orders?limit=20")
          .check(status().is(200)));
  {
    setUp(
        scn.injectOpen(
            rampUsersPerSec(1).to(25).during(Duration.ofSeconds(60)),
            constantUsersPerSec(25).during(Duration.ofMinutes(3))
        ).protocols(httpProtocol)
    ).assertions(
        global().responseTime().percentile3().lt(500),
        forAll().failedRequests().percent().lte(1.0)
    );
  }
}

Vegeta

• Use for: Constant-rate HTTP load probes, capacity checks, and small CI performance gates.
• Languages/ecosystem: Go CLI and Go library; UNIX pipeline style.
• Why it is trusted: Official docs describe Vegeta as a CLI/library with constant-rate attacks, UNIX composability, reporting, distributed use, and JSON report output.
• Official docs: https://github.com/tsenart/vegeta
• Good usage pattern:

mkdir -p build/vegeta
cat > targets.http <<EOF
GET https://staging.example.com/api/products?limit=20
EOF
vegeta attack \
  -rate=50/s \
  -duration=5m \
  -name=products \
  -targets=targets.http \
  | tee build/vegeta/products.bin \
  | vegeta report
vegeta report -type=json build/vegeta/products.bin > build/vegeta/products.json
jq -e \
  '.success >= 0.99 and .latencies["95th"] < 400000000' \
  build/vegeta/products.json

Tools

For Property-Based Testing, the useful tool capabilities are generators/arbitraries, repeated randomized checks, shrinking to minimal counterexamples, deterministic replay, and integration with the project’s normal test runner. Choose primarily by language and runner fit; choose secondarily by how well the library supports custom generators, CI run-count controls, and stateful/model-based tests. PBT libraries commonly separate generated inputs from assertions and are usually used alongside ordinary unit tests, not instead of them.

Hypothesis

• Use for: Python property tests in pytest/unittest suites, especially for data validation, parsers, serializers, and numeric edge cases.
• Languages/ecosystem: Python; commonly paired with pytest.
• Why it is trusted: Hypothesis documents per-test settings, suite profiles, environment-selected profiles, strategies, and shrinking/replay behavior.
• Official docs: https://hypothesis.readthedocs.io/en/latest/
• Good usage pattern:

from hypothesis import given, settings, strategies as st
from app.tags import normalize_tags
tag_lists = st.lists(st.text(min_size=0, max_size=30), max_size=50)
@given(tag_lists)
@settings(max_examples=300, deadline=None)
def test_normalize_tags_is_idempotent(raw_tags):
    once = normalize_tags(raw_tags)
    assert normalize_tags(once) == once
    assert once == sorted(set(once))
    assert all(t and t == t.strip().lower() for t in once)

fast-check

• Use for: JavaScript/TypeScript property tests in Jest, Vitest, Mocha, Node, or browser-oriented test suites.
• Languages/ecosystem: JavaScript and TypeScript; test-runner agnostic.
• Why it is trusted: Its docs cover arbitraries, seeded deterministic runs, configurable run counts, shrinking, and compatibility with major JS/TS test frameworks.
• Official docs: https://fast-check.dev/docs/introduction/getting-started/
• Good usage pattern:

import fc from 'fast-check';
import { expect, test } from 'vitest';
import { parseMoneyCents } from './money';
fc.configureGlobal({ numRuns: Number(process.env.FC_NUM_RUNS ?? 250) });
test('parseMoneyCents round-trips formatted cents', () => {
  fc.assert(
    fc.property(fc.integer({ min: 0, max: 1_000_000 }), (cents) => {
      const text = `$${(cents / 100).toFixed(2)}`;
      expect(parseMoneyCents(text)).toBe(cents);
    }),
  );
});

jqwik

• Use for: JVM property tests that should run naturally on the JUnit Platform.
• Languages/ecosystem: Java, Kotlin, and other JVM languages; pair with JUnit/AssertJ assertions as needed.
• Why it is trusted: The user guide documents @Property, @ForAll, default 1000 tries, shrinking, edge-case generation, property defaults, and JUnit Platform dependencies.
• Official docs: https://jqwik.net/docs/current/user-guide.html
• Good usage pattern:

import net.jqwik.api.ForAll;
import net.jqwik.api.Property;
import net.jqwik.api.constraints.IntRange;
import org.junit.jupiter.api.Assertions;
class MoneyProperties {
    @Property(tries = 500)
    void parseCentsRoundTrips(
        @ForAll @IntRange(min = 0, max = 1_000_000) int cents
    ) {
        String text = String.format("$%d.%02d", cents / 100, cents % 100);
        Assertions.assertEquals(cents, Money.parseCents(text));
    }
}

proptest

• Use for: Rust property tests where explicit strategies, shrinking, and regression replay should run under cargo test.
• Languages/ecosystem: Rust; pair with normal unit/integration tests.
• Why it is trusted: The crate docs expose the core strategy, assertion, and proptest! APIs; the official book documents configurable case counts and persisted regression seeds for CI replay.
• Official docs: https://proptest-rs.github.io/proptest/proptest/index.html
• Good usage pattern:

use proptest::prelude::*;
fn normalize_ids(mut ids: Vec<u32>) -> Vec<u32> {
    ids.sort_unstable();
    ids.dedup();
    ids
}
proptest! {
    #![proptest_config(ProptestConfig::with_cases(512))]
    #[test]
    fn normalize_ids_is_idempotent(
        ids in prop::collection::vec(0u32..10_000, 0..200)
    ) {
        let once = normalize_ids(ids);
        let twice = normalize_ids(once.clone());
        prop_assert_eq!(&once, &twice);
        prop_assert!(once.windows(2).all(|w| w[0] < w[1]));
    }
}

FsCheck

• Use for: .NET property tests, especially F# or C# suites that already use xUnit or NUnit.
• Languages/ecosystem: .NET; F#, C#, VB; integrates with xUnit and NUnit.
• Why it is trusted: The docs cover built-in runners, QuickCheckThrowOnFailure, xUnit/NUnit integrations, configurable MaxTest, shrinking output, and replay seeds.
• Official docs: https://fscheck.github.io/FsCheck/
• Good usage pattern:

using System.Linq;
using FsCheck;
using Xunit;
public class SlugProperties
{
    [Fact]
    public void NormalizeSlugIsIdempotent()
    {
        var config = Config.QuickThrowOnFailure
            .WithMaxTest(300)
            .WithQuietOnSuccess(true);
        Prop.ForAll<string>(input =>
        {
            var once = Slug.Normalize(input ?? "");
            return Slug.Normalize(once) == once
                && once.All(c =>
                    c == '-' || ('a' <= c && c <= 'z') || ('0' <= c && c <= '9'));
        }).Check(config);
    }
}

Tools

Regression Test Selection / Test Impact Analysis tools usually work at one of three layers: build graph selection, test-task caching, or test-framework-level affected-test selection. Choose build-graph tools for monorepos, framework selectors for single-stack repos, and coverage/runtime selectors when static dependency graphs are too coarse. Keep a scheduled full-suite run as a backstop.

Bazel

• Use for: Multi-language monorepos where impacted tests can be selected from reverse dependencies in the build graph.
• Languages/ecosystem: Java/JVM, C/C++, Go, Python, JavaScript, Rust, Android, iOS, and rule-based extensions.
• Why it is trusted: Bazel’s docs emphasize optimized dependency analysis, caching, multi-language support, and use in large production codebases.
• Official docs: https://bazel.build/query/language
• Good usage pattern:

# Run tests that reverse-depend on a changed library target.
impacted="$(bazel query 'tests(rdeps(//..., //lib/payments:payments))')"
test -z "$impacted" || bazel test $impacted

Gradle

• Use for: JVM and Android projects where unchanged test tasks should be skipped or restored from cache.
• Languages/ecosystem: Java, Kotlin, Scala, Groovy, Android, and JVM plugin ecosystems.
• Why it is trusted: Gradle has built-in incremental up-to-date checks, build cache support, and a cacheable built-in Test task.
• Official docs: https://docs.gradle.org/current/userguide/build_cache.html
• Good usage pattern:

# Preserve Gradle User Home between CI runs so test outputs can be reused.
./gradlew test --build-cache --fail-fast

Nx

• Use for: JavaScript/TypeScript monorepos where CI should run tests only for projects affected by a PR.
• Languages/ecosystem: JavaScript, TypeScript, Angular, React, Node, Vite, Jest, Vitest, Playwright, and Nx plugins.
• Why it is trusted: Nx’s affected command uses Git plus the project graph to find changed projects and their dependents before running tasks.
• Official docs: https://nx.dev/docs/features/ci-features/affected
• Good usage pattern:

# GitHub Actions step
- run: git fetch origin main --depth=1
- run: npx nx affected -t test --base=origin/main --head=HEAD

Jest

• Use for: JavaScript/TypeScript unit tests where changed source files can be mapped to related test files.
• Languages/ecosystem: JavaScript, TypeScript, Node, React, frontend and backend unit tests.
• Why it is trusted: Jest’s official CLI supports --findRelatedTests, --changedSince, --onlyChanged, --ci, and --passWithNoTests.
• Official docs: https://jestjs.io/docs/cli
• Good usage pattern:

mapfile -t changed < <(
  git diff --name-only --diff-filter=ACMR origin/main...HEAD -- 'src/**/*.ts' 'src/**/*.tsx'
)
((${#changed[@]} == 0)) || npx jest --ci --findRelatedTests "${changed[@]}" --passWithNoTests

pytest-testmon

• Use for: Python pytest suites that need test-level selection based on observed code dependencies.
• Languages/ecosystem: Python, pytest; internally uses Coverage.py dependency data.
• Why it is trusted: The official docs describe collecting dependencies between tests and executed code, storing .testmondata, and rerunning only affected tests after changes.
• Official docs: https://www.testmon.org/
• Good usage pattern:

# Cache .cache/testmon/.testmondata between CI runs.
mkdir -p .cache/testmon
TESTMON_DATAFILE=.cache/testmon/.testmondata pytest --testmon --testmon-env=ci