Acceptance Pipeline Catalog

Robert C. Martin Acceptance Pipeline Best Practices

Language-neutral specification for a portable acceptance-test pipeline: Gherkin feature files to JSON IR to generated acceptance tests to mutation testing. Based on Robert C. Martin's Acceptance Pipeline Specification. Contains ~50 rules across 14 categories, prioritized by impact.

When to Apply

Reference these rules when:

• Building a Gherkin parser that outputs JSON IR
• Implementing an acceptance test generator from JSON IR
• Writing an acceptance runtime that expands scenarios and dispatches steps
• Implementing mutation testing for acceptance test example values
• Setting up the full pipeline (parser, generator, runner, mutator) in a new project
• Debugging pipeline failures (parse errors, generation issues, mutation classification)

Pipeline Overview

The pipeline has two modes:

Normal acceptance run:

feature file -> gherkin parser -> JSON IR -> acceptance generator -> generated tests -> test runner

Mutation run:

feature file -> gherkin parser -> base JSON IR -> mutator (one changed IR per mutation)
  -> generator (tests per mutation) -> test runner (evaluate each) -> mutation report

The normal run proves the project satisfies the feature. The mutation run probes whether tests are strong enough to fail when example data changes.

Rule Categories by Priority

Quick Reference

1. Parser (CRITICAL)

• `parser-command-interface` - Two positional args, exit codes 0/1/2
• `parser-feature-declaration` - Feature: keyword required, trimmed name
• `parser-background` - Optional Background: section with Given/And steps
• `parser-scenarios` - Scenario: and Scenario Outline: both supported
• `parser-steps` - Given/When/Then/And keywords, keyword stored separately
• `parser-parameters` - Angle-bracket placeholders, not expanded by parser
• `parser-examples-tables` - Pipe-delimited tables, header row first
• `parser-general-rules` - Blank lines, comments, whitespace, ordering
• `parser-unsupported-syntax` - Tags, rules, localized keywords, doc strings

2. JSON IR (CRITICAL)

• `ir-feature-object` - name, scenarios, optional background
• `ir-scenario-object` - name, steps, examples arrays
• `ir-step-object` - keyword, text, optional parameters
• `ir-example-object` - String-keyed, string-valued maps

3. Generator (CRITICAL)

• `gen-command-interface` - Two positional args, exit codes 0/1/2
• `gen-requirements` - Embed IR, no Gherkin parsing, deterministic output

4. Runtime (HIGH)

• `rt-responsibilities` - Load IR, expand, dispatch, report
• `rt-scenario-expansion` - One execution per example row, background prepended
• `rt-execution-naming` - Scenario name / example index naming convention

5. Step Handlers (HIGH)

• `handler-matching` - Match by exact text value, not keyword
• `handler-world-state` - Fresh world/state per scenario execution
• `handler-value-handling` - Fetch, parse, fail on missing/malformed
• `handler-unsupported` - Unsupported step text must fail the test

6. Test Runner (HIGH)

• `runner-interface` - Input/output contract for the test runner adapter
• `runner-classification` - Three-way: failure, success, infrastructure error

7. Mutator Core (HIGH)

• `mut-command-interface` - CLI options, exit codes 0/1/2
• `mut-scope` - Only example cell values mutated
• `mut-identity` - Stable deterministic IDs, paths, descriptions
• `mut-deep-copy` - Original IR never modified in place

8. Value Mutation Rules (HIGH)

• `val-rule-order` - 8 rules applied in priority order
• `val-comma-list` - Comma-delimited list mutation
• `val-boolean` - true/false toggle
• `val-null` - null/nil/none to dithered string
• `val-integer` - Integer plus pseudo-random delta
• `val-float` - Float plus pseudo-random delta
• `val-datetime` - ISO-8601 date/time shift
• `val-duration` - Duration shift preserving syntax
• `val-string-dither` - Character-level string edits
• `val-determinism` - Pseudo-random, deterministic for fixed input

9. Result Classification (HIGH)

• `result-statuses` - killed, survived, error
• `result-classification-rules` - Mapping from test outcomes to statuses

10. Conformance (HIGH)

• `conform-checklist` - All 21 validation items

11. Agent Setup (HIGH)

• `setup-checklist` - 15-step installation guide

12. Mutation Execution (MEDIUM)

• `exec-work-directory` - Per-mutation directory structure
• `exec-workflow` - Write IR, generate, run, classify
• `exec-parallelism` - Concurrent workers, isolated directories
• `exec-timeout` - Full-run timeout, unfinished = error

13. Reports (MEDIUM)

• `report-text-format` - Summary line + per-result lines
• `report-json-format` - JSON object with summary and results array
• `report-field-requirements` - Required fields for summary and results

14. Project Layout (MEDIUM)

• `layout-required-paths` - features/, build/, acceptance/ directories
• `layout-commands` - gherkin-parser, acceptance-generator, gherkin-mutator
• `layout-scripts` - Normal acceptance and mutation scripts

How to Use

Read individual reference files for detailed spec requirements and rationale:

• Start with the category relevant to the component you are building
• Each rule file is self-contained with WHY explanations, spec requirements, and examples
• For a new project setup, read `setup-checklist` first
• For validation, use `conform-checklist`
• Check gotchas.md for known failure points

Reference Files

Gotchas

No gotchas recorded yet.

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

---

1. Parser (parser)

Impact: CRITICAL Description: The Gherkin parser converts feature files into JSON IR. Parser errors propagate to every downstream component, making correct parsing the foundation of the entire pipeline.

2. JSON IR (ir)

Impact: CRITICAL Description: The JSON Intermediate Representation is the canonical interchange format consumed by generator, runtime, and mutator. Schema errors break all consumers simultaneously.

3. Generator (gen)

Impact: CRITICAL Description: The acceptance generator converts JSON IR into executable tests. Generator correctness determines whether tests actually exercise the specified behavior.

4. Runtime (rt)

Impact: HIGH Description: The acceptance runtime expands scenarios, applies backgrounds, resolves placeholders, and dispatches steps to handlers. Runtime errors cause silent test gaps.

5. Step Handlers (handler)

Impact: HIGH Description: Step handlers bind Gherkin step text to project behavior and assertions. Handler contract violations produce false passes or misleading failures.

6. Test Runner (runner)

Impact: HIGH Description: The test runner adapter executes generated tests and classifies outcomes. Misclassification (confusing infrastructure errors with test failures) corrupts mutation results.

7. Mutator Core (mut)

Impact: HIGH Description: The mutator creates deterministic example-value mutations to probe acceptance test strength. Scope and identity errors make mutation results unreproducible or misleading.

8. Value Mutation Rules (val)

Impact: HIGH Description: Eight type-inference rules applied in priority order determine how example values are mutated. Rule ordering and determinism are essential for reproducible mutation runs.

9. Result Classification (result)

Impact: HIGH Description: Result classification maps test outcomes to killed/survived/error statuses. Misclassification directly corrupts the mutation score and misleads developers about test quality.

10. Conformance (conform)

Impact: HIGH Description: Twenty-one testable conformance items validate a complete pipeline implementation. Missing conformance coverage allows non-portable implementations.

11. Agent Setup (setup)

Impact: HIGH Description: The fifteen-step installation checklist guides agents through setting up the pipeline in a new project. Order matters — later steps depend on earlier ones.

12. Mutation Execution (exec)

Impact: MEDIUM Description: Mutation execution manages work directories, parallelism, and timeouts for running mutated tests. Isolation failures between workers corrupt results.

13. Reports (report)

Impact: MEDIUM Description: Text and JSON reports communicate mutation results to developers and CI systems. Format errors break downstream tooling and make results unactionable.

14. Project Layout (layout)

Impact: MEDIUM Description: Required paths and convenience scripts establish the project structure. Layout errors cause pipeline commands to fail on missing directories or mismatched paths.

Conformance Checklist

A conforming implementation can be validated with these 21 cases. Each item targets a specific requirement from the spec. Use this checklist to verify that a pipeline implementation is complete and correct.

Parser Conformance (items 1-5)

1. Parser accepts supported syntax. Parser accepts Feature:, Background:, Scenario:, Scenario Outline:, supported step keywords (Given, When, Then, And), parameter placeholders (<name>), and examples tables.

2. Parser writes correct IR shape. Parser output matches the JSON IR structure defined in the spec (feature object with name, scenarios array, optional background array; scenario objects with name, steps, examples; step objects with keyword, text, optional parameters).

3. Parser rejects missing feature. A file with no Feature: declaration produces exit code 1.

4. Parser rejects orphan examples. An Examples: section outside a scenario produces an error.

5. Parser rejects cell count mismatch. An examples data row with a different cell count than the header row produces exit code 1.

Generator Conformance (items 6-7)

6. Generator produces deterministic output. Running the generator twice on the same IR produces identical output files.

7. Generated tests execute the IR. Generated tests run the scenarios and examples from the IR they were generated from, not from any other source.

Runtime Conformance (items 8-11)

8. Runtime applies backgrounds. Background steps are prepended to every scenario execution.

9. Runtime handles empty examples. Scenarios without examples execute once with an empty example object.

10. Runtime fails unsupported steps. A step whose text matches no registered handler fails the current test.

11. Runtime fails invalid values. Missing or malformed example values fail the current test.

Script Conformance (item 12)

12. Script propagates failures. The normal acceptance script fails if parsing, generation, or generated tests fail.

Mutator Conformance (items 13-21)

13. Mutator targets only example values. Mutations are generated only for example cell values — not feature names, scenario names, step text, keywords, backgrounds, or headers.

14. Mutator produces stable identities. Mutation IDs (m1, m2, ...), paths ($.scenarios[i].examples[j].key), and descriptions are deterministic for a fixed IR.

15. Mutator applies all value rules. The mutator correctly applies comma-list, boolean, null-like, integer, floating-point, date/time, duration, and string-dithering mutation rules in priority order.

16. Mutator deep-copies IR. Each mutation is applied to a deep copy; the original IR is never modified.

17. Mutator classifies killed correctly. Failing generated tests result in killed status.

18. Mutator classifies survived correctly. Passing generated tests result in survived status.

19. Mutator classifies errors correctly. Parsing, generation, timeout, and infrastructure failures result in error status.

20. Mutator exit code reflects results. Exit code 1 when any mutation survived or errored; exit code 0 only when all killed.

21. Mutator emits stable reports. Text and JSON reports are emitted in stable order (by mutation ID) with all required fields.

How to Use This Checklist

Work through items sequentially. Items 1-5 verify the parser foundation. Items 6-7 verify the generator. Items 8-11 verify runtime behavior. Item 12 verifies script integration. Items 13-21 verify the mutation pipeline.

Each item can be validated with a focused test: create a minimal feature file that exercises the specific behavior, run the pipeline, and verify the expected outcome.

Examples

Incorrect (item 5 -- parser silently accepts mismatched cell counts):

Feature: Shopping Cart
  Scenario Outline: Add item
    When I add <item>
    Then cart total is <total>

    Examples:
      | item   | total |
      | apple  | 1.50  | 0.00 |

# Parser exits 0 despite 3 cells in data row vs 2 in header
gherkin-parser features/cart.feature build/acceptance/cart.json
echo $?  # 0 (should be 1)

Correct (item 5 -- parser rejects cell count mismatch with exit code 1):

Feature: Shopping Cart
  Scenario Outline: Add item
    When I add <item>
    Then cart total is <total>

    Examples:
      | item   | total |
      | apple  | 1.50  | 0.00 |

# Parser detects 3 cells vs 2 headers and exits with error
gherkin-parser features/cart.feature build/acceptance/cart.json
echo $?  # 1
# stderr: "line 8: expected 2 cells, got 3"

Parallel Execution

Mutations can run concurrently to reduce total mutation testing time. The per-mutation directory structure enables safe parallelism without locks, as long as each mutation stays within its own directory.

Spec Requirements

• Workers may run concurrently. The --workers flag controls the maximum number of parallel mutation workers.
• Each mutation must write only inside its own directory (<work-dir>/<mutation-id>/).
• Values less than 1 for --workers must be treated as 1 (sequential execution).

Why Directory Isolation Enables Parallelism

Each mutation has its own feature.json and generated/ directory. Two workers processing m1 and m2 simultaneously never touch the same files. This eliminates race conditions without file locking, mutexes, or coordination between workers.

What Must Not Be Shared

• The base IR must not be modified (per the deep-copy requirement). Workers read from it but never write to it.
• The work directory root must not have files that workers compete to write.
• Standard output/error should be captured per-mutation, not written to a shared console during execution (reports are emitted after all mutations complete).

Examples

Incorrect (shared work directory -- workers overwrite each other's files):

# Worker 1 and Worker 2 both write to the same directory
# Worker 1:
cp mutated-m1.json build/acceptance-mutation/feature.json
# Worker 2 overwrites before Worker 1 reads:
cp mutated-m2.json build/acceptance-mutation/feature.json
# Worker 1 now generates tests from m2's IR, not m1's

Correct (per-mutation directories -- workers are fully isolated):

# Worker 1 writes only to m1's directory
cp mutated-m1.json build/acceptance-mutation/m1/feature.json
acceptance-generator build/acceptance-mutation/m1/feature.json \
  build/acceptance-mutation/m1/generated/a-feature_test.ext

# Worker 2 writes only to m2's directory (no conflict)
cp mutated-m2.json build/acceptance-mutation/m2/feature.json
acceptance-generator build/acceptance-mutation/m2/feature.json \
  build/acceptance-mutation/m2/generated/a-feature_test.ext

Why This Matters

Mutation testing can be slow — each mutation requires generating tests and running them. For a feature with 50 mutations and a test suite that takes 2 seconds per run, sequential execution takes 100 seconds. With 4 workers, it takes ~25 seconds. Parallel execution makes mutation testing practical for CI integration.

The --workers flag lets developers tune the trade-off between speed and resource usage based on their hardware and test suite characteristics.

Timeout Handling

The timeout applies to the entire mutation run, not to individual mutations. When time expires, unfinished mutations are reported as errors rather than silently dropped.

Spec Requirements

• The --timeout flag sets a timeout for the full mutation run.
• Duration syntax is implementation-defined but should support seconds.
• When the timeout expires, unfinished mutations should be reported as error with useful timeout text.

Why Full-Run Timeout

A per-mutation timeout would require knowing how long each mutation should take — which varies by project, test suite size, and system load. A full-run timeout is simpler: the developer says "I have 5 minutes for mutation testing" and the mutator does as many mutations as it can in that time.

Why Error, Not Skip

Unfinished mutations are classified as error (not silently omitted) because: 1. The report's total count must be accurate — omitting mutations would undercount. 2. The exit code must be 1 (not 0) when errors exist — an incomplete run should not be reported as "all killed." 3. The developer needs to know which mutations were not evaluated so they can increase the timeout or reduce the mutation set.

Example Report Entry

error    $.scenarios[2].examples[0].value: hello -> hallo
  error: mutation timed out after 300s

Examples

Incorrect (no timeout -- pipeline hangs on a stuck mutation):

# No --timeout flag; mutation m3 hangs forever
gherkin-mutator --feature features/a-feature.feature --workers 2
# CI kills the entire job after 30 minutes -- all results lost

Correct (full-run timeout -- unfinished mutations reported as errors):

gherkin-mutator --feature features/a-feature.feature --workers 2 --timeout 300s
# After 300s, unfinished mutations are classified as error:
#   error    $.scenarios[2].examples[0].value: hello -> hallo
#     error: mutation timed out after 300s
# Completed mutations retain their real killed/survived status

Why This Matters

Without a timeout, a single slow test or a hanging process blocks the entire mutation pipeline. In CI, this means the build hangs until the CI system's own timeout kills it — losing all partial results. With a proper timeout, partial results are preserved and reported, and the developer gets actionable information about which mutations need investigation.

Work Directory Structure

Each mutation gets its own isolated work directory. This structure ensures mutations do not interfere with each other and makes cleanup straightforward.

Spec Requirements

For each mutation, create a work directory:

<work-dir>/<mutation-id>/

Write the mutated JSON IR to:

<work-dir>/<mutation-id>/feature.json

Place generated tests under:

<work-dir>/<mutation-id>/generated/

Example Layout

For --work-dir build/acceptance-mutation with mutations m1, m2, m3:

build/acceptance-mutation/
  m1/
    feature.json          # Mutated IR for m1
    generated/            # Generated tests from m1's IR
      a-feature_test.ext
  m2/
    feature.json
    generated/
      a-feature_test.ext
  m3/
    feature.json
    generated/
      a-feature_test.ext

Why Per-Mutation Directories

If multiple mutations shared a directory, concurrent workers would overwrite each other's feature.json and generated test files. Per-mutation directories provide natural isolation without file-locking complexity.

This structure also aids debugging: when a mutation survives, the developer can inspect build/acceptance-mutation/m3/feature.json to see exactly what the mutated IR looks like, and build/acceptance-mutation/m3/generated/ to see what tests were generated from it.

Examples

Incorrect (flat directory -- all mutations share a single directory):

build/acceptance-mutation/
  feature.json            # Overwritten by each mutation
  generated/
    a-feature_test.ext    # Overwritten by each mutation

Correct (per-mutation isolation -- each mutation gets its own subdirectory):

build/acceptance-mutation/
  m1/
    feature.json
    generated/
      a-feature_test.ext
  m2/
    feature.json
    generated/
      a-feature_test.ext

Why This Matters

The work directory is the mutation's entire execution context. The generator reads feature.json from it, writes generated tests into it, and the runner executes tests from it. If the directory structure is wrong, the pipeline components cannot find each other's outputs.

Mutation Execution Workflow

Each mutation follows a four-step workflow. The steps must execute in order because each depends on the previous step's output. Skipping or reordering steps produces incorrect results.

Spec Requirements

For each mutation:

1. Write the mutated IR to <work-dir>/<mutation-id>/feature.json. 2. Generate tests by invoking the acceptance generator with the mutated IR. Place generated tests under <work-dir>/<mutation-id>/generated/. 3. Run the generated tests using the test runner adapter. 4. Classify the result based on the runner's output (killed, survived, or error).

Why This Sequence

Each step feeds the next:

• Step 1 produces the IR that step 2 reads.
• Step 2 produces the tests that step 3 runs.
• Step 3 produces the outcome that step 4 classifies.

Reordering is not possible. Skipping step 2 (generating tests) and running old tests would test the base IR, not the mutation. Skipping step 3 (running tests) means no classification data.

Error Handling

If any of steps 1-3 fails, the mutation is classified as error in step 4. The specific failure (IR write failure, generation failure, runner failure) should be captured in the error text for the report.

This means the classify step always executes — it either classifies based on test results or based on the infrastructure failure.

Examples

Incorrect (skipping generation -- reuses stale tests from a prior run):

# Step 1: Write mutated IR
cp mutated-ir.json build/acceptance-mutation/m1/feature.json

# Step 2: SKIPPED -- reuses old generated tests

# Step 3: Run stale tests (tests reflect base IR, not the mutation)
run-tests acceptance/generated/

# Step 4: Classify -- result is wrong because tests don't reflect mutation

Correct (full four-step sequence -- regenerates tests from mutated IR):

# Step 1: Write mutated IR
cp mutated-ir.json build/acceptance-mutation/m1/feature.json

# Step 2: Generate tests from mutated IR
acceptance-generator build/acceptance-mutation/m1/feature.json \
  build/acceptance-mutation/m1/generated/a-feature_test.ext

# Step 3: Run freshly generated tests
run-tests build/acceptance-mutation/m1/generated/

# Step 4: Classify based on actual test outcome

Why This Matters

The workflow is the same for every mutation. The mutator orchestrates it, the generator and runner are invoked as tools. This separation means changing the generator or runner does not require changing the mutator's orchestration logic — only the tool invocation adapters change.

Enforce Generator Command Interface

The generator transforms JSON IR into executable acceptance tests. Like the parser, its command interface must be predictable so that scripts and the mutator can invoke it reliably.

Spec Requirements

The generator command accepts exactly two positional arguments:

acceptance-generator <json-ir> <generated-test-output>

• <json-ir> — path to the JSON IR file (produced by the parser).
• <generated-test-output> — path where the generated executable test file will be written.

Exit Codes

Why This Matters

The generator sits between the IR and the test runner. In the normal acceptance script, a generation failure must stop the pipeline (the script uses set -eu). In the mutation workflow, the mutator calls the generator once per mutation — a generation failure for a specific mutation classifies that mutation as error, not killed or survived.

The two-positional-arg interface mirrors the parser's interface, making the pipeline commands consistent and composable in scripts:

gherkin-parser features/a.feature build/acceptance/a.json
acceptance-generator build/acceptance/a.json acceptance/generated/a_test.ext

Examples

Incorrect (generator accepts flags instead of positional args, non-standard exit codes):

acceptance-generator --input build/a.json --output acceptance/a_test.py
# exit code 255 on generation failure

Correct (two positional args, exit codes 0/1/2 matching parser convention):

acceptance-generator build/acceptance/a.json acceptance/generated/a_test.py
# exit code 0: generation succeeded
# exit code 1: invalid IR or write failure
# exit code 2: wrong number of arguments

Follow Generator Requirements

The generator must produce executable tests that faithfully represent the JSON IR. These five requirements ensure generated tests are correct, reproducible, and independent of the original Gherkin source.

Spec Requirements

1. Generated tests must embed or load the JSON IR supplied to the generator. The tests must work from the IR, not from any other source.

2. Generated tests must not parse the source Gherkin file. The IR is the single source of truth for test generation. This ensures mutations applied to the IR are reflected in generated tests without re-parsing.

3. Generated tests must run every scenario execution represented by the IR. No scenario or example row may be silently skipped.

4. Generated tests must fail when the runtime reports an unsupported step, invalid example value, or failed assertion. Swallowing errors would hide pipeline problems.

5. The generated output must be deterministic for a fixed IR. Running the generator twice on the same IR must produce identical test files.

Why No Gherkin Parsing

This is a key architectural constraint. The mutation workflow modifies the IR, then asks the generator to produce tests from the modified IR. If the generator went back to the Gherkin source, mutations would have no effect — the generated tests would always reflect the original feature file, not the mutated IR.

Why Determinism

Deterministic output enables diffing generated tests across runs. When a mutation changes one example value, the diff between the original and mutated generated tests should show exactly that change. Non-deterministic generation (random variable names, timestamps in comments) would make such diffing impossible.

Why This Matters

The generated test format is implementation-specific — the spec does not dictate whether tests are Python, Go, Java, etc. But regardless of format, these five requirements ensure the tests serve their purpose: proving the project satisfies the specification (normal run) and detecting when mutated specifications go unnoticed (mutation run).

Examples

Incorrect (generated test re-parses Gherkin source, ignoring IR mutations):

{
  "generated_test_loads": "features/calculator.feature",
  "uses_parser": true,
  "embeds_ir": false
}

Correct (generated test embeds/loads the JSON IR directly):

{
  "generated_test_loads": "build/acceptance/calculator.json",
  "uses_parser": false,
  "embeds_ir": true
}

Match Steps by Exact Text

Step handlers connect Gherkin step text to project behavior. The matching strategy determines how a step like "the result is <result>" finds its handler. The spec defines exact text matching as the portable baseline.

Spec Requirements

The portable baseline matches handlers by exact `text` value, not by keyword:

"the result is <result>"

This means:

• Given the result is <result> and And the result is <result> route to the same handler because both have text "the result is <result>".
• The keyword (Given, When, Then, And) is not part of the match.
• Matching happens on the template text (with <placeholders> still present), not on resolved text.

Optional Extensions

A project may add regex or expression matching (e.g., Cucumber-style expressions), but exact text matching is the portable baseline that every conforming implementation must support.

Why Match on Template Text

Matching on the unresolved template means one handler registration covers all example rows. If matching happened after placeholder resolution, you would need a handler for "the result is accepted", another for "the result is rejected", etc. — defeating the purpose of parameterization.

Examples

Incorrect (matches on keyword + text, causing duplicate handler registrations):

{
  "handlers": {
    "Given the result is <result>": "handleGivenResult",
    "And the result is <result>": "handleAndResult"
  }
}

Correct (matches on text only, single handler covers all keywords):

{
  "handlers": {
    "the result is <result>": "handleResult"
  }
}

Why Ignore the Keyword

Keywords express human intent (Given = precondition, When = action, Then = assertion) but have no execution semantics. The same step text might appear as Given in one scenario and And in another. Matching on text only keeps the handler registry simple and avoids duplicate registrations for the same behavior.

Fail on Unsupported Step Text

When the runtime encounters a step whose text value does not match any registered handler, the test must fail. Silent skipping would hide incomplete handler implementations and produce false-positive test results.

Spec Requirements

Unsupported step text must fail the current test.

This applies to both:

• Steps that have no handler registered at all.
• Steps whose text was modified by a mutation such that it no longer matches any handler (this is rare since mutations target example values, not step text).

Why Not Skip or Warn

Skipping an unsupported step means the test suite reports "all tests pass" when in reality some steps were never executed. This is particularly dangerous during initial pipeline setup, where the developer is progressively implementing handlers. A test that silently skips 3 of 5 steps and passes on the remaining 2 gives false confidence.

Warning without failing has the same problem — warnings are routinely ignored in CI output.

Examples

Incorrect (silently skips unmatched steps, producing false passes):

def execute_step(step, handlers):
    handler = handlers.get(step.text)
    if handler is None:
        pass  # skip silently
    else:
        handler(step)

Correct (fails the test when step text has no registered handler):

def execute_step(step, handlers):
    handler = handlers.get(step.text)
    if handler is None:
        raise StepNotFoundError(f"no handler for: {step.text!r}")
    handler(step)

Why This Matters

The mutation workflow depends on the test suite being fully connected to the application. An unsupported step is a disconnected step — it exercises nothing. Failing immediately surfaces the gap and prompts the developer to implement the missing handler, which is exactly what the pipeline is designed to enforce.

This also means that when writing a new feature file, the natural workflow is: write the feature, run the pipeline, see failures for unimplemented handlers, implement handlers one by one until all tests pass.

Parse Placeholder Values from Examples

Step handlers receive string values from example objects and must convert them to project types. This conversion is where the pipeline's string-everywhere design meets the project's actual type system. Strict handling ensures mutations are detectable.

Spec Requirements

1. Handlers must fetch placeholder values by name from the current example object. 2. Handlers must parse string values into project types as needed (e.g., "42" to integer, "true" to boolean). 3. Missing values must fail the current test. 4. Malformed values must fail the current test. 5. Semantically invalid values must fail the current test.

Why Strict Failure

The mutation model depends on handlers detecting bad values. When the mutator changes "42" to "49", the handler that parses this integer and feeds it to the application must produce a different outcome. If the handler silently coerces "49" to "42" (via clamping, default values, or error swallowing), the mutation survives — and the survived mutation report correctly identifies a weak test.

But if the handler silently accepts any string without type checking, mutations like "42" to "4z2" (string dithering) would also pass silently, making the entire mutation testing model ineffective.

Examples

Incorrect (silently coerces malformed values, hiding mutations):

def handle_count(world, example):
    raw = example.get("count", "0")  # default hides missing
    count = int(raw) if raw.isdigit() else 0  # coercion hides mutation
    world["count"] = count

Correct (strict parsing fails on missing or malformed values):

def handle_count(world, example):
    if "count" not in example:
        raise ValueError("missing placeholder 'count'")
    raw = example["count"]
    try:
        count = int(raw)
    except ValueError:
        raise ValueError(f"malformed integer: {raw!r}")
    world["count"] = count

Why This Matters

Value handling is the bridge between the portable string-based IR and the project's strongly-typed domain. Every shortcut here — default values, silent coercion, ignored parse errors — creates a blind spot that mutation testing cannot probe. Strict failure on bad values is not pedantic; it is what makes the mutation model work.

Isolate World State per Execution

Each scenario execution gets a fresh world/state object. This isolation is fundamental to test reliability — without it, state from one execution leaks into another, causing order-dependent test results.

Spec Requirements

1. A scenario execution must get a fresh world/state object. 2. Background and scenario steps within the same execution share the same world/state object.

This means:

• Background steps set up shared state that scenario steps can read and modify.
• Each example row's execution starts with a clean slate.
• No state carries between different scenario executions.

Example

For a scenario with 2 example rows:

Execution 1 (row 0): fresh world -> background steps modify world -> scenario steps modify world
Execution 2 (row 1): fresh world -> background steps modify world -> scenario steps modify world

Execution 2 does not see any changes made by Execution 1.

Examples

Incorrect (shared world object leaks state between executions):

world = {}  # single instance reused

def run_execution(scenario, example_row):
    # world carries state from prior execution
    run_background(world, scenario.background)
    run_steps(world, scenario.steps, example_row)

Correct (fresh world object per execution prevents interference):

def run_execution(scenario, example_row):
    world = {}  # fresh per execution
    run_background(world, scenario.background)
    run_steps(world, scenario.steps, example_row)

Why This Matters

Test isolation is a first principle of reliable testing. If Execution 1 adds an item to a list in the world object and Execution 2 reads that list, the test results depend on execution order. This creates flaky tests that pass when run in one order and fail in another — the hardest kind of bug to diagnose.

The fresh-per-execution guarantee means tests are inherently parallelizable (though the spec does not require parallel scenario execution).

Store Example Values as Strings

An example object is a simple map from parameter names (column headers) to string values (cell contents). It represents one row of an examples table and drives one scenario execution.

Spec Requirements

{
  "input": "42",
  "command": "calculate total",
  "expected_status": "accepted"
}

All values must be strings, even when they represent:

• Numbers ("42", "3.14")
• Booleans ("true", "false")
• Lists ("2, 5, 8")
• Commands ("calculate total")
• Messages, enums, or any other domain type

Why Everything Is a String

This is the most important portability decision in the IR design. Three reasons:

1. No premature type coercion. The parser does not know the target language's type system. A value like "42" might be an integer, a string ID, or a port number. That decision belongs to the step handler.

2. Mutation model simplicity. The mutator applies string-based rules: detect if a string looks like an integer, boolean, date, etc., then mutate accordingly. Mixed types would require type-aware mutation logic for each possible language.

3. Cross-language portability. JSON types (number, boolean) have different precision and semantics across languages. Strings are universally lossless.

Why This Matters

If a parser stores 42 as a JSON number instead of "42", the mutator's integer detection rule would need to handle both strings and numbers. The runtime's placeholder resolution would need type-aware substitution. The IR would no longer be a stable contract — it would vary based on what the parser guesses about types.

Examples

Incorrect (values stored as native JSON types instead of strings):

{
  "input": 42,
  "enabled": true,
  "expected_status": "accepted"
}

Correct (all values stored as strings regardless of apparent type):

{
  "input": "42",
  "enabled": "true",
  "expected_status": "accepted"
}

Structure Feature Objects Correctly

The feature object is the root of the JSON IR. It is the canonical data structure consumed by the generator, runtime, and mutator. Every downstream tool depends on this shape being correct and stable.

Spec Requirements

{
  "name": "Feature name",
  "background": [
    {
      "keyword": "Given",
      "text": "a configured project state",
      "parameters": []
    }
  ],
  "scenarios": [
    {
      "name": "Scenario name",
      "steps": [],
      "examples": []
    }
  ]
}

Required fields:

Optional fields:

Why Background Is Optional

Not every feature has shared setup steps. Making background optional keeps the IR minimal for simple features while preserving the structure for features that need it. Consumers should treat a missing background field the same as an empty array.

Why This Matters

The feature object is the integration contract between all pipeline components. The parser writes it. The generator reads it to produce tests. The runtime reads it to execute scenarios. The mutator reads it to create mutations. If any tool writes a different shape, the pipeline breaks silently — tests might run but test the wrong thing.

Pretty-printing the JSON IR is expected (per parser requirements) so that humans can inspect and diff the IR during development.

Examples

Incorrect (background stored as nested object instead of step array):

{
  "name": "Calculator",
  "background": {
    "description": "common setup",
    "steps": [
      { "keyword": "Given", "text": "a configured project state" }
    ]
  },
  "scenarios": []
}

Correct (background is a flat array of step objects at the feature level):

{
  "name": "Calculator",
  "background": [
    { "keyword": "Given", "text": "a configured project state", "parameters": [] }
  ],
  "scenarios": []
}

Structure Scenario Objects Correctly

The scenario object represents a single specification scenario. Its shape is identical whether it came from Scenario: or Scenario Outline: in the Gherkin source. The presence or absence of examples determines execution behavior.

Spec Requirements

{
  "name": "Scenario name",
  "steps": [
    {
      "keyword": "Given",
      "text": "the input is <input>",
      "parameters": ["input"]
    }
  ],
  "examples": [
    {
      "input": "42"
    }
  ]
}

Required fields:

Empty Examples Behavior

If examples is empty ([]), the runtime must execute the scenario once with an empty example object ({}). This means a scenario without examples is not skipped — it still runs, just without parameter substitution.

Scenarios with empty examples cannot be mutated because there are no example cell values to change.

Why This Matters

The unified shape (no Scenario vs Scenario Outline distinction) means downstream tools have one code path. The examples array length tells the runtime how many executions to create and tells the mutator how many mutation candidates exist. This is simpler and more reliable than carrying a type flag through the pipeline.

Examples

Incorrect (scenario without examples omits the examples field entirely):

{
  "name": "Simple scenario",
  "steps": [
    { "keyword": "Given", "text": "the system is ready", "parameters": [] }
  ]
}

Correct (examples field is always present, empty array when no examples):

{
  "name": "Simple scenario",
  "steps": [
    { "keyword": "Given", "text": "the system is ready", "parameters": [] }
  ],
  "examples": []
}

Structure Step Objects Correctly

The step object is the atomic unit of specification. It pairs a keyword with descriptive text and optionally lists the parameters (placeholders) found in that text.

Spec Requirements

{
  "keyword": "Given",
  "text": "the input is <input>",
  "parameters": ["input"]
}

Required fields:

Optional fields:

Text Is Authoritative

The parameters field is derived from text. Generators and runtimes should treat text as the authoritative source and may validate that parameters agrees with the placeholders found in text.

This means if there is a conflict between text and parameters, text wins. The parameters field exists as a convenience — it saves consumers from re-parsing angle brackets.

Why Parameters Are Ordered and May Repeat

Parameters are recorded in the order they appear because position matters for some tooling. If a step contains <status> twice, the parameters array contains ["status", "status"]. This preserves the template structure faithfully.

Why This Matters

Step handlers match on the text field value. The runtime resolves <parameter_name> placeholders in text using the current example object. If text is wrong (truncated, missing placeholders, merged with keyword), both matching and resolution break.

Examples

Incorrect (parameters field disagrees with placeholders in text):

{
  "keyword": "Given",
  "text": "the input is <input> and output is <output>",
  "parameters": ["input"]
}

Correct (parameters array matches all placeholders in text, in order):

{
  "keyword": "Given",
  "text": "the input is <input> and output is <output>",
  "parameters": ["input", "output"]
}

Command Entry Points

The spec recommends three command entry points. These are the stable names that scripts, CI systems, and developers invoke. The actual implementation behind each command is project-specific.

Spec Requirements

Recommended command entry points:

gherkin-parser <feature-file> <json-output>
acceptance-generator <json-ir> <generated-test-output>
gherkin-mutator [options]

Why Named Commands, Not Direct Script Invocation

Named commands abstract the implementation. A project might implement the parser as:

• A compiled Go binary
• A Python script
• A Node.js CLI tool
• A shell script wrapping a library

The scripts and the mutator invoke gherkin-parser regardless of what is behind it. This means changing the implementation language does not require updating scripts.

How to Provide Named Commands

Common approaches:

• Add the project's bin/ directory to PATH and place executables there.
• Use the language's package manager (e.g., go install, npm link, pip install -e .).
• Define shell aliases or Makefile targets.
• Use package.json scripts or similar project-level command definitions.

Examples

Incorrect (ad-hoc script names -- scripts must hard-code implementation paths):

#!/bin/sh
set -eu
python3 scripts/parse_gherkin.py features/a-feature.feature build/acceptance/a-feature.json
node tools/gen_tests.js build/acceptance/a-feature.json acceptance/generated/a-feature_test.js

Correct (named commands -- scripts are implementation-agnostic):

#!/bin/sh
set -eu
gherkin-parser features/a-feature.feature build/acceptance/a-feature.json
acceptance-generator build/acceptance/a-feature.json acceptance/generated/a-feature_test.js

Why This Matters

Consistent command names make the pipeline self-documenting. A new developer looking at the acceptance script sees gherkin-parser and acceptance-generator — the names communicate what each step does. Custom names like parse.sh or gen_tests.py are less discoverable and harder to reference in documentation.

Required Project Paths

A conforming pipeline setup must create specific directories. These paths establish the convention that all pipeline components follow — the parser knows where to write, the generator knows where to read, and the mutator knows where to create work directories.

Spec Requirements

A conforming setup should create these paths or their project-specific equivalents:

features/a-feature.feature
build/acceptance/a-feature.json
build/acceptance-mutation/
acceptance/generated/

Why build/ for Intermediate Files

Parser output (JSON IR) and mutation work files are build artifacts — they are derived from source files and can be regenerated. Placing them under build/ follows the convention of keeping derived files separate from source files, making .gitignore rules simple (build/).

Why acceptance/generated/ Separate from build/

Generated tests are not just intermediate files — they are executable tests that the project's test runner discovers and runs. Some test frameworks require tests to be in specific locations. Keeping them under acceptance/generated/ puts them near the project's test root while clearly marking them as generated (not hand-written).

Why This Matters

The normal acceptance script and mutation script both reference these paths. If the directories do not exist, the first mkdir -p in the script creates them. But if the convention is not followed, paths in the script will not match paths expected by the pipeline components, causing file-not-found errors.

Consistent paths also make it easy to understand a project's pipeline structure at a glance.

Examples

Incorrect (non-standard paths -- pipeline components cannot find each other's output):

src/gherkin/
  a-feature.feature
out/
  a-feature.json
tmp/mutations/
tests/auto/

Correct (spec-conforming paths -- all pipeline components agree on locations):

features/
  a-feature.feature
build/acceptance/
  a-feature.json
build/acceptance-mutation/
acceptance/generated/

Convenience Scripts

The spec defines two convenience scripts: one for the normal acceptance run and one for mutation testing. These scripts provide stable, one-command entry points for the pipeline.

Normal Acceptance Script

#!/bin/sh
set -eu

mkdir -p build/acceptance acceptance/generated

gherkin-parser \
  features/a-feature.feature \
  build/acceptance/a-feature.json

acceptance-generator \
  build/acceptance/a-feature.json \
  acceptance/generated/a-feature_acceptance_test.<test-extension>

<project-test-command> acceptance/generated

Script requirements:

1. Stop on the first failed command (set -eu). A parse failure must not lead to running stale tests. 2. Create required output directories before writing files. The script must not assume directories exist. 3. Treat parser, generator, and test failures as script failures. The script exit code must reflect any failure. 4. Never run generated tests against the source feature file directly. Generated tests must be created from the JSON IR every time. This ensures mutations (when applied to the IR) are reflected in the tests.

Mutation Script

#!/bin/sh
set -eu

gherkin-mutator --feature features/a-feature.feature "$@"

The mutator command owns parsing, mutation, generation, test execution, and reporting. The mutation script is a thin wrapper that passes through additional arguments (like --workers, --timeout, --json).

Why set -eu

• set -e — Exit on the first command that fails. Without this, the script continues after a parser failure and runs stale generated tests, producing misleading results.
• set -u — Treat unset variables as errors. Prevents silent failures from typos in variable names.

Why Regenerate Every Time

Requirement 4 is critical for the mutation workflow. The mutator modifies the IR and asks the generator to produce tests from the modified IR. If the normal script cached generated tests and reused them, the mutation workflow would test the base IR instead of the mutated IR. Always regenerating from IR keeps the pipeline correct.

Examples

Incorrect (no set -eu, missing mkdir -- continues after parser failure and may fail on missing dirs):

#!/bin/sh

gherkin-parser features/a-feature.feature build/acceptance/a-feature.json
# If parser fails, script continues and runs stale generated tests
acceptance-generator build/acceptance/a-feature.json \
  acceptance/generated/a-feature_acceptance_test.ext
run-tests acceptance/generated

Correct (set -eu, mkdir -p, all failures propagated):

#!/bin/sh
set -eu

mkdir -p build/acceptance acceptance/generated

gherkin-parser \
  features/a-feature.feature \
  build/acceptance/a-feature.json

acceptance-generator \
  build/acceptance/a-feature.json \
  acceptance/generated/a-feature_acceptance_test.ext

run-tests acceptance/generated

Why This Matters

Without scripts, running the pipeline requires remembering and typing multiple commands in the correct order. Scripts encode this knowledge and make the pipeline runnable with a single command — essential for CI integration and for developers who did not write the pipeline.

Expose Stable Mutator CLI Contract

The mutator command owns the entire mutation testing workflow: parsing, mutation generation, test execution, and reporting. Its CLI interface must be stable because it is invoked from the mutation script and potentially from CI systems.

Spec Requirements

gherkin-mutator [options]

Options:

Exit Codes

Why Exit Code 1 Covers Both Survived and Errors

Both outcomes indicate the pipeline is not fully healthy. A survived mutation means weak tests. An error means the infrastructure needs fixing. Both require investigation, so both produce a non-zero exit code that stops CI pipelines.

Exit code 0 is reserved for the ideal state: every mutation was killed, meaning every example value change was detected by the acceptance tests.

Examples

Incorrect (non-standard options and missing exit-code contract):

# wrong flag names, no defined exit codes
gherkin-mutator --input features/a.feature --out build/ --parallel 4
echo $?  # returns 0 on survived mutations — CI does not catch gaps

Correct (spec-compliant options and three exit codes):

gherkin-mutator --feature features/a.feature --work-dir build/acceptance-mutation --workers 4 --timeout 5m --json
# exit 0 = all killed, exit 1 = survived or error, exit 2 = usage error
echo $?

Why Workers < 1 Becomes 1

Rather than erroring on invalid worker counts, the spec normalizes to 1. This is defensive — a misconfigured script that passes --workers 0 should still run mutations, just sequentially. The error exit code (2) is reserved for actual usage mistakes like unknown flags.

Deep Copy IR before Each Mutation

Each mutation must be applied to a deep copy of the base IR, never to the original. This is a correctness requirement, not just a best practice — in-place mutation makes the entire mutation set wrong.

Spec Requirements

The original JSON IR must not be modified in place. Each mutation is applied to a deep copy of the base IR.

Why This Is Critical

Consider three mutations: m1 changes count in row 0, m2 changes status in row 0, m3 changes count in row 1.

With in-place mutation: 1. m1 modifies the base IR (changes count in row 0). 2. m2 starts from the already-modified IR — now it has both m1's change and m2's change. 3. m3 starts from the doubly-modified IR — now it has m1, m2, and m3's changes.

Each mutation was supposed to test a single value change. Instead, later mutations test cumulative changes, making their results meaningless. A survived mutation might have been killed if tested in isolation.

Implementation Notes

Deep copying means:

• All nested objects and arrays must be copied, not just the top-level object.
• String values (which are immutable in most languages) do not need explicit copying.
• The copy must be complete before the mutation is applied.

In most languages, JSON.parse(JSON.stringify(ir)) or equivalent serialization round-trip achieves a correct deep copy. Language-specific deep copy utilities also work, but be careful of shared references in complex data structures.

Examples

Incorrect (mutates IR in place, corrupting subsequent mutations):

for mutation in mutations:
    # base_ir is modified in place — m2 sees m1's change
    base_ir["scenarios"][mutation.scenario]["examples"][mutation.row][mutation.key] = mutation.value
    result = run_tests(base_ir)

Correct (deep copies IR before each mutation):

import copy

for mutation in mutations:
    ir_copy = copy.deepcopy(base_ir)
    ir_copy["scenarios"][mutation.scenario]["examples"][mutation.row][mutation.key] = mutation.value
    result = run_tests(ir_copy)

Why This Matters

The mutation model's validity depends on each mutation being independent. The report says "mutation m2 survived" — this means "changing only this one cell value was not detected." If the IR was not deep-copied, "only this one cell" is false, and the report is misleading.

Assign Stable Deterministic Mutation IDs

Every mutation needs a stable, deterministic identity so that results can be reproduced, diffed across runs, and referenced in discussions. The spec defines three identity components: ID, path, and description.

Spec Requirements

Mutation IDs are sequential and stable for a fixed input IR:

m1
m2
m3
...

Mutation paths use this format:

$.scenarios[<scenario_index>].examples[<example_index>].<key>

• Indexes are zero-based.
• Keys are the literal example object keys.

Mutation descriptions use:

<path>: <original> -> <mutated>

Example

For a feature with one scenario, two example rows, and a key count:

m1  $.scenarios[0].examples[0].count: 20 -> 27
m2  $.scenarios[0].examples[1].count: 5 -> 12

Why Sequential IDs

Sequential IDs (m1, m2, ...) are compact, sortable, and human-friendly. They are assigned in enumeration order (scenarios by index, example rows by index, keys in lexicographic order), so the same IR always produces the same ID assignment.

Why JSON Path Notation

The $.scenarios[i].examples[j].key format precisely locates the mutation in the IR structure. Developers can use this path to navigate directly to the mutated cell in the IR JSON file, making investigation straightforward.

Examples

Incorrect (random UUIDs as mutation IDs, non-reproducible across runs):

{
  "mutations": [
    { "id": "a3f2-b8c1", "path": "scenarios[0].examples[0].count", "desc": "changed count" },
    { "id": "d7e4-19a0", "path": "scenarios[0].examples[1].count", "desc": "changed count" }
  ]
}

Correct (sequential IDs, JSON-path notation, value-change descriptions):

{
  "mutations": [
    { "id": "m1", "path": "$.scenarios[0].examples[0].count", "desc": "$.scenarios[0].examples[0].count: 20 -> 27" },
    { "id": "m2", "path": "$.scenarios[0].examples[1].count", "desc": "$.scenarios[0].examples[1].count: 5 -> 12" }
  ]
}

Why This Matters

When a mutation survives, the developer needs to understand exactly what changed and where. The description $.scenarios[0].examples[0].count: 20 -> 27 tells them: scenario 0, example row 0, the count field was changed from 20 to 27. They can then look at the step handlers for steps that use <count> and determine why the test did not catch the change.

Deterministic IDs also enable tracking mutation results over time — if m3 survived yesterday and is killed today, the fix can be correlated to the code change.

Restrict Mutation Scope to Example Values

The mutator creates candidate mutations from a precisely defined scope. Understanding what is and is not mutated is essential for both implementing the mutator and interpreting its results.

Spec Requirements

The mutator only mutates example cell values.

The mutator does not mutate:

• Feature names
• Scenario names
• Step text
• Step keywords
• Background steps
• Example headers (column names)

Mutation Enumeration

For each scenario, for each example row, for each example key in lexicographic order:

1. Read the original string value. 2. Compute the mutated value using the value mutation rules. 3. If the mutated value is identical to the original, skip it. 4. Create one mutation that changes only that single example cell.

Why Only Example Values

Example values are the concrete data that drives acceptance test behavior. If changing an example value does not cause a test failure, it means the test is not actually checking that value — it is disconnected from the application behavior it claims to verify.

Mutating step text would change the specification itself (what the test checks), not the data it checks with. Mutating keywords or names would change labeling, not behavior. Mutating backgrounds would affect all scenarios simultaneously, making it impossible to isolate which test is weak.

Why Lexicographic Key Order

JSON object key order is not guaranteed. By processing keys in lexicographic order, the mutator produces the same mutation sequence regardless of the JSON library's internal ordering. This ensures stable, deterministic mutation IDs across platforms.

Examples

Incorrect (mutates step text and scenario names, producing unclassifiable results):

{
  "scenarios": [{
    "name": "Mutated Scenario Name",
    "steps": [
      { "keyword": "Given", "text": "the mutated step text" }
    ],
    "examples": [
      { "count": "20", "status": "accepted" }
    ]
  }]
}

Correct (only example cell values are mutated, one cell per mutation):

{
  "scenarios": [{
    "name": "Verify order count",
    "steps": [
      { "keyword": "Given", "text": "the order has <count> items" }
    ],
    "examples": [
      { "count": "27", "status": "accepted" }
    ]
  }]
}

Why Skip Identical Mutations

Some mutation rules might produce the same value (e.g., dithering a single character that maps back to itself). Including these as mutations would create entries that can never be killed — they would always survive because the IR is unchanged. Skipping them keeps the mutation set meaningful.

Implement Background Step Prepending

Background captures shared precondition steps that apply to every scenario in the feature. The parser records background steps so the runtime can prepend them to each scenario execution. Getting this wrong means scenarios either miss shared setup or duplicate it inconsistently.

Spec Requirements

A feature may contain one optional background:

Background:
  Given <step text>
  And <step text>

• Background steps use Given and And keywords.
• Background steps are recorded in the IR and prepended to every scenario execution by the acceptance runtime (not the parser).
• If multiple background sections are present, portable behavior is undefined. New projects should use at most one.

Parser Responsibility

The parser stores background steps in the IR's background array. It does not prepend them to scenarios — that is the runtime's job. This separation matters because the mutator works on the IR and must not mutate background steps. If the parser had already merged backgrounds into scenarios, the mutator would need extra logic to distinguish original steps from background steps.

Why This Matters

Background exists to eliminate duplication in feature files. When every scenario shares "Given a configured project state," writing it once in Background keeps the feature file DRY. The parser must faithfully capture this structure so the runtime can apply it uniformly.

Examples

Incorrect (parser merges background steps into each scenario in the IR):

{
  "name": "Calculator",
  "scenarios": [
    {
      "name": "Addition",
      "steps": [
        { "keyword": "Given", "text": "a configured project state", "parameters": [] },
        { "keyword": "When", "text": "I add 1 and 2", "parameters": [] },
        { "keyword": "Then", "text": "the result is 3", "parameters": [] }
      ],
      "examples": []
    }
  ]
}

Correct (parser stores background separately; runtime prepends at execution time):

{
  "name": "Calculator",
  "background": [
    { "keyword": "Given", "text": "a configured project state", "parameters": [] }
  ],
  "scenarios": [
    {
      "name": "Addition",
      "steps": [
        { "keyword": "When", "text": "I add 1 and 2", "parameters": [] },
        { "keyword": "Then", "text": "the result is 3", "parameters": [] }
      ],
      "examples": []
    }
  ]
}

Enforce Parser Command Interface

The parser is the pipeline entry point. A predictable command interface ensures that the generator, mutator, and scripts can invoke it without special-casing. If the parser's interface deviates from the spec, every downstream tool must compensate, creating fragile coupling.

Spec Requirements

The parser command accepts exactly two positional arguments:

gherkin-parser <feature-file> <json-output>

• <feature-file> — path to the Gherkin source file to parse.
• <json-output> — path where the pretty-printed JSON IR will be written.

Exit Codes

Exit codes must follow this contract so scripts can branch on failure:

Why This Matters

The distinction between exit code 1 (content error) and exit code 2 (usage error) lets scripts provide different diagnostics. A usage error means the script itself is misconfigured. A parse error means the feature file needs fixing. Conflating these forces manual investigation of every failure.

The parser writes pretty-printed JSON IR. Pretty-printing costs negligible performance but makes the IR human-readable for debugging and diffing — critical during pipeline development.

Examples

Incorrect (single exit code for all errors conflates usage and parse failures):

# Non-conforming: exits 1 for both bad arguments and parse errors
gherkin-parser
# exit code 1, message: "Error: missing arguments"

gherkin-parser bad.feature out.json
# exit code 1, message: "Error: parse failed at line 3"

Correct (exit code 2 for usage errors, exit code 1 for parse errors):

# Conforming: distinct exit codes enable script-level branching
gherkin-parser
# exit code 2, message: "Usage: gherkin-parser <feature-file> <json-output>"

gherkin-parser bad.feature out.json
# exit code 1, message: "Parse error at line 3: expected Feature: declaration"

gherkin-parser valid.feature out.json
# exit code 0, out.json written with pretty-printed JSON IR

Parse Pipe-Delimited Examples Tables

Examples tables provide the concrete data that parameterized scenarios run against. Each row becomes one scenario execution, and each cell becomes a value the runtime substitutes into step text. Correct table parsing is critical because every downstream tool — runtime, generator, mutator — depends on these values.

Spec Requirements

An examples section starts with:

Examples:

It must appear inside a scenario.

Rows are pipe-delimited:

| name | count |
| one  | 1     |
| two  | 2     |

Parsing Rules

1. A row is recognized only when the trimmed line starts with |. 2. Leading and trailing | characters are removed. 3. The remaining text is split on |. 4. Each cell is trimmed. 5. The first row after Examples: is the header row. 6. Every data row must have the same number of cells as the header row (mismatch = error, exit code 1). 7. Header names become JSON object keys. 8. All cell values are stored as strings — even numbers, booleans, and other types.

Why All Values Are Strings

Storing everything as strings is a deliberate portability choice. The parser does not know the target language's type system. A value like 42 might be an integer, a string, or a port number — that decision belongs to the step handler, not the parser. String storage means no information is lost and no premature type coercion occurs.

This also simplifies the mutator: it applies string-based mutation rules (integer detection, boolean detection, etc.) to values that are always strings, without needing to handle mixed types.

Why Cell Count Must Match

A row with fewer or more cells than the header indicates either a formatting error or missing data. Rather than silently padding with empty strings or truncating, the parser rejects the mismatch. This catches feature file errors early, before they propagate as mysterious runtime failures.

Examples

Incorrect (cell count mismatch silently padded with empty string):

Examples:
  | name | count |
  | one  |

[{ "name": "one", "count": "" }]

Correct (cell count mismatch rejected with exit code 1):

Parse error: row 2 has 1 cell(s), expected 2 (matching header row)
Exit code: 1

Require Feature Declaration

Every feature file must declare a feature. The feature name is the only human-readable identifier that carries through the entire pipeline — from Gherkin source to JSON IR to generated tests to mutation reports. Without it, debugging becomes guesswork.

Spec Requirements

A feature file must contain a feature declaration:

Feature: <feature name>

• The feature name is the trimmed text after Feature:.
• A missing feature declaration is an error (exit code 1).
• If multiple feature declarations appear, a conforming parser should treat that as invalid or use the last declaration consistently. New projects should use exactly one.

Why This Matters

The feature name flows into the IR's top-level name field, which generated tests and mutation reports use for identification. A missing or empty name creates anonymous test suites that are impossible to correlate back to their source feature when multiple features exist.

Requiring exactly one declaration keeps the mapping 1:1 between feature files and IR documents, which simplifies the generator and mutator — they never need to handle multi-feature IR.

Examples

Incorrect (missing feature declaration produces unnamed IR):

Scenario: Login succeeds
  Given a valid user
  When the user logs in
  Then access is granted

{
  "name": "",
  "scenarios": [...]
}

Correct (feature declaration provides a named, identifiable IR):

Feature: User Authentication
  Scenario: Login succeeds
    Given a valid user
    When the user logs in
    Then access is granted

{
  "name": "User Authentication",
  "scenarios": [...]
}

Apply General Parsing Rules

These rules govern how the parser handles whitespace, comments, blank lines, and ordering. They apply across all parsed elements and ensure the parser produces consistent, predictable IR regardless of how the Gherkin file is formatted.

Spec Requirements

Blank lines are ignored. They exist for human readability and carry no semantic meaning.

Comment lines — lines whose first non-whitespace character is # — are ignored. Comments are not preserved in the IR.

Leading and trailing whitespace are ignored before parsing each line. This means indentation is cosmetic — Given a step and Given a step parse identically.

Free-form lines that do not match any supported syntax are ignored. This allows brief feature descriptions after the Feature: line, but they are not preserved in the IR.

Ordering is preserved. The parser must preserve the order of:

• Background steps
• Scenarios (in declaration order)
• Steps within each scenario
• Example rows within each examples table

Key column ordering note: Example columns become object keys in the JSON IR. Consumers that need deterministic key traversal must sort keys explicitly, because JSON object key order is not guaranteed by all implementations.

Why Free-Form Lines Are Ignored

Gherkin traditionally supports description blocks after Feature: and Scenario:. This spec ignores them because they serve no role in the pipeline — the IR, generator, runtime, and mutator never reference descriptions. Silently ignoring them is more practical than erroring, because it lets users write natural Gherkin without the parser rejecting documentation prose.

Why Ordering Matters

The runtime executes steps in order. Background steps establish preconditions, Given sets up state, When triggers actions, Then asserts results. Reordering would change the test semantics. Similarly, example rows map to mutation IDs by position — reordering rows would produce different mutation IDs for the same data.

Examples

Incorrect (comment preserved in IR and indentation changes parse result):

Feature: Calculator
  # This is a setup comment
  Scenario: Addition
    Given the input is 1

{
  "name": "Calculator",
  "comment": "This is a setup comment",
  "scenarios": [...]
}

Correct (comments ignored, indentation is cosmetic, ordering preserved):

{
  "name": "Calculator",
  "scenarios": [
    {
      "name": "Addition",
      "steps": [
        { "keyword": "Given", "text": "the input is 1", "parameters": [] }
      ],
      "examples": []
    }
  ]
}

Preserve Parameter Placeholders

Parameters are placeholders inside step text that get resolved at runtime using example values. The parser extracts parameter names but does not expand them — expansion happens in the runtime. This separation is essential for the mutator, which needs to see the template form of steps.

Spec Requirements

Parameters appear as angle-bracket placeholders inside step text:

<parameter_name>

Parameter names must match this pattern:

[A-Za-z0-9_]+

Parsing rules:

• The parser records parameter names in the order they appear in each step's text.
• Repeated parameter names are preserved as repeated entries in the parameters array.
• Parameters are not expanded by the parser. They remain as <parameter_name> in the step text.
• Resolution happens in the acceptance runtime using the current example object.

Example

Given this step:

Then the <status> response contains <status> code

The parser produces:

{
  "keyword": "Then",
  "text": "the <status> response contains <status> code",
  "parameters": ["status", "status"]
}

Why the Parser Does Not Expand

If the parser expanded parameters, the IR would contain fully resolved step text — one copy per example row. This would lose the template structure that the mutator needs. The mutator changes example values, not step text. It needs to see <parameter_name> in the template to know which values are substitutable.

The runtime resolves placeholders at execution time, keeping the IR compact and the mutation model clean.

Examples

Incorrect (parser expands placeholders, losing the template structure):

{
  "keyword": "Then",
  "text": "the accepted response contains accepted code",
  "parameters": []
}

Correct (parser preserves angle-bracket placeholders for runtime resolution):

{
  "keyword": "Then",
  "text": "the <status> response contains <status> code",
  "parameters": ["status", "status"]
}

Support Both Scenario Forms

Scenarios are the core units of specification. The parser must handle both Scenario: and Scenario Outline: because real-world feature files use both forms. Treating them differently in the IR would force downstream tools to branch on scenario type.

Spec Requirements

The parser accepts both keywords:

Scenario: <scenario name>
  Given <step text>
  When <step text>
  Then <step text>

Scenario Outline: <scenario name>
  Given <step text containing <parameter_name>>

  Examples:
    | parameter_name |
    | value          |

Both forms produce the same JSON IR shape. The IR does not distinguish between Scenario: and Scenario Outline: — the difference is whether the scenario has examples.

Key Behaviors

• A scenario with examples can be mutated (mutations target example cell values).
• A scenario without examples is valid, executes once with an empty example object, and cannot be mutated.
• The scenario name is the trimmed text after the keyword.

Why This Matters

Unifying both forms into one IR shape is a deliberate design choice. It means the generator, runtime, and mutator each have exactly one code path for scenarios. The Scenario: vs Scenario Outline: distinction is syntactic sugar in the Gherkin source — what matters downstream is whether examples is populated.

Examples

Incorrect (IR distinguishes Scenario from Scenario Outline with a type field):

{
  "name": "Addition",
  "type": "scenario_outline",
  "steps": [
    { "keyword": "Given", "text": "the input is <input>", "parameters": ["input"] }
  ],
  "examples": [{ "input": "42" }]
}

Correct (unified IR shape for both forms; presence of examples determines behavior):

{
  "name": "Addition",
  "steps": [
    { "keyword": "Given", "text": "the input is <input>", "parameters": ["input"] }
  ],
  "examples": [{ "input": "42" }]
}

Separate Step Keywords from Text

Steps are the atomic instructions within scenarios. Each step pairs a keyword with descriptive text. The parser must store these separately because handlers match on text (not keyword), while the keyword carries semantic meaning for human readers and potential tooling.

Spec Requirements

Supported step keywords:

Given
When
Then
And

A step line must be one of:

Given <step text>
When <step text>
Then <step text>
And <step text>

Parsing rules:

• The keyword is stored separately from the step text.
• The step text is the trimmed text after the keyword.
• A step outside a background or scenario is an error (exit code 1).

Why Keyword and Text Are Separate

Handlers match by exact text value, not by keyword. This means Given the system is ready and And the system is ready route to the same handler. If keyword and text were merged, handler authors would need to register multiple patterns for the same step.

Storing the keyword separately also preserves the Gherkin author's intent (Given = precondition, When = action, Then = assertion, And = continuation) without burdening the matching logic.

Error on Orphan Steps

A step that appears outside any Background or Scenario context is structurally invalid — it has no scenario to belong to and no execution context. The parser must reject this rather than silently dropping it, because a dropped step likely means the feature file has a formatting error the author needs to fix.

Examples

Incorrect (keyword merged with text, preventing handler matching):

{
  "keyword": "Given",
  "text": "Given the system is ready"
}

Correct (keyword stored separately, text is keyword-free for handler matching):

{
  "keyword": "Given",
  "text": "the system is ready"
}

Reject Unsupported Gherkin Syntax

The spec deliberately excludes several Gherkin features. This is not an oversight — it is a portability and simplicity decision. Each excluded feature adds parser complexity, IR fields, and downstream handling without contributing to the core pipeline goal of acceptance testing with mutation coverage.

Excluded Syntax

What Happens When Unsupported Syntax Appears

Lines that do not match supported syntax are treated as free-form lines and ignored (per the general parsing rules). This means a @tag line is silently dropped, not treated as an error.

The exception is structural violations — for example, an Examples: section outside a scenario is still an error. But decorative syntax like tags simply has no effect.

Why This Matters

A small, deterministic syntax subset means every conforming parser produces identical IR for the same input. This is essential for the mutation testing model, which depends on stable, predictable IR structure. Adding optional syntax would create parser variants that produce different IR, breaking cross-tool compatibility.

Projects that need tags or rules can preprocess their Gherkin files before feeding them to this pipeline.

Examples

Incorrect (parser attempts to support tags, creating non-portable IR):

@smoke @login
Feature: User Authentication
  Scenario: Login succeeds
    Given a valid user

{
  "name": "User Authentication",
  "tags": ["smoke", "login"],
  "scenarios": [...]
}

Correct (unsupported syntax silently ignored, clean portable IR):

{
  "name": "User Authentication",
  "scenarios": [
    {
      "name": "Login succeeds",
      "steps": [
        { "keyword": "Given", "text": "a valid user", "parameters": [] }
      ],
      "examples": []
    }
  ]
}

Report Field Requirements

Both text and JSON reports must include specific fields. These requirements ensure that any conforming report consumer can extract the information it needs regardless of which implementation produced the report.

Summary Fields

Invariant: Total = Killed + Survived + Errors

Result Object Fields

Each result must include:

Why All Fields Are Required

Optional fields in reports create compatibility problems. A CI tool that expects Mutation.Path but finds it missing must either crash or produce incomplete dashboards. By requiring all fields (even if some are empty strings), consumers can always destructure the result without null checks.

Why Duration Is Implementation-Defined

Different languages represent duration differently: nanoseconds (Go), milliseconds (JS), floating-point seconds (Python). Mandating a specific format would be impractical. Implementations should document their duration format and keep it stable.

Examples

Incorrect (missing Mutation.Path and Duration fields -- consumer crashes on destructure):

{
  "summary": { "Total": 1, "Killed": 1, "Survived": 0, "Errors": 0 },
  "results": [
    {
      "Mutation": {
        "ID": "m1",
        "Description": "$.scenarios[0].examples[0].count: 20 -> 27",
        "Original": "20",
        "Mutated": "27"
      },
      "Status": "killed",
      "Output": "FAIL: expected 20, got 27"
    }
  ]
}

Correct (all required fields present, including empty strings for unused fields):

{
  "summary": { "Total": 1, "Killed": 1, "Survived": 0, "Errors": 0 },
  "results": [
    {
      "Mutation": {
        "ID": "m1",
        "Path": "$.scenarios[0].examples[0].count",
        "Description": "$.scenarios[0].examples[0].count: 20 -> 27",
        "Original": "20",
        "Mutated": "27"
      },
      "Status": "killed",
      "Output": "FAIL: expected 20, got 27",
      "Error": "",
      "Duration": 125000000
    }
  ]
}

Why This Matters

The report is the pipeline's final output — everything the pipeline produces is summarized here. If the report is incomplete or inconsistent, the developer cannot make informed decisions about test quality. Complete, consistent fields ensure the report serves its purpose: telling the developer exactly which mutations survived and why.

JSON Report Format

When --json is supplied, the mutator emits a structured JSON report suitable for programmatic consumption by CI systems, dashboards, and analysis tools.

Spec Requirements

{
  "summary": {
    "Total": 2,
    "Killed": 1,
    "Survived": 1,
    "Errors": 0
  },
  "results": [
    {
      "Mutation": {
        "ID": "m1",
        "Path": "$.scenarios[0].examples[0].count",
        "Description": "$.scenarios[0].examples[0].count: 20 -> 27",
        "Original": "20",
        "Mutated": "27"
      },
      "Status": "killed",
      "Output": "<test runner output>",
      "Error": "",
      "Duration": 125000000
    }
  ]
}

Structure

The JSON object has two top-level fields:

• `summary` — Aggregate counts for quick evaluation.
• `results` — Array of individual mutation results in stable order (by mutation ID).

Why JSON Report

The text report is for humans; the JSON report is for machines. CI systems can parse the JSON to:

• Fail builds when Survived > 0.
• Track mutation kill rates over time.
• Generate HTML dashboards.
• Compare mutation results across branches.

Key Casing Note

Implementations may choose idiomatic JSON key casing (e.g., camelCase or snake_case instead of PascalCase), but they should document it and keep it stable. Changing key casing between versions breaks consumers.

Examples

Incorrect (flat array without summary -- consumers must recompute counts):

[
  { "id": "m1", "status": "killed", "output": "..." },
  { "id": "m2", "status": "survived", "output": "..." }
]

Correct (summary + results structure -- consumers get pre-computed counts and full mutation details):

{
  "summary": {
    "Total": 2,
    "Killed": 1,
    "Survived": 1,
    "Errors": 0
  },
  "results": [
    {
      "Mutation": {
        "ID": "m1",
        "Path": "$.scenarios[0].examples[0].count",
        "Description": "$.scenarios[0].examples[0].count: 20 -> 27",
        "Original": "20",
        "Mutated": "27"
      },
      "Status": "killed",
      "Output": "<test runner output>",
      "Error": "",
      "Duration": 125000000
    }
  ]
}

Why This Matters

Machine-readable output enables automation. Without a JSON report, CI integration requires parsing the text report with regex — which is fragile and breaks when the text format changes. The JSON report provides a stable contract for tooling.

Text Report Format

The default text report is designed for human consumption in terminal output. Its format is stable so that developers can scan it quickly and grep for specific statuses.

Spec Requirements

The report starts with one summary line:

total=<total> killed=<killed> survived=<survived> errors=<errors>

Then one line per result:

<status> <path>: <original> -> <mutated>

Status should be left-aligned to 8 characters for readability.

For survived and error results, include available details:

  error: <error text>
  output:
<runner output>

Example

total=2 killed=1 survived=1 errors=0
killed   $.scenarios[0].examples[0].count: 20 -> 27
survived $.scenarios[1].examples[0].status: accepted -> accfpted
  output:
<test runner output>

Why Summary Line First

Developers want the high-level answer first: "how many survived?" Placing the summary before individual results means they can read one line and decide whether to investigate further. This follows the principle of progressive disclosure — summary first, details on demand.

Why Left-Aligned Status

Fixed-width status alignment (killed , survived, error ) creates visual columns that make scanning large reports fast. The eye can track the left column to find all "survived" entries without reading each line fully.

Why Details Only for Survived and Error

Killed mutations are working correctly — they need no investigation. Only survived (test gap) and error (infrastructure issue) results need diagnostic detail. Including output for every killed mutation would bloat the report with noise.

Examples

Incorrect (missing summary line and inconsistent status alignment):

killed $.scenarios[0].examples[0].count: 20 -> 27
survived $.scenarios[1].examples[0].status: accepted -> accfpted
  output:
<test runner output>

Correct (summary line first, statuses left-aligned to 8 characters):

total=2 killed=1 survived=1 errors=0
killed   $.scenarios[0].examples[0].count: 20 -> 27
survived $.scenarios[1].examples[0].status: accepted -> accfpted
  output:
<test runner output>

Why This Matters

The text report is the primary output developers see in their terminal. A well-structured report reduces the time from "mutation testing finished" to "I know what to fix." The consistent format also enables simple tooling (grep, awk) to extract metrics from the report.

Classification Rules

These rules map test runner outcomes to mutation statuses. The mapping must be applied consistently for every mutation — any deviation corrupts the report.

Spec Requirements

The Key Distinction

The distinction between killed and error requires understanding why the tests failed:

• If the generated tests ran and produced assertion failures — that is killed. The tests are doing their job.
• If the generated tests could not be created or started — that is error. The infrastructure failed.

The test runner adapter's three-way output (test failure, test success, infrastructure error) provides exactly this distinction. The classification rules simply map that output to mutation statuses.

Why This Matters

Consider a mutation that makes the IR invalid for the generator. The generator fails, no tests are created, and no tests run. If this is classified as killed, the report claims the tests caught the mutation — but no tests ran. The report would be wrong.

Correct classification ensures the mutation report accurately reflects which mutations were genuinely caught by tests (killed), which were missed by tests (survived), and which could not be evaluated (error).

Examples

Incorrect (generator crash classified as killed -- inflates kill rate):

{
  "Mutation": { "ID": "m1", "Path": "$.scenarios[0].examples[0].count" },
  "Status": "killed",
  "Error": "generation failed: invalid IR schema",
  "Output": ""
}

Correct (generator crash classified as error -- keeps kill rate honest):

{
  "Mutation": { "ID": "m1", "Path": "$.scenarios[0].examples[0].count" },
  "Status": "error",
  "Error": "generation failed: invalid IR schema",
  "Output": ""
}

Result Statuses

Every mutation has exactly one of three statuses. These statuses are the output of the entire mutation testing pipeline — they tell the developer whether their acceptance tests are strong enough.

Spec Requirements

What Each Status Means for Test Quality

Killed is the desired outcome. It means the acceptance tests detected the changed specification value. The test is connected to the application behavior it claims to verify.

Survived means the acceptance tests did not detect the changed specification value. This should be investigated — it usually indicates a gap in assertions, a loose match, or a handler that ignores the value.

Error is not a test-quality result. It means the mutation could not be evaluated reliably. The infrastructure needs fixing before the mutation can be classified as killed or survived.

Why Three Statuses, Not Two

A two-status model (killed/survived) would force infrastructure failures into one of those categories. Classifying a generator crash as "killed" inflates test quality metrics. Classifying it as "survived" creates false alarms. Neither is honest.

The three-status model keeps the mutation report trustworthy: killed and survived counts reflect actual test behavior, and error counts reflect infrastructure health.

Examples

Incorrect (two-status model -- infrastructure failures classified as killed):

{
  "results": [
    {
      "Mutation": { "ID": "m1", "Description": "$.scenarios[0].examples[0].count: 20 -> 27" },
      "Status": "killed",
      "Error": "generator crashed: template not found"
    }
  ]
}

Correct (three-status model -- infrastructure failures classified as error):

{
  "results": [
    {
      "Mutation": { "ID": "m1", "Description": "$.scenarios[0].examples[0].count: 20 -> 27" },
      "Status": "error",
      "Error": "generator crashed: template not found"
    }
  ]
}

Why This Matters

The mutation report is only useful if the statuses are accurate. Developers make decisions based on these numbers: "3 survived, let me investigate" or "all killed, tests are strong." Wrong classification leads to wrong decisions — either false confidence (ignoring real gaps) or wasted investigation (chasing phantom gaps).

Apply Execution Naming Convention

Each scenario execution needs a stable, human-readable name for test output, error reporting, and debugging. The naming convention must be deterministic so that test results can be correlated back to specific scenario + example combinations.

Spec Requirements

Suggested execution naming format:

<scenario name>/example_<one-based-index>

• For scenarios with examples: My scenario/example_1, My scenario/example_2, etc.
• For scenarios without examples: Use example_1 or another stable name.

The index is one-based, matching human expectations (first example is example 1, not example 0).

Why One-Based

While the IR uses zero-based indexing for mutation paths ($.scenarios[0].examples[0]), execution names use one-based indexing because they appear in test output read by humans. This follows the convention of most test frameworks that number test cases starting from 1.

Why This Matters

When a mutation test reports "survived" for a specific mutation path, the developer needs to find the corresponding test execution in the output. Consistent naming makes this lookup straightforward. Without a naming convention, each implementation would invent its own scheme, making cross-project debugging harder.

The / separator creates a natural hierarchy: scenario name as the group, example index as the specific case. This aligns with test framework conventions for nested test suites.

Examples

Incorrect (zero-based indexing, no hierarchy separator):

Addition_example_0
Addition_example_1

Correct (one-based indexing with hierarchy separator):

Addition/example_1
Addition/example_2

Fulfill Runtime Responsibilities

The acceptance runtime is the shared execution engine used by generated tests. It sits between the IR and the project step handlers, orchestrating scenario execution. Every generated test delegates to the runtime rather than implementing execution logic directly.

Spec Requirements

The runtime must:

1. Load or receive the JSON IR. 2. Expand each scenario into scenario executions. Scenarios with examples produce one execution per row; scenarios without examples produce one execution with an empty example object. 3. Prepend background steps to each execution. 4. Execute steps in order. 5. Resolve placeholder values from the current example object (replacing <parameter_name> with the corresponding string value). 6. Route each step to a project step handler based on the step's text field. 7. Report failures — unsupported step, missing value, invalid conversion, or failed assertion must be reported as test failures.

Why a Shared Runtime

Without a shared runtime, every generated test file would need to implement scenario expansion, background prepending, placeholder resolution, and handler dispatch. This duplicates logic and creates inconsistency risk. The runtime centralizes these responsibilities so the generator only needs to emit the glue that connects the test framework to the runtime.

Why This Matters

The runtime is where the specification meets the project. If the runtime skips a background step, preconditions are missing. If it fails to resolve a placeholder, step handlers receive raw template text instead of values. If it swallows an unsupported step, the test passes when it should fail. Each of these failures is subtle and hard to diagnose without understanding the runtime's contract.

Examples

Incorrect (runtime swallows unsupported step instead of failing the test):

{
  "step": { "keyword": "When", "text": "I do something undefined" },
  "handler_found": false,
  "result": "skipped"
}

Correct (runtime reports unsupported step as a test failure):

{
  "step": { "keyword": "When", "text": "I do something undefined" },
  "handler_found": false,
  "result": "failed",
  "error": "No handler registered for step text: 'I do something undefined'"
}

Expand Scenarios into Executions

Scenario expansion is the process of turning one scenario definition into one or more concrete executions. This is where the parameterized specification becomes concrete test runs.

Spec Requirements

With examples: Create one execution per example row. Each execution uses that row's values for placeholder resolution.

Without examples: Create one execution with an empty example object ({}). The scenario still runs — it is not skipped.

Background prepending: Background steps are prepended to each execution. This means:

• If a scenario has 3 example rows and the feature has 2 background steps, each of the 3 executions gets those 2 background steps at the beginning.
• Background and scenario steps within the same execution share the same world/state object.

Execution Structure

For a scenario with 2 background steps and 3 scenario steps with 2 example rows:

Execution 1: [bg_step_1, bg_step_2, step_1, step_2, step_3] with example row 0
Execution 2: [bg_step_1, bg_step_2, step_1, step_2, step_3] with example row 1

Why Empty Example Object, Not Skip

A scenario without examples is a valid specification — it describes behavior that does not vary. Skipping it would mean untested behavior. Running it once with {} means placeholder resolution finds nothing to resolve (no <param> in step text, or if present, it would fail as missing — which is correct).

Why This Matters

The mutation model depends on correct expansion. Mutation IDs reference $.scenarios[i].examples[j].key — if expansion does not align with this indexing, mutation results are attributed to the wrong scenario execution.

Examples

Incorrect (scenario without examples is skipped entirely):

{
  "scenario": "Simple check",
  "examples": [],
  "executions_created": 0,
  "result": "skipped"
}

Correct (scenario without examples runs once with empty example object):

{
  "scenario": "Simple check",
  "examples": [],
  "executions_created": 1,
  "execution_example": {},
  "result": "executed"
}

Classify Runner Outcomes Three Ways

The test runner must distinguish three outcomes, not just pass/fail. This three-way distinction is essential for accurate mutation classification — conflating infrastructure errors with test failures corrupts the mutation report.

Spec Requirements

The adapter must distinguish:

Why Three-Way, Not Two-Way

Consider a mutation that produces invalid IR (e.g., an empty string where a number was expected). The generator might fail to produce tests. If this is classified as "killed" (tests failed), the report incorrectly suggests the acceptance tests caught the mutation. If classified as "survived" (tests passed), the report incorrectly suggests a gap in test coverage.

Neither is right. The mutation could not be evaluated, so the correct answer is "error" — try again or investigate the infrastructure.

Examples of Each

• Test failure: Tests ran, assertions failed because the mutated value produced wrong behavior. This is a "killed" mutation — the tests work.
• Test success: Tests ran, all assertions passed despite the mutated value. This is a "survived" mutation — the tests need strengthening.
• Infrastructure error: Generator crashed on malformed IR, test runner timed out, file system full, test framework not installed. This is an "error" — not a test quality signal.

Examples

Incorrect (two-way classification, conflating infrastructure errors with test failures):

{
  "m1": { "status": "killed" },
  "m2": { "status": "survived" },
  "m3": { "status": "killed" }
}

In this two-way model, m3 was actually a generator crash (infrastructure error), but it is misclassified as "killed."

Correct (three-way classification distinguishes test failure, test success, and infrastructure error):

{
  "m1": { "status": "killed" },
  "m2": { "status": "survived" },
  "m3": { "status": "error", "error": "generator crashed on malformed IR" }
}

Why This Matters

The mutator uses this three-way classification directly for result status assignment. Without it, the mutation report would either over-count "killed" (inflating test quality metrics) or over-count "survived" (creating false alarms). Both undermine trust in the mutation testing process.

Implement Runner Adapter Interface

The test runner adapter wraps the project's test execution mechanism. It provides a uniform interface that the mutator uses to evaluate each mutation. Without this abstraction, the mutator would need project-specific test invocation logic.

Spec Requirements

Inputs:

Outputs:

Why These Four Outputs

• passed? — The binary signal the mutator needs for killed/survived classification.
• output — Diagnostic text shown in the mutation report for survived and error cases, helping developers understand why a mutation was not caught.
• error text — Distinguishes infrastructure failures (could not start tests) from test failures (tests ran and failed). This drives the three-way classification.
• duration — Enables timeout enforcement and performance reporting.

Examples

Incorrect (returns only pass/fail, losing diagnostic and error context):

{
  "passed": false
}

Correct (returns all four required output fields):

{
  "passed": false,
  "output": "FAIL: expected 200 but got 404 at step 'the API responds with <status>'",
  "error": "",
  "duration": "1.23s"
}

Why This Matters

The runner adapter is the only component that touches the project's actual test framework. Encapsulating this behind a clean interface means the mutator, normal acceptance script, and any future tooling all share the same test invocation mechanism. Changing test frameworks (e.g., switching from pytest to unittest) only requires updating the adapter.

Agent Setup Checklist

When installing the acceptance pipeline in a new project, follow these 15 steps in order. Each step builds on the previous ones. Skipping steps produces an incomplete pipeline that fails when first invoked.

Step-by-Step Installation

Phase 1: Feature and Parser (steps 1-3)

1. Create the feature file. Create features/a-feature.feature with at least one scenario that exercises real project behavior. Use concrete examples — abstract features produce weak tests.

2. Implement the Gherkin parser. Build the gherkin-parser command that reads the supported Gherkin subset and writes JSON IR. Follow the parser rules (command interface, supported syntax, exit codes).

3. Implement JSON IR reader/writer. Build the IR serialization layer used by the parser (writing) and the generator/runtime/mutator (reading). Validate against the IR schema.

Phase 2: Runtime and Handlers (steps 4-5)

4. Implement the acceptance runtime. Build the engine that expands scenarios, applies backgrounds, resolves placeholders, and dispatches steps to handlers. This is the execution core.

5. Implement step handlers. Write a handler for every step text in the feature file. Each handler connects the step's specification language to actual project behavior and assertions.

Phase 3: Generator and Normal Run (steps 6-8)

6. Implement the acceptance generator. Build the acceptance-generator command that reads JSON IR and writes executable tests for the project's test framework.

7. Add the normal acceptance script. Create the convenience script that chains parser, generator, and test runner. Verify it follows the four script requirements (stop on failure, create directories, propagate errors, always regenerate from IR).

8. Run and verify. Execute the normal acceptance script and confirm generated tests pass. This validates the entire parser-to-test-runner chain.

Phase 4: Mutation Testing (steps 9-12)

9. Implement the mutator. Build the gherkin-mutator command using the same parser, IR, generator, and test runner adapter. Implement value mutation rules, deep copy, stable identity, and result classification.

10. Add the mutation script. Create the thin wrapper script that invokes the mutator with the feature file path.

11. Run and inspect. Execute the mutation script and inspect survived mutations. Each survived mutation indicates a test gap — the acceptance test does not verify the specific value.

12. Strengthen tests. Add or improve acceptance scenarios and assertions until important mutations are killed. Not every mutation needs to be killed, but survived mutations should be conscious decisions, not oversights.

Phase 5: Quality and Integration (steps 13-15)

13. Add unit tests. Write unit tests for the parser, generator, runtime, and mutator components. These catch implementation bugs independently of acceptance tests.

14. Add normal acceptance to CI. Add the normal acceptance script to the project's regular verification workflow (e.g., on every push or PR). This ensures the project always satisfies its feature specifications.

15. Add mutation to quality workflow. Add the mutation script to an explicit quality workflow (e.g., nightly or weekly). Mutation testing may be slower than normal verification, so it does not need to run on every push.

Why This Order

The steps follow dependency order:

• The parser must exist before the generator can read IR.
• The runtime and handlers must exist before generated tests can execute.
• The normal run must work before the mutation run can meaningfully test it.
• Unit tests and CI integration come last because they validate what is already working.

Examples

Incorrect (skipping phase 1 -- jumping to generator without a working parser):

# Step 1-3: SKIPPED -- no parser, no IR

# Step 6: Try to run generator with a hand-written JSON file
acceptance-generator hand-written.json acceptance/generated/test.go
# Fails: hand-written.json doesn't match IR schema
# Developer wastes hours debugging generator when the real issue is missing parser

Correct (incremental phases -- verify each phase before moving to the next):

# Phase 1: Parser
gherkin-parser features/a-feature.feature build/acceptance/a-feature.json
# Verify: build/acceptance/a-feature.json matches IR schema

# Phase 2: Runtime + handlers (unit tested independently)

# Phase 3: Generator + normal run
acceptance-generator build/acceptance/a-feature.json \
  acceptance/generated/a-feature_test.go
go test ./acceptance/generated/...
# Verify: tests pass against base feature

# Phase 4: Mutation testing
gherkin-mutator --feature features/a-feature.feature
# Verify: report shows killed/survived/error counts

Why This Matters

Installing a pipeline incrementally — testing each phase before moving to the next — catches problems early. If the parser output is wrong, the generator will fail. If the runtime does not resolve placeholders, generated tests will fail. Discovering these issues in phase order is much easier than debugging the full pipeline end-to-end.

Toggle Boolean Values

Boolean values have exactly two states. The only meaningful mutation is toggling to the opposite value. This directly tests whether the application behaves differently for true vs. false.

Spec Requirements

Condition: Lowercase trimmed is true or false.

Mutation: Replace with the opposite lowercase boolean value.

"true"  -> "false"
"false" -> "true"
"True"  -> "false"
"FALSE" -> "true"

The mutated value is always lowercase ("true" or "false"), regardless of the original casing.

Why Lowercase Output

The IR stores all values as strings. Using consistent lowercase for the mutated boolean avoids case-sensitivity issues downstream. If the original was "TRUE" and the mutation produced "FALSE", a case-insensitive handler might treat both the same way, making the mutation meaningless. Lowercase is the canonical form.

Examples

Incorrect (dithers "true" as a string, producing an unparseable value):

{
  "original": "true",
  "rule": "string-dither",
  "mutated": "troe"
}

Correct (toggles boolean to its opposite in lowercase):

{
  "original": "True",
  "rule": "boolean",
  "mutated": "false"
}

Why This Rule Exists Before Integer

Without this rule, "true" would fall through to string dithering, producing something like "troe". That is a string mutation, not a semantic boolean mutation. A handler that parses "troe" would fail with a parse error, not with a behavioral difference. The boolean rule produces "false", which the handler can parse — and the test should detect the behavioral change.

Why This Matters

Boolean flags control branching in application logic. A mutation from "true" to "false" tests whether the acceptance test actually verifies the behavior associated with that flag. If the test passes with both values, it is not checking the flag's effect — a clear test gap.