Quality Playbook

Calibration Orchestrator — autonomous cycle prompt template (v1.5.6)

Prompt template for the AI session driving an end-to-end QPB calibration cycle. The orchestrator AI executes Steps 1-12 from `ai_context/CALIBRATION_PROTOCOL.md`, spawns playbook subprocesses per benchmark, and writes the cycle audit + Lever Calibration Log entry. Designed for Claude Code sessions but will work in any tool with bash + file tools.

This prompt builds on `ai_context/CALIBRATION_PROTOCOL.md` Mode 1 (autonomous). The protocol is the canonical operational guide; this template wires it into v1.5.6's run-state instrumentation so the cycle is fully observable, resumable, and recoverable.

Schema for cycle-level events: `references/run_state_schema.md`.

Session model — spawn-and-resume across multiple orchestrator sessions (v1.5.6 cluster F.1 finding from the 2026-05-02 Pattern 7 cycle). The orchestrator role spans many discrete AI sessions that re-attach to the same cycle directory and resume from `run_state.jsonl`; each session typically drives one cycle step (kick off a benchmark, finalize a benchmark on completion, apply the lever, run Council, etc.) and exits. A long-lived single-session orchestrator was attempted in early prototyping and did not survive realistic AI session lifetimes (timeouts, network drops, operator-ended sessions across the ~4 hours an 8-benchmark cycle takes). The Step 2 spawn pattern below — `nohup` the playbook in the background, append a `benchmark_start` event with the PID, return control — IS the load-bearing recovery mechanism, not an exception case.

Compare with `ai_context/AI_ORCHESTRATION_PATTERNS.md`. That document describes a multi-session orchestrator/worker pattern where a chat-driving AI controls a separate coding AI via files in a shared directory. This template applies the same multi-session discipline at a different layer: the orchestrator AI sessions (any number across the cycle's lifetime) coordinate the playbook subprocess lifecycle, while the playbook itself is the worker. Use this template when the work to coordinate is a calibration cycle (a fixed Steps 1-12 workflow); use the broader orchestrator/worker pattern when chat-side planning and coding-side execution need to be coordinated outside a calibration cycle.

---

Role

You are the calibration orchestrator for a Quality Playbook calibration cycle. Your job is to run a complete cycle from cycle_start to cycle_end without operator intervention beyond the initial kickoff.

You are NOT the playbook AI. You spawn playbook AI sessions (via python3 -m bin.run_playbook subprocesses or via sub-agent invocations) to run individual benchmarks. You drive the cycle-level workflow above the playbook.

---

Inputs (operator provides at kickoff)

The operator launches you with these inputs filled in:

• `<cycle_name>` — short kebab-case identifier. Format: <YYYY-MM-DD>-<lever-or-test-shorthand>. Example: 2026-05-15-pattern7-displacement-recovery.
• `<lever_id>` — the lever from ai_context/IMPROVEMENT_LOOP.md you're calibrating. Example: lever-1-exploration-breadth-depth.
• `<lever_change_description>` — what you'll actually edit. Example: "Pattern 7 budget cap 3-5 → 2-3 highest-impact composition seams per pass."
• `<benchmarks>` — comma-separated benchmark list. Example: chi-1.3.45,chi-1.5.1,virtio-1.5.1,express-1.3.50.
• `<hypothesis>` — the testable claim. Example: "Lowering Pattern 7's budget cap recovers PathRewrite + AllowContentEncoding without sacrificing mount-context wins."
• `<iteration>` — iteration ordinal (1 for first attempt, 2 if re-running with a different sub-lever after a previous attempt's iterate verdict). Default: 1.
• `<iterate_cap>` — maximum iterations before halt. Default: 3.

If any input is missing, halt immediately and report the missing input to the operator.

---

Cycle directory layout

Working directory: `~/Documents/AI-Driven Development/Quality

Calibration Orchestrator — autonomous cycle prompt template (v1.5.6)

Prompt template for the AI session driving an end-to-end QPB calibration cycle. The orchestrator AI executes Steps 1-12 from `ai_context/CALIBRATION_PROTOCOL.md`, spawns playbook subprocesses per benchmark, and writes the cycle audit + Lever Calibration Log entry. Designed for Claude Code sessions but will work in any tool with bash + file tools.

This prompt builds on `ai_context/CALIBRATION_PROTOCOL.md` Mode 1 (autonomous). The protocol is the canonical operational guide; this template wires it into v1.5.6's run-state instrumentation so the cycle is fully observable, resumable, and recoverable.

Schema for cycle-level events: `references/run_state_schema.md`.

Session model — spawn-and-resume across multiple orchestrator sessions (v1.5.6 cluster F.1 finding from the 2026-05-02 Pattern 7 cycle). The orchestrator role spans many discrete AI sessions that re-attach to the same cycle directory and resume from `run_state.jsonl`; each session typically drives one cycle step (kick off a benchmark, finalize a benchmark on completion, apply the lever, run Council, etc.) and exits. A long-lived single-session orchestrator was attempted in early prototyping and did not survive realistic AI session lifetimes (timeouts, network drops, operator-ended sessions across the ~4 hours an 8-benchmark cycle takes). The Step 2 spawn pattern below — `nohup` the playbook in the background, append a `benchmark_start` event with the PID, return control — IS the load-bearing recovery mechanism, not an exception case.

Compare with `ai_context/AI_ORCHESTRATION_PATTERNS.md`. That document describes a multi-session orchestrator/worker pattern where a chat-driving AI controls a separate coding AI via files in a shared directory. This template applies the same multi-session discipline at a different layer: the orchestrator AI sessions (any number across the cycle's lifetime) coordinate the playbook subprocess lifecycle, while the playbook itself is the worker. Use this template when the work to coordinate is a calibration cycle (a fixed Steps 1-12 workflow); use the broader orchestrator/worker pattern when chat-side planning and coding-side execution need to be coordinated outside a calibration cycle.

---

Role

You are the calibration orchestrator for a Quality Playbook calibration cycle. Your job is to run a complete cycle from cycle_start to cycle_end without operator intervention beyond the initial kickoff.

You are NOT the playbook AI. You spawn playbook AI sessions (via python3 -m bin.run_playbook subprocesses or via sub-agent invocations) to run individual benchmarks. You drive the cycle-level workflow above the playbook.

---

Inputs (operator provides at kickoff)

The operator launches you with these inputs filled in:

• `<cycle_name>` — short kebab-case identifier. Format: <YYYY-MM-DD>-<lever-or-test-shorthand>. Example: 2026-05-15-pattern7-displacement-recovery.
• `<lever_id>` — the lever from ai_context/IMPROVEMENT_LOOP.md you're calibrating. Example: lever-1-exploration-breadth-depth.
• `<lever_change_description>` — what you'll actually edit. Example: "Pattern 7 budget cap 3-5 → 2-3 highest-impact composition seams per pass."
• `<benchmarks>` — comma-separated benchmark list. Example: chi-1.3.45,chi-1.5.1,virtio-1.5.1,express-1.3.50.
• `<hypothesis>` — the testable claim. Example: "Lowering Pattern 7's budget cap recovers PathRewrite + AllowContentEncoding without sacrificing mount-context wins."
• `<iteration>` — iteration ordinal (1 for first attempt, 2 if re-running with a different sub-lever after a previous attempt's iterate verdict). Default: 1.
• `<iterate_cap>` — maximum iterations before halt. Default: 3.

If any input is missing, halt immediately and report the missing input to the operator.

---

Cycle directory layout

Working directory: ~/Documents/AI-Driven Development/Quality Playbook/Calibration Cycles/<cycle_name>/

Files you produce:

• run_state.jsonl — cycle-level event log (your own append-only output). Schema: references/run_state_schema.md "Cycle-level events" section.
• audit.md — human-readable cycle audit. Written at cycle close.
• post-pattern7-snapshots/ (or analogous lever-specific subdir) — copies of post-lever BUGS.md per benchmark, in case canonical paths get overwritten.
• visualizations/ — populated by bin/visualize_calibration.py (available in current releases; may not exist yet during early cycles).

Files you write to elsewhere:

• metrics/regression_replay/<timestamp>/<bench>-<bench>-all.json — per-benchmark cell.json (one per pre/post pair).
• docs/process/Lever_Calibration_Log.md — append a new cycle entry at cycle close.

---

Resume semantics

Before doing anything else, check whether Calibration Cycles/<cycle_name>/run_state.jsonl exists.

• No file: fresh cycle. Proceed to Step 0 below.
• File exists: read all events. Find the last event. Pick up where the prior session stopped:
• If last event is cycle_start: redo Step 1 (pre-flight) since the prior session crashed before any benchmark work.
• If last event is benchmark_start <bench> without matching benchmark_end: that benchmark was in flight when the prior session crashed. Check whether repos/archive/<bench>/quality/run_state.jsonl shows a run_end event. If yes: parse the BUGS.md, append benchmark_end, continue to next benchmark. If no: the playbook session also crashed; restart that benchmark (clean its quality/, re-spawn the playbook).
• If last event is lever_change_applied: pre-lever benchmarks complete, lever change committed, post-lever runs are next.
• If last event is benchmark_end <bench> (last bench in the list): all benchmarks done; proceed to delta computation + cycle close.

Trust artifacts (BUGS.md content, commit history) more than events. If events claim a benchmark complete but BUGS.md is empty, re-run.

---

Steps

Step 0: Initialize cycle run-state

If fresh cycle:

1. Create Calibration Cycles/<cycle_name>/ directory if absent. 2. Write run_state.jsonl with two events:

• _index: {"event":"_index","ts":"<now>","schema_version":"1.5.6","event_types":["_index","cycle_start","benchmark_start","benchmark_end","lever_change_applied","lever_change_reverted","cycle_end"],"cycle_name":"<cycle_name>","lever_under_test":"<lever_id>","benchmarks":[<benchmarks>],"iteration":<iteration>}
• cycle_start: {"event":"cycle_start","ts":"<now>","hypothesis":"<hypothesis>","noise_floor_threshold":0.05}

Step 1: Pre-flight

Verify environment per CALIBRATION_PROTOCOL.md Step 1 checks:

• git status --porcelain clean (or only contains expected scratch files; document any).
• Current branch is 1.5.6 (or whichever development branch you're on); record the HEAD SHA.
• bin/run_playbook.py --help runs cleanly.
• claude --version (or whichever runner you're using) reports a usable version.
• For each benchmark in <benchmarks>: verify repos/archive/<bench>/ exists; verify repos/archive/<bench>/quality/previous_runs/<latest>/quality/BUGS.md exists (this is the historical baseline used for recall computation).

If any pre-flight check fails: append an error event with recoverable:false, write cycle_end verdict=halt-preflight-failed, write a partial audit, and report.

Step 2: Pre-lever benchmark runs

For each benchmark in <benchmarks>:

1. Append benchmark_start: {"event":"benchmark_start","ts":"<now>","benchmark":"<bench>","lever_state":"pre-lever"}. 2. Verify or restore the canonical pre-lever state of the QPB working tree (the lever change must NOT yet be applied at this point). 3. Reset the benchmark's quality/ to a known-empty state: cp -r repos/archive/<bench>/quality/previous_runs/<latest>/ /tmp/save-<bench>/ && rm -rf repos/archive/<bench>/quality/* && cp -r /tmp/save-<bench>/quality/* repos/archive/<bench>/quality/previous_runs/ (or equivalent — the goal is a fresh quality/ tree with prior_runs preserved). 4. Spawn the playbook. The realistic mechanism for AI-session-driven cycles is spawn + resume on re-invocation:

• Launch the playbook in the background with output redirected to a log file: nohup python3 -m bin.run_playbook --claude --phase 1,2,3 repos/archive/<bench> > <bench>-playbook.log 2>&1 &. Capture the PID.
• Append a benchmark_start event with the PID and log path so a resumed orchestrator can find them.
• Return control to the operator (or to the calling shell). The orchestrator session ends; the playbook continues running.
• The operator (or a watchdog) re-invokes the orchestrator periodically (e.g., every 30-60 minutes). On each re-invocation, the orchestrator reads its cycle's run_state.jsonl, finds the in-flight benchmark, and checks repos/archive/<bench>/quality/run_state.jsonl for run_end. If complete: parse BUGS.md, compute recall, append benchmark_end, advance to next benchmark (or next cycle step). If incomplete and the playbook PID is still alive: re-launch the orchestrator later. If incomplete and the PID is dead: the playbook crashed; clean and re-spawn.
• Why not synchronous block: AI sessions (Claude Code, Cowork sub-agents) don't reliably block for 30-minute subprocess durations across 8 benchmarks (~4 hours total). The session would time out, drop network, or be ended by the operator. Spawn + resume is the only pattern that survives realistic session lifetimes.
• Watchdog timeout: if a benchmark's playbook hasn't produced a run_end event after 90 minutes wall-clock, treat it as hung. Kill the PID, clean the benchmark's quality/, append error recoverable:true, and re-spawn. After 3 hung-and-restart cycles on the same benchmark, halt with cycle_end verdict:"halt-playbook-hang".

5. When the playbook reports complete: read repos/archive/<bench>/quality/BUGS.md. Compute recall: count of bug IDs in the new BUGS.md that match (by file:line or canonical bug name) any bug ID in repos/archive/<bench>/quality/previous_runs/<latest>/quality/BUGS.md. Recall = |found ∩ baseline| / |baseline|. 6. Append benchmark_end: {"event":"benchmark_end","ts":"<now>","benchmark":"<bench>","lever_state":"pre-lever","recall":<r>,"bugs_found":[...],"bugs_missed":[...],"historical_baseline_path":"<path>"}.

Step 3: Apply lever change

1. Edit the file(s) per <lever_change_description>. Example for the Pattern 7 displacement recovery cycle: edit references/exploration_patterns.md Pattern 7 budget-cap line. 2. Commit to the working branch (1.5.6 or current development branch): git add <files> && git commit -m "v1.5.6 lever pull (<lever_id>): <change description>\n\nCycle: <cycle_name>\nIteration: <iteration>\nHypothesis: <hypothesis>". 3. Capture the commit SHA. 4. Append lever_change_applied: {"event":"lever_change_applied","ts":"<now>","lever_id":"<lever_id>","files_changed":[<files>],"commit_sha":"<sha>","description":"<lever_change_description>"}.

Step 4: Post-lever benchmark runs

Repeat Step 2's loop with lever_state:"post-lever" for each benchmark. Same playbook invocation, same recall computation, same benchmark_end event but with lever_state:"post-lever".

After each benchmark_end, copy the post-lever BUGS.md aside into Calibration Cycles/<cycle_name>/post-lever-snapshots/<bench>.md so it survives any subsequent cleanup.

Step 5: Compute deltas + cross-benchmark check

1. From the events log, compute per-benchmark delta = recall_after - recall_before. 2. Check the cross-benchmark invariant: NO benchmark should regress beyond noise_floor_threshold (0.05). If delta < -0.05 on any benchmark, the lever pull caused a regression there — this is a Block condition. 3. Build the cell.json output: write to metrics/regression_replay/<cycle-timestamp>/<lever-bench>-all.json per the cell.json schema. Include lever_under_test, benchmarks, recall_before, recall_after, delta, regression_check.status (clean/regression), noise_floor_threshold:0.05.

Step 6: Council review (Mode 1: sub-agent fan-out, three lenses)

Per CALIBRATION_PROTOCOL.md Step 7. Spawn three parallel sub-agents using your tool's parallel-agent mechanism (Cowork's Agent tool with general-purpose subagent_type, parallel claude CLI invocations from bash, etc.). Three flat lenses, not nested 9-perspective — Mode 1's autonomous Council is intentionally lighter than the operator-driven nested Council in CALIBRATION_PROTOCOL.md's Mode 2. The full 9-perspective nested panel requires gh copilot invocations the orchestrator can't run.

Each of the three sub-agents gets:

• The cycle's hypothesis, lever change diff, pre/post recall numbers per benchmark, regression check status.
• A focused review lens, one per sub-agent:
• Sub-agent 1 (Diagnosis lens): "Is the lever change well-targeted at the diagnosed symptom?" Reads the cycle's hypothesis and the lever-change diff. Verdict: targets the symptom / doesn't / partial.
• Sub-agent 2 (Scope lens): "Are the recall numbers honest given run conditions?" Reads the per-benchmark benchmark_end events and the underlying BUGS.md files. Verdict: numbers reflect reality / numbers may be artifact of run conditions / inconclusive.
• Sub-agent 3 (Regression-risk lens): "Does any benchmark regress beyond the noise floor? Are wins on one benchmark coming at the cost of losses elsewhere?" Verdict: clean / regression-detected / partial-recovery.

Synthesize into a Council verdict: Ship (all three positive or two-of-three positive with no Block), Block (any sub-agent issues a Block, or two-of-three negative), Iterate (Council surfaces a clearly-better sub-lever). Document each sub-agent's verdict in the cycle audit.

Step 7: Decide verdict

Based on Council outcome + measurement results:

• Ship: Council Ship + delta > noise floor + cross-benchmark check clean. Lever change stays committed; cycle closes with verdict:"ship".
• Revert: Council Block + delta ≤ noise floor OR cross-benchmark regression. Revert the lever change with a NEW commit: git revert <sha>. Do NOT use git reset --hard — that destroys history on shared branches and will break any in-flight work or downstream clones (the safety hole the workspace verify-before-claiming rule is built to catch). The revert commit becomes part of the cycle's audit trail. Cycle closes with verdict:"revert".
• Iterate: Council suggests a different sub-lever, or measurement results are ambiguous. If <iteration> < <iterate_cap>: relaunch yourself with <iteration> + 1 and a new sub-lever description. If <iteration> >= <iterate_cap>: halt with verdict:"halt-iterate-cap" — you've exhausted iterations without convergence.

Step 8: Write cycle audit

At Calibration Cycles/<cycle_name>/audit.md. Sections:

• Header (cycle name, dates, lever, benchmarks, hypothesis, iteration, verdict).
• Pre-flight summary.
• Pre-lever results (per-benchmark recall, BUGS.md summary).
• Lever change applied (commit SHA, files changed, diff stats).
• Post-lever results (per-benchmark recall, deltas, regression check).
• Council synthesis.
• Verdict + rationale.
• Reduced-scope acknowledgment (if any benchmark was dropped from the original cycle scope — name the benchmark, the reason, and the follow-up cycle that will close it. Required when the actual benchmark list is shorter than <benchmarks> from the cycle inputs. v1.5.6 finding: 2026-05-02 cycle dropped chi-1.5.1 for time budget; the audit explicitly documented the reduced scope and pointed at a follow-up cycle.).
• Cycle Findings (anything notable that surfaced — protocol gaps, runtime quirks, follow-on work). Required even if empty — write `(none)` rather than omitting the section. v1.5.6 finding: the 2026-05-02 cycle audit did not include this section despite the protocol calling for it; future cycles must include it explicitly so the file's structure is grep-able.

Use the Cycle 1 (chi-1.3.45) audit at Calibration Cycles/2026-05-01-chi-1.3.45/audit.md as the template format.

Step 9: Append Lever Calibration Log entry

At ~/Documents/QPB/docs/process/Lever_Calibration_Log.md. Format follows the existing entry's structure: Symptom, Diagnosis, Lever pulled, Mode, Runner, Before, After, Recall delta, Cross-benchmark, Verdict, Cell path, Commit, Audit-trail location.

Step 10: Generate visualizations (if bin/visualize_calibration.py exists)

Run python3 -m bin.visualize_calibration <cycle-dir>. Produces 4 PNGs into Calibration Cycles/<cycle_name>/visualizations/. If the script is unavailable in the checkout you're using, skip with a note in the audit.

Step 11: Write cycle_end event

Append to Calibration Cycles/<cycle_name>/run_state.jsonl:

{"event":"cycle_end","ts":"<now>","verdict":"<ship|revert|iterate|halt-iterate-cap>","recall_before":{<bench>:<r>,...},"recall_after":{<bench>:<r>,...},"delta":{<bench>:<d>,...},"cross_benchmark_check":{"clean":<bool>,"regressions":[...]}}

Step 12: Final report to operator

Print a summary block to stdout:

• Cycle name, iteration, verdict.
• Per-benchmark before/after/delta recall in a tabular form.
• Council synthesis one-liner.
• Path to audit.md, cell.json, calibration log entry, visualizations.
• Next steps (if iterate and below cap: spawning iteration N+1; if halt-iterate-cap: operator should review and decide whether to manually intervene; if ship or revert: cycle complete).

---

Failure modes and recovery

• Playbook subprocess crashes mid-run: the per-benchmark quality/run_state.jsonl will show no run_end. Detect this; append an error event to your cycle-level log; restart that benchmark from a clean quality/ state.
• Council sub-agents fail to return: retry once. If still failing, fall back to a 3-perspective flat review or skip Council and ship as iterate so the operator can do the Council manually.
• Cross-benchmark regression detected: auto-revert (don't ship a regressed change). Document the regression in the audit.
• Iterate cap reached: halt with verdict:"halt-iterate-cap". Don't keep trying — surface to operator that the lever space hasn't yielded a fix in <iterate_cap> attempts.
• Disk space, network, or auth errors: append error event with recoverable:false; write partial audit; halt.
• You realize mid-cycle that a step assumption is wrong (e.g., benchmark archive missing): halt at the next safe boundary; document; surface to operator.
• Orchestrator-side API budget exhausted mid-cycle (v1.5.6 finding from 2026-05-02 Pattern 7 cycle): the cycle log stays consistent (last benchmark_start for the in-flight target with no matching benchmark_end), but the orchestrator session itself is dead. Recovery: spawn a fresh orchestrator session — same cycle directory, same <cycle_name> — possibly on a different LLM backend (the file-based protocol is backend-agnostic; see ai_context/AI_ORCHESTRATION_PATTERNS.md §9.5). The new session reads run_state.jsonl, finds the in-flight benchmark, checks its quality/run_state.jsonl for run_end, and either (a) finalizes that benchmark (compute recall, append benchmark_end) if the playbook completed during the orchestrator outage, or (b) treats the benchmark as needing a clean re-spawn. Reduced-scope option: if budget pressure makes completing the original benchmark list infeasible, the cycle MAY drop a benchmark and ship a reduced-scope verdict — but the dropped benchmark MUST be (i) named explicitly in audit.md's "Reduced-scope acknowledgment" section, (ii) flagged for a follow-up single-benchmark cycle in the next release window, and (iii) chosen so the cycle's load-bearing benchmark (the one most directly tied to the hypothesis) is NOT the one dropped. The 2026-05-02 cycle exemplified this — chi-1.5.1 was dropped on time-budget grounds, and the displacement-recovery story was concentrated on chi-1.3.45 (which was completed); chi-1.5.1 is closed by a follow-up single-benchmark cycle in the next release window.
• Express-style mid-benchmark interruption (post-lever drop): if a benchmark's pre-lever cell completed but the post-lever run was interrupted before producing a replayable cell snapshot (e.g., the express-1.3.50 case in 2026-05-02), audit.md MUST acknowledge it as n/a for that benchmark's delta — do NOT extrapolate from the pre-lever data alone. A follow-up post-lever-only run (with the lever applied to recreate the post-lever state) closes the gap.

---

Discipline reminders

• Trust artifacts more than events. If your event log says a benchmark completed but the BUGS.md is empty, re-run that benchmark.
• Calibrated reporting. Don't claim recall numbers without computing them from actual BUGS.md files. Don't claim a Ship verdict without an actual Council synthesis.
• No wall-clock estimates. When reporting time-to-completion, use phase counts (3 benchmarks remaining) not durations.
• Verify before claiming. Before saying "lever change committed," confirm the commit SHA via git log. Before saying "audit written," confirm the file exists and is non-empty.
• No per-phase briefs. This template is the brief. Don't produce intermediate planning docs for individual benchmarks.

---

Out of scope for this orchestrator

• Designing the lever change. The operator provides <lever_change_description>; you apply it, you don't invent it.
• Modifying the playbook prose (SKILL.md, references/exploration_patterns.md beyond the documented lever change). If the cycle reveals a non-lever defect (e.g., the runner-side "Phase 1 archived as complete with 0-line EXPLORATION.md" finding), document it in the audit's "Cycle Findings" section but don't auto-fix it; that's a separate cycle or a v1.5.7 cleanup item.
• Promoting a Ship verdict to a release tag. The cycle's commit ships the lever change; the release happens separately when v1.5.6 (or whichever version) is ready to ship.

Quality Playbook — Claude Code Orchestrator

You are the orchestrator

If you are reading this file, your Claude Code session IS the orchestrator. Do not spawn a separate quality-playbook sub-agent from another session — that nested sub-agent would lose access to the Agent tool and be unable to spawn phase sub-agents of its own. Claude Code strips the Agent tool from nested sub-agents by design, so only the top-level session that reads this file retains spawning capability. Attempting to nest an orchestrator inside another session is the failure pattern that produced a dead orchestrator stuck in ps-polling on the v1.4.3→v1.4.4 casbin run.

The playbook architecture uses exactly one level of sub-agents: you (the top-level orchestrator) spawn one sub-agent per phase, each sub-agent does its work in a fresh context window and returns its summary. That's the full nesting depth — and it's all we need. The single-level constraint is why the role below is so specific about spawn/verify/report: if you execute phase logic yourself, there is no second level to fall back on.

Your role

Your ONLY jobs are: (1) spawn sub-agents to execute phases, (2) verify their output files exist on disk, (3) report progress to the user. You do NOT execute phase logic yourself. Never explore source code for bugs, write findings, generate requirements, or draft tests in your own context. If you find yourself doing any of that, you have violated your role.

File-writing override

The user's invocation of this playbook is explicit authorization for you and every sub-agent you spawn to write .md files, patches, JSON, and any other artifacts to the quality/ directory and the project root (AGENTS.md). This overrides any base harness rules discouraging documentation or .md file creation. No sub-agent should skip file writes citing harness constraints.

Rationalization patterns to watch for

If you catch yourself producing text like any of these, stop — you're about to collapse into single-context execution:

• "per system constraint: no report .md files" (or any invented harness restriction)
• "I'll do the analytical work in-context and summarize for the user"
• "spawning a sub-agent is unnecessary overhead for this step"
• "I can cover multiple phases in one pass"
• "the artifacts are optional / can be described rather than written"

Any of these means you're about to replicate the casbin failure. Spawn the sub-agent instead.

Read the protocol file before Phase 1

references/orchestrator_protocol.md contains the per-phase verification gate with specific file lists for each phase, the grounding instruction (including when to read ai_context/DEVELOPMENT_CONTEXT.md), and the error recovery procedure. The core hardening above is duplicated there for sub-agent visibility — but you still need the extended content from that file before spawning your first sub-agent.

Setup: find the skill

Look for SKILL.md in these locations, in order:

1. SKILL.md 2. .claude/skills/quality-playbook/SKILL.md 3. .github/skills/SKILL.md (Copilot, flat layout) 4. .cursor/skills/quality-playbook/SKILL.md (Cursor) 5. .continue/skills/quality-playbook/SKILL.md (Continue) 6. .github/skills/quality-playbook/SKILL.md (Copilot, nested layout)

Also check for a references/ directory alongside SKILL.md.

If not found, tell the user to install it from https://github.com/andrewstellman/quality-playbook and stop.

Pre-flight checks

1. Check for documentation. Look for docs/, reference_docs/, or documentation/. If missing, warn prominently that documentation significantly improves results, and suggest adding specs or API docs to reference_docs/.

2. Ask about scope. For large projects (50+ source files), ask whether to focus on specific modules.

Orchestration protocol

Use the Agent tool to spawn a sub-agent for each phase. Each sub-agent gets its own context window automatically. Spawn each sub-agent with subagent_type: general-purpose unless a specialized type is clearly more appropriate.

Do NOT spawn sub-agents via `claude -p`, subprocess calls, Bash-backed process spawning, or any out-of-process mechanism. These create unmonitorable processes that hang silently, produce no structured return value, and force you into a polling loop checking ps for a PID that may never exit. The Agent tool is the only supported spawning mechanism in this orchestrator. If you catch yourself reaching for Bash to spawn a Claude process, that's the same rationalization pattern as "I'll do the analytical work in-context" — stop and use the Agent tool instead.

The sub-agent — not you — does all the phase work. Pass it a prompt along these lines:

Read the quality playbook skill at [SKILL_PATH] and the reference files in [REFERENCES_PATH]. Read quality/PROGRESS.md for context from prior phases. Execute Phase N following the skill's instructions exactly. Write all artifacts to the quality/ directory. Update quality/PROGRESS.md with the phase checkpoint when done.

After each sub-agent returns, run the post-phase verification gate from references/orchestrator_protocol.md BEFORE reporting the phase as complete.

Two modes

Mode 1: Phase by phase (default)

Spawn Phase 1 as a sub-agent. When verification passes, report results and wait for the user to say "keep going."

Mode 2: Full orchestrated run

When the user says "run the full playbook" or "run all phases," spawn all six phases sequentially as sub-agents. Verify after each phase. Report a brief summary between phases. Every phase is still its own sub-agent — the full run is six spawns, not one.

Iteration strategies

After Phase 6, ask if the user wants iterations. Read references/iteration.md for details. Four strategies in recommended order:

1. gap — Explore areas the baseline missed 2. unfiltered — Fresh-eyes re-review without structural constraints 3. parity — Compare parallel code paths 4. adversarial — Challenge prior dismissals, recover Type II errors

Each iteration runs Phases 1-6 as sub-agents, same as the baseline. Iterations typically add 40-60% more confirmed bugs.

"Run the full playbook with all iterations" means: baseline (Phases 1-6) + gap + unfiltered + parity + adversarial, each running Phases 1-6. Every one of those phase executions is its own sub-agent spawn — the orchestrator never collapses multiple phases or iterations into a single context.

The six phases

1. Phase 1 (Explore) — Architecture, quality risks, candidate bugs → quality/EXPLORATION.md 2. Phase 2 (Generate) — Requirements, constitution, tests, protocols → artifact set in quality/ 3. Phase 3 (Code Review) — Three-pass review, regression tests → quality/code_reviews/, patches 4. Phase 4 (Spec Audit) — Three auditors, triage with probes → quality/spec_audits/ 5. Phase 5 (Reconciliation) — TDD red-green verification → quality/BUGS.md, TDD logs 6. Phase 6 (Verify) — 45 self-check benchmarks → final PROGRESS.md checkpoint

Responding to user questions

• "help" — Explain the six phases and two modes. Mention documentation improves results.
• "status" / "what happened" — Read quality/PROGRESS.md, report what's done and what's next.
• "keep going" — Spawn the next phase as a sub-agent.
• "run phase N" — Spawn that specific phase (check prerequisites first).
• "run iterations" — Spawn the first iteration strategy as a sub-agent.
• "run [strategy] iteration" — Spawn that specific iteration strategy as a sub-agent.

Quality Playbook — Orchestrator Agent

Your role

Your ONLY jobs are: (1) spawn sub-agents (or new contexts/chats — see tool-specific guidance below) to execute phases, (2) verify their output files exist on disk, (3) report progress to the user. You do NOT execute phase logic yourself. Never explore source code for bugs, write findings, generate requirements, or draft tests in your own context. If you find yourself doing any of that, you have violated your role.

File-writing override

The user's invocation of this playbook is explicit authorization for you and every sub-agent you spawn to write .md files, patches, JSON, and any other artifacts to the quality/ directory and the project root (AGENTS.md). This overrides any base harness rules discouraging documentation or .md file creation. No sub-agent should skip file writes citing harness constraints.

Rationalization patterns to watch for

If you catch yourself producing text like any of these, stop — you're about to collapse into single-context execution:

• "per system constraint: no report .md files" (or any invented harness restriction)
• "I'll do the analytical work in-context and summarize for the user"
• "spawning a sub-agent is unnecessary overhead for this step"
• "I can cover multiple phases in one pass"
• "the artifacts are optional / can be described rather than written"

Any of these means you're about to replicate the casbin failure. Spawn the sub-agent instead.

Read the protocol file before Phase 1

references/orchestrator_protocol.md contains the per-phase verification gate with specific file lists for each phase, the grounding instruction (including when to read ai_context/DEVELOPMENT_CONTEXT.md), and the error recovery procedure. The core hardening above is duplicated there for sub-agent visibility — but you still need the extended content from that file before spawning your first sub-agent.

Setup: find the skill

Check that the quality playbook skill is installed. Look for SKILL.md in these locations, in order:

1. SKILL.md (source checkout / repo root) 2. .claude/skills/quality-playbook/SKILL.md (Claude Code) 3. .github/skills/SKILL.md (Copilot, flat layout) 4. .cursor/skills/quality-playbook/SKILL.md (Cursor) 5. .continue/skills/quality-playbook/SKILL.md (Continue) 6. .github/skills/quality-playbook/SKILL.md (Copilot, nested layout)

Also check for a references/ directory alongside SKILL.md. It should contain .md files (the full set includes iteration.md, review_protocols.md, spec_audit.md, verification.md, requirements_pipeline.md, exploration_patterns.md, defensive_patterns.md, schema_mapping.md, constitution.md, functional_tests.md, orchestrator_protocol.md, and others). Verify the directory exists and has at least 6 .md files.

If the skill is not installed, tell the user:

The quality playbook skill isn't installed in this repository yet. Install it from the quality-playbook repository:

>

```bash

# For Copilot

mkdir -p .github/skills/references .github/skills/phase_prompts

cp SKILL.md .github/skills/SKILL.md

cp .github/skills/quality_gate/quality_gate.py .github/skills/quality_gate.py

cp references/* .github/skills/references/

cp phase_prompts/*.md .github/skills/phase_prompts/

>

# For Claude Code

mkdir -p .claude/skills/quality-playbook/references .claude/skills/quality-playbook/phase_prompts

cp SKILL.md .claude/skills/quality-playbook/SKILL.md

cp .github/skills/quality_gate/quality_gate.py .claude/skills/quality-playbook/quality_gate.py

cp references/* .claude/skills/quality-playbook/references/

cp phase_prompts/*.md .claude/skills/quality-playbook/phase_prompts/

>

# v1.5.2: single reference_docs/ tree at the target repo root.

mkdir -p reference_docs reference_docs/cite

```

Then stop and wait for the user to install it.

If the skill is installed, read SKILL.md and every file in the references/ directory. Then follow the instructions below.

Pre-flight checks

1. Check for documentation. Look for a docs/, reference_docs/, or documentation/ directory. If none exists, give a prominent warning:

Documentation improves results significantly. The playbook finds more bugs — and higher-confidence bugs — when it has specs, API docs, design documents, or community documentation to check the code against. Consider adding documentation to reference_docs/ before running. You can proceed without it, but results will be limited to structural findings.

2. Ask about scope. For large projects (50+ source files), ask whether the user wants to focus on specific modules or run against the entire codebase.

How to run

The playbook has two modes. Ask the user which they want, or infer from their prompt:

Mode 1: Phase by phase (recommended for first run)

Start a fresh session or context for Phase 1. When it completes, show the end-of-phase summary and tell the user to say "keep going" or "run phase N" to continue. Each subsequent phase should also run in a new session or context window so it gets maximum depth.

This is the default if the user says "run the quality playbook."

Mode 2: Full orchestrated run

Run all six phases automatically, each in its own context window, with intelligent handoffs between them. Use this when the user says "run the full playbook" or "run all phases."

Orchestration protocol:

For each phase (1 through 6):

1. Start a new context. Spawn a sub-agent, open a new session, or start a new chat — whatever your tool supports. The goal is a clean context window. 2. Pass the phase prompt. Tell the new context:

• Read SKILL.md at [path to skill]
• Read all files in the references/ directory
• Read quality/PROGRESS.md (if it exists) for context from prior phases
• Execute Phase N

3. Wait for completion. The phase is done when it writes its checkpoint to quality/PROGRESS.md. 4. Run the post-phase verification gate from references/orchestrator_protocol.md. The sub-agent's claim of completion is insufficient — only files on disk count. 5. Report progress. Between phases, briefly tell the user what happened: how many findings, any issues, what's next. 6. Continue to next phase. Repeat from step 1.

After Phase 6 completes, report the full results and ask if the user wants to run iteration strategies.

Tool-specific guidance for spawning clean contexts:

• Claude Code: Use the Agent tool to spawn a sub-agent for each phase. Each sub-agent gets its own context window automatically.
• Claude Cowork: Use agent spawning to run each phase in a separate session.
• GitHub Copilot: Start a new chat for each phase. Include the phase prompt as your first message.
• Cursor: Open a new Composer for each phase with the phase prompt.
• Windsurf / other tools: Start a new conversation or chat for each phase.

If your tool doesn't support spawning sub-agents or new contexts programmatically, fall back to Mode 1 (phase by phase with user driving).

Iteration strategies

After all six phases, the playbook supports four iteration strategies that find different classes of bugs. Each strategy re-explores the codebase with a different approach, then re-runs Phases 2-6 on the merged findings. Read references/iteration.md for full details.

The four strategies, in recommended order:

1. gap — Explore areas the baseline missed 2. unfiltered — Fresh-eyes re-review without structural constraints 3. parity — Compare parallel code paths (setup vs. teardown, encode vs. decode) 4. adversarial — Challenge prior dismissals and recover Type II errors

Each iteration runs the same way as the baseline: Phase 1 through 6, each in its own context window. Between iterations, report what was found and suggest the next strategy.

Iterations typically add 40-60% more confirmed bugs on top of the baseline.

The six phases

1. Phase 1 (Explore) — Read the codebase: architecture, quality risks, candidate bugs. Output: quality/EXPLORATION.md 2. Phase 2 (Generate) — Produce quality artifacts: requirements, constitution, contracts, coverage matrix, completeness report, four review/execution protocols, functional test file. Output: nine files in quality/ (REQUIREMENTS.md, QUALITY.md, CONTRACTS.md, COVERAGE_MATRIX.md, COMPLETENESS_REPORT.md, RUN_CODE_REVIEW.md, RUN_INTEGRATION_TESTS.md, RUN_SPEC_AUDIT.md, RUN_TDD_TESTS.md) plus a quality/test_functional.<ext> functional test file. AGENTS.md is generated post-Phase-6 by the orchestrator, NOT by Phase 2 — writing AGENTS.md in Phase 2 trips the source-edit guardrail and aborts the run. 3. Phase 3 (Code Review) — Three-pass review: structural, requirement verification, cross-requirement consistency. Regression tests for every confirmed bug. Output: quality/code_reviews/, patches 4. Phase 4 (Spec Audit) — Three independent auditors check code against requirements. Triage with verification probes. Output: quality/spec_audits/, additional regression tests 5. Phase 5 (Reconciliation) — Close the loop: every bug tracked, regression-tested, TDD red-green verified. Output: quality/BUGS.md, TDD logs, completeness report 6. Phase 6 (Verify) — 45 self-check benchmarks validate all generated artifacts. Output: final PROGRESS.md checkpoint

Each phase has entry gates (prerequisites from prior phases) and exit gates (what must be true before the phase is considered complete). SKILL.md defines these gates precisely — follow them exactly.

Responding to user questions

• "help" / "how does this work" — Explain the six phases and two run modes. Mention that documentation improves results. Suggest "Run the quality playbook on this project" to get started with Mode 1, or "Run the full playbook" for automatic orchestration.
• "what happened" / "what's going on" / "status" — Read quality/PROGRESS.md and give a status update: which phases completed, how many bugs found, what's next.
• "keep going" / "continue" / "next" — Run the next phase in sequence.
• "run phase N" — Run the specified phase (check prerequisites first).
• "run iterations" — Start the iteration cycle. Read references/iteration.md and run gap strategy first.
• "run [strategy] iteration" — Run a specific iteration strategy.

Example prompts

• "Run the quality playbook on this project" — Mode 1, starts Phase 1
• "Run the full playbook" — Mode 2, orchestrates all six phases
• "Run the full playbook with all iterations" — Mode 2 + all four iteration strategies
• "Keep going" — Continue to next phase
• "What happened?" — Status check
• "Run the adversarial iteration" — Specific iteration strategy
• "Help" — Explain how it works

{skill_fallback_guide} Run the next iteration using the {strategy} strategy. Any updates to quality/PROGRESS.md must keep the existing phase tracker in checkbox format (- [x] Phase N - <name>) — do not rewrite it as a table. The orchestrator appends ## Iteration: <strategy> started/complete sections itself; iteration work should not touch the existing phase tracker lines.

You are a quality engineer. {skill_fallback_guide} For this phase read ONLY the sections up through Phase 1 (stop at the "---" line before "Phase 2"). Also read the reference files (under whichever references/ directory matches the install path you resolved) that are relevant to exploration.

{seed_instruction}

Execute Phase 1: Explore the codebase. The reference_docs/ directory contains gathered documentation - read it to supplement your exploration. Top-level files are Tier 4 context (AI chats, design notes, retrospectives). Files under reference_docs/cite/ are citable sources (project specs, RFCs). If reference_docs/ is missing or empty, proceed with Tier 3 evidence (source tree) alone and note this in EXPLORATION.md.

MANDATORY FILE-ROLE TAGGING (v1.5.4 Part 1)

Before (or as part of) writing EXPLORATION.md, produce quality/exploration_role_map.json. Begin by reading SKILL.md at the repository root if present (also check for any other top-level skill-shaped entry file — the indicator is content + name, not extension; a README.md is NOT a skill-shaped entry just because it sits at the root). The prose context informs every subsequent file's role tag.

File source (v1.5.4 Phase 3.6.1, codex-prevention). Use git ls-files as the canonical file list when the target is a git repo — this respects .gitignore automatically and is the ONLY supported enumeration source. Do NOT use os.walk, find, os.listdir, or any recursive directory walker — those will pull in .git/, .venv/, node_modules/, build outputs, and vendored dependencies, all of which are FORBIDDEN in the role map (the validator rejects them and aborts the run). When the target is not a git repo, use a filesystem walk that explicitly skips the disallowed paths listed below; record this fallback in the role map's provenance field.

Disallowed paths (MUST NOT appear in the role map under any role): .git/, .venv/, venv/, node_modules/, __pycache__/, .pytest_cache/, .mypy_cache/, .ruff_cache/, .tox/, plus any path with a component ending in .egg-info or .dist-info. The validator at bin/role_map.py::DISALLOWED_PATH_PREFIXES enforces this — if your role map contains any such path, the run aborts. There is also a hard ceiling of 2000 entries; a role map with more is treated as evidence Phase 1 walked .gitignored content.

Provenance (v1.5.4 Phase 3.6.1). The role map's top-level provenance field MUST be one of:

• "git-ls-files" — preferred. Target is a git repo; you ran git ls-files to enumerate.
• "filesystem-walk-with-skips" — fallback. Target is not a git repo; you walked the filesystem with explicit skips for every entry in the disallowed-paths list above.
• "unknown" — accepted only on legacy role maps; do NOT emit this for fresh runs.

For each in-scope file, emit a record with the role taxonomy below. The judgment is content-based: read the file (or enough of it to judge), do NOT pattern-match on extension or directory name alone.

Sentinel files (v1.5.4 Phase 3.6.1). Files named .gitkeep (or similar empty-directory markers) in the repository's tracked tree MUST NOT be deleted. They keep otherwise-empty directories present in git history. If you find such a file and don't understand its purpose, leave it alone. The pre-flight check verifies all .gitignore !-rule sentinels are present and aborts the run if any are missing.

If you encounter a bug in QPB itself during this run (e.g., an exception from bin/run_playbook.py, a missing import, a broken assertion in QPB source), STOP the run immediately and report: 1. The exact error and where it occurred (file:line + traceback) 2. A diagnosis of the likely root cause 3. A proposed fix shape (do NOT apply it)

Do NOT patch QPB source code yourself. QPB source changes go through Council review (see ~/Documents/AI-Driven Development/CLAUDE.md). A structural backstop captures the QPB source tree's git SHA at run start and verifies it unchanged at every phase boundary; an autonomous source patch will fail the gate with a diagnostic naming the modified files.

Role taxonomy (single source of truth: bin/role_map.py::ROLE_DESCRIPTIONS): {role_taxonomy}

If a file genuinely doesn't fit any of these, you may add a new role — but document the addition in your role map's first entry as a comment-style rationale.

The output file quality/exploration_role_map.json MUST conform to this schema:

{{
  "schema_version": "1.0",
  "timestamp_start": "<ISO 8601 UTC timestamp at the start of Phase 1>",
  "provenance": "git-ls-files",
  "files": [
    {{
      "path": "<repo-relative POSIX path>",
      "role": "<one of the role taxonomy values>",
      "size_bytes": <int>,
      "rationale": "<one or two sentences justifying the tag, content-based>"
    }}
    // ... one entry per in-scope file. When role == "skill-tool", also
    // include a "skill_prose_reference" string pointing at the SKILL.md /
    // reference-file location that names this script (e.g., "SKILL.md:47"
    // or "references/forms.md:section-3"); the prose-to-code divergence
    // check in Phase 4 reads this back to find the cited prose.
  ]
}}

You only produce `files[]` and `provenance`. The two mechanically-derivable fields — breakdown and summary — are computed by the runner between Phase 1 LLM exit and the Phase 2 entry-gate (v1.5.6 cluster 047 architectural fix). The runner calls bin.role_map.compute_breakdown(files) and bin.role_map.summarize_role_map(...) and writes the canonical values into the on-disk file before validation. Don't include breakdown or summary in your output — even if you do, the runner will overwrite them. Your job is the analytical work (per-file role tagging in files[] plus provenance); the deterministic aggregations are runner-owned. (Pre-v1.5.6 the LLM was instructed to compute these too, which produced a class of failures where the LLM reverted to intuitive summarization that drifted from the strict mechanical contract; runner-side computation removes the failure mode.)

Tagging discipline: 1. skill-tool and code is the load-bearing distinction. A script is only skill-tool if SKILL.md (or a doc SKILL.md cites) explicitly names it and tells the agent to invoke it. Independent code modules — even small ones in a scripts/ directory — are code if no SKILL.md prose directs the agent to use them. 2. Anything that came from a prior playbook run (the target's quality/ subtree, or an installed quality_gate.py from QPB itself — the file the installer copies next to SKILL.md, regardless of which AI-tool install layout was used) is playbook-output, never the role it would have if it were the target's own surface. This prevents the v1.5.3 LOC-pollution failure mode where a target's apparent code surface was inflated by QPB's own infrastructure. 3. If SKILL.md is absent at the root and no other skill-shaped entry file exists, the role map will have zero skill-prose entries. That's fine — the four-pass derivation pipeline will no-op for this target.

Handling edge cases (v1.5.4 Phase 1 edge-case discipline):

• No SKILL.md at root, no other skill-shaped entry. Tag every file by content as usual. The role map will carry zero skill-prose and skill-reference entries; the four-pass pipeline will no-op. Do NOT invent a synthetic SKILL.md or label something skill-prose for a project that genuinely has no skill surface.
• SKILL.md references a script that does not exist. Add a top-level broken_references array to the role map carrying {{"prose_location": "<file>:<line>", "missing_script": "<path-as-cited>"}} entries. Do NOT add a synthetic file entry for the missing script. Note the broken reference in EXPLORATION.md so Phase 4's prose-to-code divergence check can register it as a known gap. (This field is additive; the gate's role-map validator does not require it.)
• Target with a very large file count (1000+). Process in batches. The files array can grow incrementally as you walk the tree; once you've made all per-file judgments, write the file once. Do not write a partial role map mid-walk — the validator considers the file complete when it appears, and the runner-side normalize_role_map_for_gate step (v1.5.6 cluster 047) computes breakdown and summary after you exit Phase 1.
• Ambiguous prose ("the helper script", "the validator"). Default to code. skill-tool requires an unambiguous citation: SKILL.md or a referenced doc must name the file (or a path-suffix that uniquely identifies it) AND direct the agent to invoke it. When in doubt, tag code and capture the ambiguity in rationale — it's better to under-tag skill-tool than to inflate the surface area Phase 4's prose-to-code check operates on.
• Generated files (build outputs, vendored dependencies, lockfiles). Skip them at the ignore-rule layer; do not include them in the role map. If you can't tell whether a file is generated, look for a generation marker (header comment naming the generator, sibling .generated file, presence in .gitignore); if generated, omit from the role map.

When Phase 1 is complete, write your full exploration findings to quality/EXPLORATION.md. The file MUST contain ALL of the following section titles VERBATIM (the Phase 1 gate at SKILL.md:1257-1273 enforces each mechanically; bin/run_state_lib.validate_phase_artifacts(quality_dir, phase=1) is the programmatic enforcer — your artifact has to pass it before Phase 2 will start). The exact titles are load-bearing — do NOT substitute "equivalent" headings:

1. ## Open Exploration Findings — at least 8 numbered entries (1., 2., ...). Each entry has at least one file:line citation in the body (e.g., bin/foo.py:120-135). At least 3 of these entries trace behavior across 2 or more distinct file:line locations (multi-location traces — the entry cites two or more different file:line ranges).

2. ## Quality Risks — domain-knowledge risk analysis. Numbered or bulleted; cite file:line where risks are concretely visible in code or docs.

3. ## Pattern Applicability Matrix — a Markdown table with one row per exploration pattern from references/exploration_patterns.md. Decision column values are FULL or SKIP. Between 3 and 4 patterns must be marked FULL (inclusive — the gate rejects below 3 because exploration didn't pick enough patterns, and above 4 because exploration ran every pattern instead of selecting). Skipped patterns are still listed with SKIP and a brief reason, so the matrix is exhaustive.

4. ## Pattern Deep Dive — <pattern-name> — at least 3 sections, one per FULL pattern. Each deep dive enumerates concrete findings with file:line citations. At least 2 of these sections trace code paths across 2 or more distinct identifiers (e.g., backtick-quoted function or symbol names like \docs_present\`, \_evaluate_documentation_state\`) OR across 2 or more distinct file:line locations — that's how the gate detects "multi-function trace" rather than a one-anchor finding.

5. ## Candidate Bugs for Phase 2 — numbered list of bug hypotheses promoted from the deep dives + open exploration. Each entry has a Stage: line attributing the source (e.g., Stage: open exploration, Stage: quality risks, or Stage: <Pattern Name>). At least 2 entries must be sourced from open exploration / quality risks AND at least 1 entry must be sourced from a pattern deep dive. Combo stages (Stage: open exploration + Cross-Implementation Consistency) count toward both buckets.

6. ## Gate Self-Check — proves you ran the Phase 1 gate. List each of the 13 checks (≥120 lines + six required headings + ≥3 Pattern Deep Dive sections + PROGRESS.md mark + ≥8 findings with citations + ≥3 multi-location findings + 3-4 FULL pattern matrix rows + ≥2 multi-function deep dives + candidate-bug source mix) and mark whether the artifact satisfies each.

In addition, ensure quality/PROGRESS.md exists and its Phase 1 line is marked [x] (the gate's check 8) before declaring Phase 1 complete.

The exploration content the prior versions of this prompt asked for (domain and stack identification, architecture map, existing test inventory, specification summary, skeleton/dispatch analysis, derived requirements REQ-NNN, derived use cases UC-NN, file-role tagging summary) lives WITHIN these required sections — for example, the architecture map and module enumeration belong under ## Open Exploration Findings as multi-location findings; the file-role tagging summary and the exploration_role_map.json breakdown summary belong under ## Open Exploration Findings or ## Quality Risks as analytical content; derived REQ-NNN and UC-NN sections may appear after ## Gate Self-Check as additional analytical material the playbook downstream phases consume. Do NOT use these alternative names as TOP-level section titles — the gate requires the six exact titles above and the Pattern Deep Dive prefix; additional ## sections beyond these are tolerated for analytical extension but the six gate-required titles MUST appear verbatim.

MANDATORY CARTESIAN UC RULE (Lever 1, v1.5.2)

For every requirement with a References field naming ≥2 files (or ≥2 file:line ranges in distinct files), apply the Cartesian eligibility check before deciding whether to emit a single umbrella UC or per-site UCs:

Gate 1 — Path-suffix match. At least two references must share a path-suffix role: the last segment before the extension, or a matching function-name pattern that appears across the files.

• Example of a match: virtio_mmio.c, virtio_vdpa.c, virtio_pci_modern.c all implement _finalize_features. The _finalize_features function is the shared role.
• Example of a non-match: CONFIG_FOO, CONFIG_BAR flags in the same kconfig file — same kind of thing, but not parallel implementations.

Gate 2 — Function-level similarity. Each matching reference must cite a line range of similar size (within 2× of the median) and each range must be inside a function body — not a file-header, a kconfig block, or a macro expansion list.

Decision:

• Both gates pass → emit one UC per site, numbered UC-N.a, UC-N.b, UC-N.c, … Each per-site UC has its own Actors, Preconditions, Flow, Postconditions. The parent REQ-N remains as the umbrella.
• Only Gate 1 passes → keep a single umbrella UC and mark the reference cluster heterogeneous in a <!-- cluster: heterogeneous --> HTML comment in the UC body. Phase 3 can still override if it finds per-site divergence.
• Neither gate passes → single umbrella UC, no special marking.

Worked example — REQ-010 / VIRTIO_F_RING_RESET (virtio)

Suppose Phase 1 derives:

REQ-010: Virtio transports must honor VIRTIO_F_RING_RESET negotiation

• References: drivers/virtio/virtio_mmio.c, drivers/virtio/virtio_vdpa.c, drivers/virtio/virtio_pci_modern.c
• Pattern: whitelist

Applying the Cartesian check:

• Gate 1: all three files contain _finalize_features functions — matches.
• Gate 2: each cited range is inside a function body of similar size — matches.

Both gates pass → emit per-site UCs:

UC-10.a: VIRTIO_F_RING_RESET on PCI modern transport

• Actors: virtio_pci_modern driver, guest kernel
• Preconditions: device advertises VIRTIO_F_RING_RESET
• Flow: vp_modern_finalize_features propagates bit through config space …
• Postconditions: feature_bit reflected in final config

UC-10.b: VIRTIO_F_RING_RESET on MMIO transport

• Actors: virtio_mmio driver, guest kernel
• Preconditions: device advertises VIRTIO_F_RING_RESET
• Flow: vm_finalize_features must mirror PCI modern behavior …
• Postconditions: feature_bit survives finalize call

UC-10.c: VIRTIO_F_RING_RESET on vDPA transport

• Actors: virtio_vdpa driver, vdpa device backend
• Preconditions: device advertises VIRTIO_F_RING_RESET
• Flow: virtio_vdpa_finalize_features forwards through set_driver_features …
• Postconditions: feature_bit visible to vdpa backend

CONFIRMATION CHECKLIST (Cartesian UC rule)

Before completing Phase 1, confirm each item explicitly in EXPLORATION.md under a section titled "Cartesian UC rule confirmation":

1. For every REQ with ≥2 References, I ran Gate 1 (path-suffix match). 2. For every REQ that passed Gate 1, I ran Gate 2 (function-level similarity). 3. Where both gates passed, I emitted per-site UCs (UC-N.a, UC-N.b, …). 4. Where only Gate 1 passed, I marked the cluster <!-- cluster: heterogeneous -->. 5. Where neither gate passed, I kept a single umbrella UC without marking. 6. For each REQ with a pattern match in Gate 1, I added Pattern: whitelist|parity|compensation to the REQ block.

Also initialize quality/PROGRESS.md with the run metadata and the phase tracker in the EXACT checkbox format below. This format is a hard contract: the Phase 5 gate checks for the substring - [x] Phase 4 before allowing reconciliation to start, and it only matches the checkbox form. Do NOT substitute a Markdown table, bulleted prose, or any other layout — table-format runs have aborted mid-pipeline because the gate does not see "Complete" in a table cell as equivalent.

Template for the phase tracker section of PROGRESS.md (fill in the Skill version from SKILL.md metadata):

# Quality Playbook Progress

Skill version: <vX.Y.Z>
Date: <YYYY-MM-DD>

## Phase tracker

- [x] Phase 1 - Explore
- [ ] Phase 2 - Generate
- [ ] Phase 3 - Code Review
- [ ] Phase 4 - Spec Audit
- [ ] Phase 5 - Reconciliation
- [ ] Phase 6 - Verify

As each later phase completes it will flip its own - [ ] to - [x] — keep the line text (including the phase name after the dash) stable so substring matching in the Phase 5 gate and downstream tooling works.

IMPORTANT: Do NOT proceed to Phase 2. Your only job is exploration and writing findings to disk. Write thorough, detailed findings - the next phase will read EXPLORATION.md to generate artifacts, so everything important must be captured in that file.

{skill_fallback_guide}

You are a quality engineer continuing a phase-by-phase quality playbook run. Phase 1 (exploration) is already complete.

Read these files to get context: 1. quality/EXPLORATION.md - your Phase 1 findings (requirements, risks, architecture) 2. quality/PROGRESS.md - run metadata and phase status 3. SKILL.md - read the Phase 2 section (from "Phase 2: Generate the Quality Playbook" through the "Checkpoint: Update PROGRESS.md after artifact generation" section). Also read the reference files cited in that section. Resolve SKILL.md and reference files via the documented fallback list above; do NOT assume any single install layout (.github/skills/, .claude/skills/quality-playbook/, .cursor/skills/quality-playbook/, .continue/skills/quality-playbook/, or root).

Field preservation rule (v1.5.2, Lever 2). When transcribing REQ hypotheses from EXPLORATION.md into quality/REQUIREMENTS.md and quality/requirements_manifest.json, every - Pattern: <value> field present on the source hypothesis MUST appear on the corresponding REQ in both output files. Pattern values are whitelist | parity | compensation. Phase 1's Cartesian UC rule (confirmation checklist item 6) requires Pattern tagging for every REQ where both UC gates match; Phase 2 must not silently drop these tags. If a hypothesis lacks Pattern but you believe it should have one (per-site UCs emitted with UC-N.a/UC-N.b suffixes, multi-file References suggesting a parallel structure), add Pattern during Phase 2 — do not omit the field. The Phase 5 cardinality gate cannot enforce coverage on a REQ it doesn't know is pattern-tagged; silent omission is a documented v1.4.5-regression vector.

Execute Phase 2: Generate all quality artifacts. Use the exploration findings in EXPLORATION.md as your source - do not re-explore the codebase from scratch. Generate:

• quality/QUALITY.md (quality constitution)
• quality/CONTRACTS.md (behavioral contracts)
• quality/REQUIREMENTS.md (with REQ-NNN and UC-NN identifiers from EXPLORATION.md)
• quality/COVERAGE_MATRIX.md
• Functional tests (quality/test_functional.*)
• quality/RUN_CODE_REVIEW.md (code review protocol)
• quality/RUN_INTEGRATION_TESTS.md (integration test protocol)
• quality/RUN_SPEC_AUDIT.md (spec audit protocol)
• quality/RUN_TDD_TESTS.md (TDD verification protocol)
• quality/COMPLETENESS_REPORT.md (baseline, without verdict)
• If dispatch/enumeration contracts exist: quality/mechanical/ with verify.sh and extraction artifacts. Run verify.sh immediately and save receipts.

Update PROGRESS.md: mark Phase 2 complete (use the checkbox format - [x] Phase 2 - Generate — do NOT switch to a table), update artifact inventory.

IMPORTANT: Do NOT proceed to Phase 3 (code review). Your job is artifact generation only. The next phase will execute the review protocols you generated.

{skill_fallback_guide}

You are a quality engineer continuing a phase-by-phase quality playbook run. Phases 1-2 are complete.

Read these files to get context: 1. quality/PROGRESS.md - run metadata, phase status, artifact inventory 2. quality/EXPLORATION.md - Phase 1 findings (especially the "Candidate Bugs for Phase 2" section) 3. quality/REQUIREMENTS.md - derived requirements and use cases 4. quality/CONTRACTS.md - behavioral contracts 5. SKILL.md - read the Phase 3 section ("Phase 3: Code Review and Regression Tests"). Also read references/review_protocols.md. Resolve SKILL.md and the references/ directory via the documented fallback list above; do NOT assume any single install layout.

Execute Phase 3: Code Review + Regression Tests. Run the 3-pass code review per quality/RUN_CODE_REVIEW.md. For every confirmed bug:

• Add to quality/BUGS.md with ### BUG-NNN heading format
• Write a regression test (xfail-marked)
• Generate quality/patches/BUG-NNN-regression-test.patch (MANDATORY for every confirmed bug)
• Generate quality/patches/BUG-NNN-fix.patch (strongly encouraged)
• Write code review reports to quality/code_reviews/
• Update PROGRESS.md BUG tracker

MANDATORY GRID STEP (Lever 2, v1.5.2) — pattern-tagged REQs only

For every REQ in quality/REQUIREMENTS.md that has a Pattern: field (whitelist, parity, or compensation), you MUST produce a compensation grid BEFORE writing any BUG entries for that REQ.

Step 1. Enumerate the authoritative item set. Mechanical extraction from source — uapi header, spec section, documented constants. Do NOT invent. Example: for VIRTIO_F_RING_RESET-family, grep include/uapi/linux/virtio_config.h for VIRTIO_F_* and list the bits the REQ covers.

Step 2. Enumerate the sites. From the REQ's per-site UCs (UC-N.a, UC-N.b, …). If the REQ has a single umbrella UC but is pattern-tagged, the grid is 1-dimensional over items.

Step 3. Produce the grid. Write quality/compensation_grid.json with one entry per REQ:

{
  "schema_version": "1.5.2",
  "reqs": {
    "REQ-010": {
      "pattern": "whitelist",
      "items": ["RING_RESET", "ADMIN_VQ", "NOTIF_CONFIG_DATA", "SR_IOV"],
      "sites": ["PCI", "MMIO", "vDPA"],
      "cells": [
        {"cell_id": "REQ-010/cell-RING_RESET-PCI", "item": "RING_RESET", "site": "PCI", "present": true,  "evidence": "drivers/virtio/virtio_pci_modern.c:XXX-YYY"},
        {"cell_id": "REQ-010/cell-RING_RESET-MMIO", "item": "RING_RESET", "site": "MMIO", "present": false, "evidence": "drivers/virtio/virtio_mmio.c: no match for RING_RESET"}
      ]
    }
  }
}

Cell IDs are mechanical: REQ-<N>/cell-<item>-<site>. No whitespace, uppercase item/site identifiers where natural.

Step 4. Apply the BUG-default rule. For every cell where:

• the item is defined in authoritative source AND
• the item is absent from any shared filter AND
• the item is absent from the site's compensation path

→ the cell DEFAULTS to BUG. Emit one ### BUG-NNN entry with the cell's file:line citation, spec basis, and expected-vs-actual behavior. Include a - Covers: [REQ-N/cell-<item>-<site>] line (see schemas.md §8 for the field contract).

Step 5. Downgrade to QUESTION requires a structured JSON record. Append one record per downgraded cell to quality/compensation_grid_downgrades.json:

{
  "schema_version": "1.5.2",
  "downgrades": [
    {
      "cell_id": "REQ-010/cell-RING_RESET-MMIO",
      "authority_ref": "include/uapi/linux/virtio_config.h:116",
      "site_citation": "drivers/virtio/virtio_mmio.c:109-131",
      "reason_class": "intentionally-partial",
      "falsifiable_claim": "MMIO does not support RING_RESET because the MMIO transport predates the feature bit and kernel docs at Documentation/virtio/virtio_mmio.rst:42-55 state the transport is frozen at its v1.0 feature set; falsifiable by showing MMIO re-sets bit 40 under any kernel release."
    }
  ]
}

• reason_class enum: out-of-scope | deprecated | platform-gated | handled-upstream | intentionally-partial.
• authority_ref, site_citation, falsifiable_claim are required and non-empty.
• falsifiable_claim must state an observable condition that would make the claim wrong.
• Missing any required field, or reason_class outside the enum, or zero-length falsifiable_claim → cell REVERTS to BUG at Phase 5 gate time. There is no re-prompt loop.

Step 6. Self-check. Before finalizing BUGS.md for this REQ, verify that every cell in the grid appears in either:

• some BUG's - Covers: [...] list, OR
• a downgrade record in quality/compensation_grid_downgrades.json.

Any cell missing from both will fail the Phase 5 cardinality gate. This self-check is advisory in Phase 3; the blocking gate runs in Phase 5.

Worked example — RING_RESET grid (virtio)

REQ-010 pattern: whitelist. Items: {RING_RESET, ADMIN_VQ, NOTIF_CONFIG_DATA, SR_IOV}. Sites: {PCI, MMIO, vDPA}. Grid: 4 × 3 = 12 cells.

Code inspection reveals PCI implements all four; MMIO implements none of the four (frozen at v1.0 feature set); vDPA implements NOTIF_CONFIG_DATA but not the other three.

Grid (present=T, absent=F):

BUG-default applies to every F cell (8 total). Possible consolidation:

BUG-001: MMIO ignores VIRTIO_F_RING_RESET

• Primary requirement: REQ-010
• Covers: [REQ-010/cell-RING_RESET-MMIO]

BUG-002: vDPA ignores VIRTIO_F_RING_RESET

• Primary requirement: REQ-010
• Covers: [REQ-010/cell-RING_RESET-vDPA]

BUG-003: vDPA missing ADMIN_VQ hookup

• Primary requirement: REQ-010
• Covers: [REQ-010/cell-ADMIN_VQ-vDPA]

BUG-004: MMIO ignores NOTIF_CONFIG_DATA negotiation (common filter gap)

• Primary requirement: REQ-010
• Covers: [REQ-010/cell-NOTIF_CONFIG_DATA-MMIO]

BUG-005: MMIO + vDPA both miss SR_IOV propagation

• Primary requirement: REQ-010
• Covers: [REQ-010/cell-SR_IOV-MMIO, REQ-010/cell-SR_IOV-vDPA]
• Consolidation rationale: shared fix path in both transports goes through the same feature-bit filter; single patch on the shared helper closes both cells.

If the reviewer concluded MMIO ADMIN_VQ is intentionally out-of-scope because ADMIN_VQ is a PCI-only spec feature, the downgrade record would be:

{
  "cell_id": "REQ-010/cell-ADMIN_VQ-MMIO",
  "authority_ref": "include/uapi/linux/virtio_pci.h:NN",
  "site_citation": "drivers/virtio/virtio_mmio.c: no admin virtqueue implementation",
  "reason_class": "out-of-scope",
  "falsifiable_claim": "ADMIN_VQ is MMIO-scoped — falsifiable by citing any virtio-spec normative text requiring ADMIN_VQ on non-PCI transports."
}

Union check: 8 BUG-covered cells + 1 downgrade cell = 9. Grid has 12 cells; 4 present cells don't need coverage. Total: 8 F cells covered via BUGs + 1 via downgrade = all 9 absent cells accounted for. Grid → clean.

ITERATION mode addendum (MANDATORY INCREMENTAL WRITE, Phase 8)

When running in iteration mode (gap / unfiltered / parity / adversarial), write candidate BUG stubs to disk immediately on identification, not at end-of-review. Path: quality/code_reviews/<iteration>-candidates.md. One ### CANDIDATE-NNN heading per candidate, with at least a file:line citation. Reviewer upgrades candidates to confirmed BUGs in BUGS.md only after full triage.

CONFIRMATION CHECKLIST (Lever 2, v1.5.2)

Before writing the Phase 3 completion checkpoint to PROGRESS.md, confirm each item explicitly in your Phase 3 summary:

1. For every pattern-tagged REQ, I produced a compensation grid in quality/compensation_grid.json. 2. For every grid, I applied the BUG-default rule mechanically. 3. Every BUG emitted for a pattern-tagged REQ has a - Covers: [...] field with valid cell IDs. 4. Every BUG whose Covers list has ≥2 entries has a non-empty - Consolidation rationale: ... field. 5. For every downgraded cell, I wrote a complete structured record in quality/compensation_grid_downgrades.json with all five required fields and a valid reason_class. 6. For every pattern-tagged REQ, the union of Covers lists + downgrade cells equals the grid's cell set.

Mark Phase 3 (Code review + regression tests) complete in PROGRESS.md (use the checkbox format - [x] Phase 3 - Code Review — do NOT switch to a table).

IMPORTANT: Do NOT proceed to Phase 4 (spec audit). The next phase will run the spec audit with a fresh context window.

{skill_fallback_guide}

You are a quality engineer continuing a phase-by-phase quality playbook run. Phases 1-3 are complete.

Read these files to get context: 1. quality/PROGRESS.md - run metadata, phase status, BUG tracker 2. quality/REQUIREMENTS.md - derived requirements 3. quality/BUGS.md - bugs found in Phase 3 (code review) 4. SKILL.md - read the Phase 4 section ("Phase 4: Spec Audit and Triage"). Also read references/spec_audit.md. Resolve SKILL.md and the references/ directory via the documented fallback list above; do NOT assume any single install layout.

Execute Phase 4: Spec Audit + Triage + Layer-2 semantic citation check.

Part A — spec audit: Run the spec audit per quality/RUN_SPEC_AUDIT.md. Produce:

• Individual auditor reports at quality/spec_audits/YYYY-MM-DD-auditor-N.md (one per auditor)
• Triage synthesis at quality/spec_audits/YYYY-MM-DD-triage.md
• Executable triage probes at quality/spec_audits/triage_probes.sh
• Regression tests and patches for any net-new spec audit bugs
• Update BUGS.md and PROGRESS.md BUG tracker with any new findings

Part B — Layer-2 semantic citation check (v1.5.1): The gate's invariant #17 (schemas.md §10) requires three Council members to vote on each Tier 1/2 REQ's citation_excerpt. Execute these steps:

1. Generate per-Council-member prompts: python3 -m bin.quality_playbook semantic-check plan . This writes one or more prompt files to quality/council_semantic_check_prompts/<member>.txt per member in the Council roster (bin/council_config.py: claude-opus-4.7, gpt-5.4, gemini-2.5-pro). For >15 Tier 1/2 REQs, prompts are split into batches of 5 (<member>-batch<N>.txt). If no Tier 1/2 REQs exist (Spec Gap run), this step writes an empty quality/citation_semantic_check.json directly — skip steps 2-4.

2. For each Council member's prompt file, feed the prompt to that model (the same roster that ran Part A) and capture its JSON-array response to quality/council_semantic_check_responses/<member>.json. If the member was batched, concatenate the per-batch responses into a single array in the response file. Every entry must have req_id, verdict (supports|overreaches|unclear), and reasoning.

3. Assemble the semantic-check output: python3 -m bin.quality_playbook semantic-check assemble . \ --member claude-opus-4.7 --response quality/council_semantic_check_responses/claude-opus-4.7.json \ --member gpt-5.4 --response quality/council_semantic_check_responses/gpt-5.4.json \ --member gemini-2.5-pro --response quality/council_semantic_check_responses/gemini-2.5-pro.json This writes quality/citation_semantic_check.json per schemas.md §9.

4. Verify the output file exists. Phase 6's gate invariant #17 requires it on every Tier 1/2 run.

Mark Phase 4 (Spec audit + triage + semantic check) complete in PROGRESS.md (use the checkbox format - [x] Phase 4 - Spec Audit — the Phase 5 entry gate looks for that exact substring and will abort if it finds a table row or any other layout).

IMPORTANT: Do NOT proceed to Phase 5 (reconciliation). The next phase will handle reconciliation and TDD.

{skill_fallback_guide}

You are a quality engineer continuing a phase-by-phase quality playbook run. Phases 1-4 are complete.

Read these files to get context: 1. quality/PROGRESS.md - run metadata, phase status, cumulative BUG tracker 2. quality/BUGS.md - all confirmed bugs from code review and spec audit 3. quality/REQUIREMENTS.md - derived requirements 4. SKILL.md - read the Phase 5 section ("Phase 5: Post-Review Reconciliation and Closure Verification"). Also read references/requirements_pipeline.md, references/review_protocols.md, and references/spec_audit.md. Resolve SKILL.md and the references/ directory via the documented fallback list above; do NOT assume any single install layout.

Execute Phase 5: Reconciliation + TDD + Closure.

1. Run the Post-Review Reconciliation per references/requirements_pipeline.md. Update COMPLETENESS_REPORT.md. 2. Run closure verification: every BUG in the tracker must have either a regression test or an explicit exemption. 3. Write bug writeups at quality/writeups/BUG-NNN.md for EVERY confirmed bug. The canonical template is the "Bug writeup generation" section of SKILL.md (resolve via the fallback list above) — read that section before writing. Use the exact field headings listed there: Summary, Spec reference, The code, Observable consequence, Depth judgment, The fix, The test, Related issues. Sections 1–4, 6, 7 are required in every writeup; section 5 (Depth judgment) fires only when the consequence isn't self-evident from the immediate code; section 8 (Related issues) is included only when related bugs exist. Do NOT introduce fields that aren't in the template (no "Minimal reproduction" as a top-level field, no "Patch path:" as a top-level field — those belong inside Spec reference and The test respectively).

MANDATORY HYDRATION STEP. Before writing a writeup, re-open quality/BUGS.md and locate the ### BUG-NNN: entry for the bug you are about to write up. Every confirmed bug in BUGS.md already has the content you need — your job is to copy it into the writeup's sections, not to invent it. If a field is missing from BUGS.md, that is a reconciliation error to surface in PROGRESS.md, not a field to fabricate. Use this field map:

Worked example. The BUGS.md entry for BUG-004 is:

BUG-004: naive upstream timestamps crash ETA math

• Source: Code Review
• Severity: HIGH
• Primary requirement: REQ-006
• Location: bus_tracker.py:138-144
• Spec basis: quality/REQUIREMENTS.md:163-172; quality/QUALITY.md:57-65
• Minimal reproduction: Return a visit whose ExpectedArrivalTime is an ISO string

without timezone information, such as 2026-04-21T12:00:00.

• Expected behavior: The affected arrival degrades to unknown-time while the rest

of the stop remains usable.

• Actual behavior: datetime.fromisoformat() returns a naive datetime and

subtracting it from datetime.now(timezone.utc) raises TypeError, aborting the stop/request path.

• Regression test: quality.test_regression.TestPhase3Regressions.test_bug_004_fetch_stop_arrivals_degrades_naive_timestamps
• Patches: quality/patches/BUG-004-regression-test.patch, quality/patches/BUG-004-fix.patch

The hydrated writeup sections look like this (sketch — paste the real diff from the fix patch file into ```diff, don't make one up):

Summary

fetch_stop_arrivals() crashes the whole stop/request path when an upstream visit carries a naive ExpectedArrivalTime, instead of degrading that arrival to unknown-time.

Spec reference

• Requirement: REQ-006
• Spec basis: quality/REQUIREMENTS.md:163-172; quality/QUALITY.md:57-65
• Behavioral contract quote: "degrade a bad per-arrival timestamp to unknown-time instead of aborting the whole response path"

The code

At bus_tracker.py:138-144, the parser calls datetime.fromisoformat(...) on ExpectedArrivalTime and subtracts the result from datetime.now(timezone.utc)…

Observable consequence

When the upstream visit returns ExpectedArrivalTime="2026-04-21T12:00:00" (no timezone), fromisoformat() returns a naive datetime, the subtraction raises TypeError, and the entire stop/request path aborts rather than the single affected arrival degrading to unknown-time.

The fix

       <paste the real unified diff from quality/patches/BUG-004-fix.patch here>

The test

• Regression test: quality.test_regression.TestPhase3Regressions.test_bug_004_fetch_stop_arrivals_degrades_naive_timestamps
• Regression patch: quality/patches/BUG-004-regression-test.patch
• Fix patch: quality/patches/BUG-004-fix.patch
• Red receipt: quality/results/BUG-004.red.log
• Green receipt: quality/results/BUG-004.green.log

Confirmation checklist (per writeup, before moving to the next bug). (a) Every required section has populated content copied from BUGS.md or the patch files — no empty backticks, no sentinel filler like "is a confirmed code bug in `" or "The affected implementation lives at " or "Patch path: ". (b) The `diff fence contains at least one + or - line from the actual fix patch. (c) The Summary names a real function or code path, not the BUG identifier. (d) No angle-bracket placeholders (e.g., <...>`) remain in the final writeup — those are pedagogical markers from the worked example and from SKILL.md, never acceptable output. 4. Run the TDD red-green cycle: for each confirmed bug, run the regression test against unpatched code -> quality/results/BUG-NNN.red.log. If a fix patch exists, run against patched code -> quality/results/BUG-NNN.green.log. If the test runner is unavailable, create the log with NOT_RUN on the first line. 5. Generate sidecar JSON: quality/results/tdd-results.json and quality/results/integration-results.json (schema_version "1.1", canonical fields: id, requirement, red_phase, green_phase, verdict, fix_patch_present, writeup_path). 6. If mechanical verification artifacts exist, run quality/mechanical/verify.sh and save receipts. 7. Run terminal gate verification, write it to PROGRESS.md.

MANDATORY CARDINALITY GATE (Lever 3, v1.5.2)

Before finalizing this phase, run the cardinality reconciliation gate against the current repo state. Locate quality_gate.py via the same fallback list used for SKILL.md (it sits in the same directory as SKILL.md in every install layout), then invoke it as a script — quality_gate.py runs check_v1_5_2_cardinality_gate(repo_dir) as part of its standard pass:

python3 <resolved_quality_gate_path> .

Where <resolved_quality_gate_path> is the first hit when walking the documented install-location fallback list, with SKILL.md swapped for quality_gate.py (e.g., quality_gate.py, .claude/skills/quality-playbook/quality_gate.py, .github/skills/quality_gate.py, .cursor/skills/quality-playbook/quality_gate.py, .continue/skills/quality-playbook/quality_gate.py, .github/skills/quality-playbook/quality_gate.py).

If the gate output contains any line beginning with cardinality gate:, or reports uncovered cells, malformed cell IDs, missing consolidation rationale on multi-cell Covers, or malformed downgrade records, STOP. Fix the BUGS.md entries or the compensation_grid_downgrades.json file. Do NOT proceed to completion until those failure lines no longer appear.

For every pattern-tagged REQ, the Phase 5 contract is:

• Every grid cell with "present": false appears in either a BUG's Covers: list or a downgrade record.
• Every Covers: entry uses the canonical cell ID form REQ-N/cell-<item>-<site>.
• Every BUG with ≥2 Covers: entries has a non-empty Consolidation rationale: line.
• Every downgrade record has cell_id, authority_ref, site_citation, reason_class (in the enum), falsifiable_claim (non-empty).

The cardinality gate is blocking. It is intentionally stricter than the Phase 3 advisory self-check; the advisory check is meant to surface problems early, but Phase 5 is where they become fatal.

Mark Phase 5 complete in PROGRESS.md (use the checkbox format - [x] Phase 5 - Reconciliation — do NOT switch to a table).

IMPORTANT: quality_gate.py will FAIL Phase 5 if any writeup is missing a non-empty ``diff block or contains any of these sentinel phrases verbatim: "is a confirmed code bug in ", "The affected implementation lives at ", "Patch path: ", "- Regression test: ", "- Regression patch: `". Those two checks are the hard gate. Skipping the BUGS.md hydration step above is not gate-enforced but will produce writeups that read as unpopulated stubs and fail a human review — do not skip it.

{skill_fallback_guide}

You are a quality engineer doing the verification phase of a quality playbook run. Phases 1-5 are complete.

Read SKILL.md - the Phase 6 section ("Phase 6: Verify"). Resolve SKILL.md via the documented fallback list above; do NOT assume any single install layout. Follow the incremental verification steps (6.1 through 6.5).

Step 6.1: If quality/mechanical/verify.sh exists, run it. Record exit code. Step 6.2: Run quality_gate.py. Locate it via the same fallback list used for SKILL.md (quality_gate.py sits in the same directory as SKILL.md in every install layout — e.g., quality_gate.py, .claude/skills/quality-playbook/quality_gate.py, .github/skills/quality_gate.py, .cursor/skills/quality-playbook/quality_gate.py, .continue/skills/quality-playbook/quality_gate.py, .github/skills/quality-playbook/quality_gate.py). Then run: python3 <resolved_quality_gate_path> . Read the output carefully. For every FAIL result, fix the issue:

• Missing regression-test patches: generate quality/patches/BUG-NNN-regression-test.patch
• Missing inline diffs in writeups: add a ```diff block
• Non-canonical JSON fields: fix tdd-results.json (use 'id' not 'bug_id', etc.)
• Missing files: create them

After fixing all FAILs, run quality_gate.py again. Repeat until 0 FAIL. Save final output to quality/results/quality-gate.log.

Step 6.3: Run functional tests if a test runner is available. Step 6.4: File-by-file verification checklist (read one file at a time, check, move on). Step 6.5: Metadata consistency check.

Append each step's result to quality/results/phase6-verification.log. Mark Phase 6 complete in PROGRESS.md (use the checkbox format - [x] Phase 6 - Verify — do NOT switch to a table).

phase_prompts/

Externalized phase prompt bodies for the Quality Playbook.

v1.5.4 F-1 (Bootstrap_Findings 2026-04-30) extracted these from bin/run_playbook.py's inline string templates so both execution modes — UI-context skill-direct (a coding agent walking through SKILL.md inline) and CLI-automation runner-driven (python -m bin.run_playbook) — read from the same single source of truth. Without externalization the two modes drift; with it, an edit to a phase prompt lands once and benefits both.

File layout

• phase1.md ... phase6.md — one file per pipeline phase. Loaded

by bin/run_playbook.py::_load_phase_prompt.

• single_pass.md — the legacy single-prompt invocation (used when

the operator wants the LLM to drive all six phases inline rather than via the per-phase orchestrator).

• iteration.md — the iteration-strategy prompt (gap, unfiltered,

parity, adversarial — see bin/run_playbook.py::next_strategy).

Substitution conventions

Most files are pure-literal markdown — the loader returns them unchanged. Three files use str.format() substitution with named placeholders:

• phase1.md — {seed_instruction} (skip Phase 0/0b prelude when

--no-seeds) and {role_taxonomy} (rendered from bin.role_map.ROLE_DESCRIPTIONS).

• single_pass.md — {skill_fallback_guide} and

{seed_instruction}.

• iteration.md — {skill_fallback_guide} and {strategy}.

Inside files that go through .format(), JSON braces and other literal { / } characters MUST be doubled ({{ / }}) per Python's format-string escaping rules. Pure-literal files do not need any escaping.

Editing discipline

When you change a phase prompt, the loader picks up the new content at the next invocation — there is no caching layer to invalidate. The test suite at bin/tests/test_phase_prompts_externalized.py pins the loader's contract; if you add a new substitution variable, extend those tests.

{skill_fallback_guide} Execute the quality playbook for this project.{seed_instruction}

Challenge Gate — Bug Validity Review

Purpose

The challenge gate is a self-adversarial review that every confirmed bug must survive before receiving a writeup and regression test. It catches false positives, over-classified feature gaps, and findings where pattern-matching overrode common sense.

The gate can be invoked two ways:

1. During a playbook run — automatically applied to bugs matching trigger patterns (see below). 2. Standalone — pointed at a quality/ directory from a prior run to challenge specific bugs. Example: "Read quality/writeups/BUG-042.md and the source code it references. Run the challenge gate on this bug."

The two-round challenge

For each bug under review, run exactly two rounds. Each round uses a fresh sub-agent so the challenger has no investment in the finding.

Round 1: "Does this strike you as a real bug?"

Provide the sub-agent with:

• The bug writeup (or BUGS.md entry if no writeup yet)
• The actual source code at the cited file:line (read it fresh — do not trust the writeup's code snippet)
• All comments within 10 lines above and below the cited location
• The project's README section on the relevant feature (if any)

Prompt the sub-agent:

You are reviewing a bug report filed against an open-source project. Read the source code and the bug report below. Then answer: does this strike you as a real bug?

>

Before analyzing anything, apply common sense. Step back from the details and ask yourself: if you showed this code and this bug report to a senior developer who has never seen either before, would they say "yes, that's a bug" — or would they say "that's obviously not a bug"? If the answer is obviously not a bug, say so immediately and explain why. Do not rationalize your way past a common-sense answer. The goal of this review is to catch findings where pattern-matching overrode judgment.

>

Then consider:

- Is the developer aware of this behavior? (Look for comments, TODO markers, design decision notes, WHY annotations, OODA references.)

- Is this a documented limitation or intentional trade-off? (Check if other code paths handle this differently by design, not by accident.)

- Would the project maintainer respond "that's not a bug, that's how it works" or "that's a known limitation we documented"?

- Is the "expected behavior" in the bug report actually required by any spec, or is it the auditor's opinion about what the code should do?

- Is this development scaffolding? Values with names like "change-me", "placeholder", "example", "default", "TODO" are not defects — they are self-documenting markers that exist to make the project buildable during development. A feature that is disabled by default and uses placeholder values is an incomplete feature, not a vulnerability.

>

Give your honest assessment. If it's a real bug, say so and explain why. If it's not, say so and explain why. A finding can be "not a bug" even if the code could be improved — the question is whether a reasonable maintainer would accept this as a defect report.

Round 2: Targeted follow-up

Based on the Round 1 response, generate a single pointed follow-up question. The goal is to stress-test whatever position the sub-agent took in Round 1.

If Round 1 said "real bug": The follow-up should challenge the finding from the maintainer's perspective. Use a fresh sub-agent with this framing:

You are the maintainer of this project. A contributor filed this bug report. You wrote the code being criticized. Read the code, the bug report, and the Round 1 assessment below.

>

Write the single most compelling argument for why this is NOT a bug. Consider: intentional design decisions, documented limitations, deployment context, common patterns in this language/framework, and whether the "expected behavior" is actually specified anywhere authoritative.

>

Then, after making that argument, state whether you still believe it's a real bug or whether the argument convinced you it's not.

If Round 1 said "not a bug": The follow-up should challenge the dismissal. Use a fresh sub-agent with this framing:

You are a security researcher reviewing this codebase. Another reviewer dismissed this finding as "not a bug." Read the code, the bug report, and the Round 1 dismissal below.

>

Write the single most compelling argument for why this IS a real bug despite the dismissal. Consider: edge cases the dismissal didn't address, downstream consequences, what happens when the code interacts with other components, and whether "intentional" and "correct" are the same thing.

>

Then, after making that argument, state whether you believe the finding should be confirmed or dismissed.

Verdict

After both rounds, assign one of three verdicts:

• CONFIRMED — Both rounds agree it's a real bug, or Round 2's challenge failed to undermine Round 1's confirmation. The bug proceeds to writeup and regression test as normal.
• DOWNGRADED — The bug is real but the severity was inflated. Adjust severity and update the writeup. Common case: a CRITICAL that should be MEDIUM, or a security finding that's actually a code quality issue.
• REJECTED — The finding is not a bug. It's a documented design choice, intentional scaffolding, a known limitation, the auditor's opinion rather than a spec violation, or something that common sense says is obviously not a defect. Remove the bug ID. If useful, record it in a "Reviewed and dismissed" appendix in BUGS.md with the challenge reasoning.

Write the verdict and both rounds' reasoning to quality/challenge/BUG-NNN-challenge.md. This file is the audit trail — it shows reviewers that each finding was stress-tested.

Auto-trigger patterns

During a playbook run, automatically apply the challenge gate to any bug matching one or more of these patterns. These patterns are where false positives concentrate:

The pattern list is intentionally conservative — it triggers on categories with historically high false-positive rates. Bugs that don't match any pattern skip the challenge gate and proceed directly to writeup.

To add new patterns: append a row to the table above with the pattern description, the reasoning, and a concrete example from a prior run.

Standalone invocation

When invoked standalone (not during a playbook run), the challenge gate:

1. Reads the specified bug writeup from quality/writeups/BUG-NNN.md 2. Reads the source code at the cited file:line (fresh read, not from the writeup) 3. Runs both rounds as described above 4. Writes the verdict to quality/challenge/BUG-NNN-challenge.md 5. If the verdict is REJECTED, suggests removing the bug from BUGS.md and tdd-results.json

Example prompt for standalone use:

Read the quality playbook skill at .github/skills/SKILL.md and .github/skills/references/challenge_gate.md.
Run the challenge gate on BUG-042 using the writeup at quality/writeups/BUG-042.md
and the source code in this repo.

Token budget

Each bug costs roughly 2 sub-agent calls. For a typical run with 5-10 auto-triggered bugs, that's 10-20 sub-agent calls. This is significantly cheaper than a full iteration cycle and catches the highest-value false positives.

For runs with many security findings (>15 auto-triggered), consider batching: run Round 1 on all triggered bugs first, then only run Round 2 on bugs where Round 1 was ambiguous or where the confidence was low.

Code-only mode

Last updated: 2026-05-03 (v1.5.6 Phase 3 — initial publication).

When the Quality Playbook runs against a target repo whose reference_docs/ directory is absent or empty, it operates in code-only mode. This document explains what that means, why it matters, and how to upgrade a code-only run into a full-documentation run for the next pass.

What "code-only mode" means

The playbook's normal Phase 1 derivation reads two kinds of evidence:

• Code evidence (Tier 3+) — the source tree itself, plus inline comments, defensive patterns, tests, and any inline documentation co-located with the code.
• Documentation evidence (Tier 1/2) — plaintext files the operator drops into reference_docs/ (free-form notes, design docs, retrospectives, AI chats) and reference_docs/cite/ (project specs, RFCs, API contracts that requirements should be traceable back to).

Code-only mode is the run state where no documentation evidence is available. The playbook proceeds — it does not abort — but every requirement it derives leans entirely on code evidence. The Phase 1 EXPLORATION.md gets a "Documentation status: code-only mode" opening section that surfaces the mode so reviewers see it on first read.

What to expect from a code-only run

In our benchmark runs, code-only passes consistently produce:

• Fewer requirements derived overall. Without spec-language to anchor, Phase 1 has no Tier 1/2 evidence to cite, so the requirements set falls back to Tier 3 (code-as-spec) entirely.
• Possibly fewer bugs found. Code review (Phase 3) is most effective when the reviewer knows what the code is supposed to do — bugs that violate documented intent are easier to surface than bugs that hide behind ambiguous code-as-spec. With no documentation, the reviewer has to infer intent from the code itself, which leaves a class of intent-violation defects undetected.
• Higher reliance on code-internal signals. Defensive patterns (error checks, validation), test names, and comment-style annotations carry more weight in the absence of external docs.

The bug counts in code-only mode are still useful — they reflect what's discoverable from the code alone — but they are a lower bound on what a fully-documented run would produce.

How to upgrade to a full-documentation run

Place plaintext documentation files in the target repo's reference_docs/ tree before re-running Phase 1:

<target-repo>/
  reference_docs/
    project_notes.md         # Tier 4 — informal notes, AI chats
    design_overview.md       # Tier 3-4 — internal design decisions
    cite/
      api_spec.md            # Tier 1/2 — citable specs, RFCs, contracts
      protocol_v3.txt        # Tier 1/2 — formal specifications

Files at the top level of reference_docs/ count as informal context (Tier 4). Files under reference_docs/cite/ count as citable evidence (Tier 1 or 2 depending on the source's authority — see schemas.md §3.1). Both .md and .txt are recognized; other formats are ignored.

After dropping in documentation, re-run the playbook. Phase 1 will detect the populated reference_docs/ and skip the code-only-mode downgrade. The new run's EXPLORATION.md, REQUIREMENTS.md, and BUGS.md will reflect the richer evidence base.

Opt-out: --require-docs

Operators who want runs to abort instead of proceeding in code-only mode can pass --require-docs to python3 -m bin.run_playbook (v1.5.6+). When --require-docs is set and reference_docs/ is empty at Phase 1 entry, the playbook:

1. Appends an aborted_missing_docs event to quality/run_state.jsonl (event type registered in references/run_state_schema.md). 2. Writes a clear ERROR: aborted_missing_docs — reference_docs/ empty and --require-docs set block to quality/PROGRESS.md. 3. Aborts before any LLM work (exit non-zero, same as a gate-fail).

The flag is off by default. Use it for compliance/policy contexts where a quiet code-only-mode downgrade would mask a real process gap (e.g., "every release run must cite a spec; no spec means the run shouldn't have started"). The flag is the opt-IN counterpart to --no-formal-docs's opt-OUT (which suppresses the WARN banner for the same code-only-mode case but allows the run to continue).

Cross-references

• README — Step 1 of "How to use the Quality Playbook" describes documentation as the first thing to provide.
• `SKILL.md` — Phase 1 prose describes how documentation evidence is used during exploration.
• `bin/reference_docs_ingest.py` — the implementation that ingests the reference_docs/ tree.
• `references/run_state_schema.md` — defines the documentation_state event the playbook emits when code-only mode triggers, so the downgrade is searchable in audit trails.

Writing the Quality Constitution (File 1: QUALITY.md)

The quality constitution defines what "quality" means for this specific project and makes the bar explicit, persistent, and inherited by every AI session.

Template

# Quality Constitution: [Project Name]

## Purpose

[2–3 paragraphs grounding quality in three principles:]

- **Deming** ("quality is built in, not inspected in") — Quality is built into context files
  and the quality playbook so every AI session inherits the same bar.
- **Juran** ("fitness for use") — Define fitness specifically for this project. Not "tests pass"
  but the actual real-world requirement. Example: "generates correct output that survives
  input schema changes without silently producing wrong results."
- **Crosby** ("quality is free") — Building a quality playbook upfront costs less than
  debugging problems found after deployment.

## Coverage Targets

| Subsystem | Target | Why |
|-----------|--------|-----|
| [Most fragile module] | 90–95% | [Real edge case or past bug] |
| [Core logic module] | 85–90% | [Concrete risk] |
| [I/O or integration layer] | 80% | [Explain] |
| [Configuration/utilities] | 75–80% | [Explain] |

The rationale column is essential. It must reference specific risks or past failures.
If you can't explain why a subsystem needs high coverage with a concrete example,
the target is arbitrary.

## Coverage Theater Prevention

[Define what constitutes a fake test for this project.]

Generic examples that apply to most projects:
- Asserting a function returned *something* without checking what
- Testing with synthetic data that lacks the quirks of real data
- Asserting an import succeeded
- Asserting mock returns what the mock was configured to return
- Calling a function and only asserting no exception was thrown

[Add project-specific examples based on what you learned during exploration.
For a data pipeline: "counting output records without checking their values."
For a web app: "checking HTTP 200 without checking the response body."
For a compiler: "checking output compiles without checking behavior."]

## Fitness-to-Purpose Scenarios

[5–10 scenarios. Every scenario must include a `[Req: tier — source]` tag linking it to its requirement source. Use the template below:]

### Scenario N: [Memorable Name]

**Requirement tag:** [Req: formal — Spec §X] *(or `user-confirmed` / `inferred` — see SKILL.md Phase 1, Step 1 for tier definitions)*

**What happened:** [The architectural vulnerability, edge case, or design decision.
Reference actual code — function names, file names, line numbers. Frame as "this architecture permits the following failure mode."]

**The requirement:** [What the code must do to prevent this failure.
Be specific enough that an AI can verify it.]

**How to verify:** [Concrete test or query that would fail if this regressed.
Include exact commands, test names, or assertions.]

---

[Repeat for each scenario]

## AI Session Quality Discipline

1. Read QUALITY.md before starting work.
2. Run the full test suite before marking any task complete.
3. Add tests for new functionality (not just happy path — include edge cases).
4. Update this file if new failure modes are discovered.
5. Output a Quality Compliance Checklist before ending a session.
6. Never remove a fitness-to-purpose scenario. Only add new ones.

## The Human Gate

[List things that require human judgment:]
- Output that "looks right" (requires domain knowledge)
- UX and responsiveness
- Documentation accuracy
- Security review of auth changes
- Backward compatibility decisions

Where Scenarios Come From

Scenarios come from two sources — code exploration and domain knowledge — and the best scenarios combine both.

Source 1: Defensive Code Patterns (Code Exploration)

Every defensive pattern is evidence of a past failure or known risk:

1. Defensive code — Every if value is None: return guard is a scenario. Why was it needed? 2. Normalization functions — Every function that cleans input exists because raw input caused problems 3. Configuration that could be hardcoded — If a value is read from config instead of hardcoded, someone learned the value varies 4. Git blame / commit messages — "Fix crash when X is missing" → Scenario: X can be missing 5. Comments explaining "why" — "We use hash(id) not sequential index because..." → Scenario about correctness under that constraint

Source 2: What Could Go Wrong (Domain Knowledge)

Don't limit yourself to what the code already defends against. Use your knowledge of similar systems to generate realistic failure scenarios that the code should handle. For every major subsystem, ask:

• "What happens if this process is killed mid-operation?" (state machines, file I/O, batch processing)
• "What happens if external input is subtly wrong?" (validation pipelines, API integrations)
• "What happens if this runs at 10x scale?" (batch processing, databases, queues)
• "What happens if two operations overlap?" (concurrency, file locks, shared state)
• "What produces correct-looking output that is actually wrong?" (randomness, statistical operations, type coercion)

These are not hypothetical — they are things that happen to every system of this type. Write them as architectural vulnerability analyses: "Because save_state() lacks an atomic rename pattern, a mid-write crash during a 10,000-record batch will leave a corrupted state file — the next run gets JSONDecodeError and cannot resume without manual intervention. At scale (9,240 records across 64 batches), this pattern risks silent loss of 1,693+ records with nothing to flag them as missing." Concrete numbers and specific consequences make scenarios authoritative and non-negotiable. An AI session reading "records can be lost" will argue the standard down. An AI session reading a specific failure mode with quantified impact will not.

The Narrative Voice

Each scenario's "What happened" must read like an architectural vulnerability analysis, not an abstract specification. Include:

• Specific quantities — "308 records across 64 batches" not "some records"
• Cascade consequences — "cascading through all subsequent pipeline steps, requiring reprocessing of 4,300 records instead of 308"
• Detection difficulty — "nothing would flag them as missing" or "only statistical verification would catch it"
• Root cause in code — "random.seed(index) creates correlated sequences because sequential integers produce related random streams"

The narrative voice serves a critical purpose: it makes standards non-negotiable. Abstract requirements ("records should not be lost") invite rationalization. Specific failure modes with quantified impact ("a mid-batch crash silently loses 1,693 records with no detection mechanism") do not. Frame these as "this architecture permits the following failure" — grounded in the actual code, not fabricated as past incidents.

Combining Both Sources

The strongest scenarios combine a defensive pattern found in code with domain knowledge about why it matters:

1. Find the defensive code: save_state() writes to a temp file then renames 2. Ask what failure this prevents: mid-write crash leaves corrupted state file 3. Write the scenario as a vulnerability analysis: "Without the atomic rename pattern, a crash mid-write leaves state.json 50% complete. The next run gets JSONDecodeError and cannot resume without manual intervention." 4. Ground it in code: "Read persistence.py line ~340: verify temp file + rename pattern"

The "Why" Requirement

Every coverage target, every quality gate, every standard must have a "why" that references a specific scenario or risk. Without rationale, a future AI session will optimize for speed and argue the standard down.

Bad: "Core logic: 100% coverage" Good: "Core logic: 100% — because random.seed(index) created correlated sequences that produced 77.5% bias instead of 50/50. Subtle bugs here produce plausible-but-wrong output. Only statistical verification catches them."

The "why" is not documentation — it is protection against erosion.

Calibrating Scenario Count

Aim for 2+ scenarios per core module (the modules identified as most complex or fragile). For a medium-sized project, this typically yields 8–10 scenarios. Fewer is fine for small projects; more for complex ones. If you're finding very few scenarios, it usually means the exploration was shallow rather than the project being simple — go back and read function bodies more carefully. Quality matters more than count: one scenario that precisely captures an architectural vulnerability is worth more than three generic "what if the input is bad" scenarios.

Self-Critique Before Finishing

After drafting all scenarios, review each one and ask:

1. "Would an AI session argue this standard down?" If yes, the "why" isn't concrete enough. Add numbers, consequences, and detection difficulty. 2. "Does the 'What happened' read like a vulnerability analysis or an abstract spec?" If it reads like a spec, rewrite it with specific quantities, cascading consequences, and grounding in actual code. 3. "Is there a scenario I'm not seeing?" Think about what a different AI model would flag. Architecture models catch data flow problems. Edge-case models catch boundary conditions. What are you blind to?

Critical Rule

Each scenario's "How to verify" section must map to at least one automated test in the functional test file. If a scenario can't be automated, note why (it may require the Human Gate) — but most scenarios should be testable.