Agent Output Audit

Agent Output Audit

Independent verification of AI-implemented work. The skill that asks: "Did the implementing agent actually do what `task_NN.md` says it did?" — not "Would a real user succeed at this product?" (that's qa-execution).

Required Reading Router

Match your task to the row. Read the listed files in full before producing output. They are not appendices — they are load-bearing. Inline content in this SKILL.md is a pointer, not a substitute.

Reference Index

• references/project-signals.md — Heuristics for picking install/lint/test/build/start commands across ecosystems when the repo lacks an umbrella gate.
• references/e2e-coverage.md — Taxonomy for existing-e2e / needs-e2e / manual-only / blocked and how to detect harness support.
• references/checklist.md — Audit checklist by category: contract discovery, baseline, task audit, AI hygiene, flaky detection, quality gates.
• references/ai-implementation-audit.md — Red Flag scanners (RF-1..RF-6), Requirement→Test mapping, verdict matrix for completed tasks.
• references/independent-evaluator-protocol.md — What counts (and doesn't count) as evidence; transcript classification (genuine-failure / grader-bug / ambiguous-task / bypass-exploit).
• references/flaky-triage.md — Taxonomy, diagnosis protocol, and quarantine workflow for retry-passes-without-code-change failures.

Required Inputs

• audit-output-path (optional): Directory where audit artifacts (bugs, audit report, evidence) are stored. When provided, create the directory if it does not exist and use it for all audit outputs. When omitted, fall back to repository conventions or /tmp/agent-output-audit-<slug>.

Procedures

Step 1: Discover the Repository Verification Contract

1. Read root instructions, repository docs, and CI/build files before running commands. 2. Execute python3 scripts/discover-project-contract.py --root . to surface candidate install, verify, build, test, lint, start commands, and E2E signals. 3. STOP. Read `references/project-signals.md` in full before picking commands when discovery surfaces more than one plausible gate or the repo mixes ecosystems. 4. STOP. Read `references/e2e-coverage.md` in full before classifying any flow. 5. Prefer repository-defined umbrella commands such as make verify, just verify, or CI entrypoints over language-default commands. 6. Resolve the audit artifact directory. If the user provided an audit-output-path argument, use it. Otherwise use repository conventions, falling back to /tmp/agent-output-audit-<slug>. Create the audit/ subdirectory; store all bugs and reports under <audit-output-path>/audit/. 7. Detect Compozy mode. If .compozy/tasks/<slug>/ exists, record the slug and switch into Compozy-aware audit:

• Read state.yaml (read-only — never write to it; `scripts/update-state.py` owns mutation per the cy-codex-loop contract).
• Read _techspec.md (deliverable source of truth) and _tasks.md (task roster) when present.
• List every task_NN.md and capture its frontmatter status: value (allowed: pending, in_progress, completed). When task_NN.md frontmatter disagrees with state.yaml, treat frontmatter as the source of truth.
• Note the canonical memory slot .compozy/tasks/<slug>/memory/qa-execution.md — Step 4 writes audit notes there before any status flips.

Step 2: Run the Baseline Verification Gate

1. Install dependencies with the repository-preferred command. 2. Run the canonical verification gate once before any audit work. Execute in fastest-first order: lint and type-check, then build, then unit tests, then integration tests. 3. If the E2E command is separate from the umbrella gate, decide whether to run it now or after runtime prerequisites are ready, then record that plan explicitly. 4. If the baseline fails, read the first failing output carefully and determine whether it is pre-existing or introduced by current work before moving on. 4a. Flaky-failure protocol. When a baseline command fails, before classifying as pre-existing or new, run the failing test in isolation 3-5 times on the same SHA. If it passes at least once without code changes, classify as flaky-suspect, record in audit-report.md under SUITE HEALTH SNAPSHOT (test name, attempts, retry outcome, suspected category), and do NOT promote to PASS via retry. STOP. Read `references/flaky-triage.md` in full before assigning a suspected category or proposing a quarantine.

Step 3: Audit Task Implementations (Compozy mode and any AI-implemented tasks)

Skip this step only when no task, phase, PRD, tech spec, or implementation-plan artifacts exist.

1. STOP. Read `references/independent-evaluator-protocol.md` in full before forming any task verdict. Tripwire summary: never accept the implementing agent's transcript, success message, or memory note as evidence. In Compozy mode, read the implementing agent's .compozy/tasks/<slug>/memory/<phase>.md artifacts and classify anomalies (genuine-failure / grader-bug / ambiguous-task / bypass-exploit) in the Errors / Corrections section of memory/qa-execution.md before judging the task. 2. Read each task_NN.md and its body. Summarize each task into a Task Implementation Matrix (column names mirror cy-codex-loop frontmatter):

• task_path (e.g., .compozy/tasks/<slug>/task_07.md)
• declared_status — literal frontmatter status: value
• title, type, complexity, dependencies — mirrored from frontmatter
• techspec_deliverable — linked section in _techspec.md when present
• Requirements, subtasks, checklist items, success criteria, dependent files
• implementation_evidence — files, modules, routes, commands, migrations, seeds, tests
• verification_evidence — commands executed, exit codes, output summaries
• qa_verdict — PASS | PARTIAL | FAIL | REOPEN | BLOCKED (distinct from declared_status)
• ai_audit_findings — red flag IDs that fired in Step 4 with verdict
• action — none | fixed | reopened-frontmatter | BUG-NNN.md filed
• linked_bugs — BUG IDs

3. Do not treat a task declared_status, checked checkbox, memory note, or prior agent summary as proof. Verify every completed or claimed-complete task against actual files, public behavior, automated tests, and acceptance criteria. 4. Classify each task with qa_verdict:

• PASS: every material requirement and success criterion has implementation and fresh verification evidence.
• PARTIAL: implementation exists but one or more non-critical requirements, tests, or evidence are missing.
• FAIL: claimed behavior does not work or a critical requirement is absent.
• REOPEN: the source task_NN.md has status: completed in frontmatter but the QA verdict is PARTIAL or FAIL.
• BLOCKED: audit cannot continue because a concrete prerequisite is missing.

Step 4: AI Test-Hygiene Scan (RF-1..RF-6)

1. STOP. Read `references/ai-implementation-audit.md` in full before scanning the test diff of any task with `declared_status: completed`. That file owns the Red Flag scanners (RF-1..RF-6), the Requirement→Test mapping rules, and the verdict matrix. 2. Run the scans against the diff since the task baseline (git log --follow <test_file>, git diff <baseline_sha>..HEAD). 3. Emit verdict FAIL automatically when scanners detect:

• Weakened assertions on P0/P1 Success Criterion (RF-2).
• .skip / .only / xit / t.Skip inserted in the diff (RF-1).
• Mocks inserted in tests whose corresponding TC declared External Dependencies as Integration/E2E (RF-3).
• Snapshot drift on P0/P1 with no requirement-change justification (RF-4).

4. Record findings in the Task Implementation Matrix column ai_audit_findings and in the per-task block of audit-report.md. 5. Apply the Requirement → Test mapping table from references/ai-implementation-audit.md. For every Success Criterion in task_NN.md (frontmatter or body) and every linked bullet in _techspec.md, find the corresponding test by name, reference, or assertion content. Mark each criterion covers / weak / missing. A checked item or status: completed without a covers row is an audit failure.

Step 5: Reopen, File Bugs, Write Memory

1. Mark incomplete completed tasks as REOPEN in the matrix. 2. In Compozy mode, write audit notes to .compozy/tasks/<slug>/memory/qa-execution.md using the canonical sections required by cy-codex-loop: Objective Snapshot, Important Decisions, Learnings, Files / Surfaces, Errors / Corrections, Ready for Next Run. This file must be written before any task_NN.md frontmatter is flipped (memory-precedes-status invariant). 3. Edit the offending task_NN.md frontmatter status: back to pending (or in_progress if salvageable). Never write to `state.yaml` — cy-codex-loop's update-state.py owns mutation; frontmatter wins because the next iteration reconciles from it. 4. File BUG-<num>.md under <audit-output-path>/audit/issues/ using assets/issue-template.md. Include:

• The task path under Reopens task:.
• The failed Success Criterion under Summary:.
• The original strict assertion (when RF-2 fired) under Root cause:.
• The red flag ID and verdict under Automation Follow-up: notes.
• The transcript anomaly classification (when applicable) under Related:.

5. When the missing work is a bounded root-cause fix inside the audit scope, you may implement it, add regression coverage, and rerun the task proof. Otherwise reopen the task — do not silently pass it.

Step 6: Quality Gates Verdict

1. Re-run the canonical verification gate from scratch after the last code change made during the audit. 2. Compile the Quality Gates section of audit-report.md. Each gate is PASS / FAIL / N/A:

• Flaky rate <2% in canonical suite.
• Zero FAIL from AI test-hygiene audit on P0/P1 tasks.
• Zero Critical / High issues open.
• Coverage delta ≥ baseline (no regression).
• Zero unresolved flaky-suspect on P0 flows.

3. A FAIL on any gate blocks an unconditional PASS verdict for the run.

Step 7: Write the Audit Report

1. Summarize the audit using assets/audit-report-template.md and write the report to <audit-output-path>/audit/audit-report.md. 2. Mandatory sections:

• Claim / Command / Exit code / Verdict per command executed in Step 2 and Step 6.
• AUTOMATED COVERAGE — support detected, harness, canonical command, required flows with classification, specs added or updated.
• TASK IMPLEMENTATION AUDIT — Compozy slug, plan sources, matrix totals, per-task verdicts, reopened/fixed/blocked tasks, links to bugs.
• SUITE HEALTH SNAPSHOT — flaky rate, flaky events list, mutation score (when harness exists), coverage delta vs baseline, blocked count, manual-only count, AI audit findings count.
• QUALITY GATES — PASS/FAIL/N/A per gate.
• ISSUES FILED — total, by severity, with Reopens task: annotations.

3. When running in a Compozy slug, the final audit-report.md PASS feeds cy-codex-loop's verify.last_status=PASS precondition for Phase E — do not call `update-state.py`; cy-codex-loop owns that mutation. 4. Report blocked scenarios, missing credentials, or environment gaps with the exact command or prerequisite that stopped execution.

Error Handling

• If command discovery returns multiple plausible gates, prefer the broadest repository-defined command and explain the tie-breaker.
• If E2E support signals are weak or contradictory, prefer explicit config files and runnable commands before claiming the repository supports E2E.
• If no canonical verify command exists, read references/project-signals.md, choose the broadest safe install, lint, test, and build commands for the detected ecosystem, and state that assumption explicitly.
• If a required live dependency is unavailable, validate every local boundary that does not require the missing dependency and report the blocked live validation separately.
• If a failure appears unrelated to the audited tasks, prove that with a clean reproduction before excluding it from the audit scope.
• If the repository has an E2E harness but credentials, runtime services, or test data prevent execution, keep the affected flow classified as blocked and report the exact prerequisite that is missing.
• If task_NN.md files are marked status: completed but contain unchecked subtasks, missing deliverables, or unverified criteria, do not call the audit a pass. Write memory/qa-execution.md first, then edit frontmatter status: back to pending or in_progress, and file BUG-<num>.md per Step 5. Never write to state.yaml.
• If a test fails and passes on retry without a code change, do not promote to PASS. Register as flaky-suspect per references/flaky-triage.md, record the event in the Suite Health Snapshot, and treat any unresolved flaky-suspect on a P0 flow as a blocker for the final verdict.
• If the AI test-hygiene scan (Step 4) detects weakened assertions, skipped tests, or mocks hiding integration in a task with declared_status: completed, do not call the audit a pass. Apply the verdict matrix in references/ai-implementation-audit.md, file BUG-<num>.md with Type Functional, and flip frontmatter status: per Step 5.

Companion: qa-execution

agent-output-audit validates that the implementing AI agent did what it claimed. qa-execution validates that a real human user can succeed at the product. They are complementary, not redundant:

• Run agent-output-audit to certify that task_NN.md status: completed reflects real work.
• Run qa-execution to certify that the product, taken as a whole, is acceptable to end users.

A Compozy slug typically wants both: audit the task implementations, then exercise the resulting product through user-flow QA. They share no output directory, no bug taxonomy, and no procedures — keep them separate.

AUDIT REPORT ------------ Claim: <what is being audited (e.g., "Compozy slug auth-refactor task_07 status: completed")> Compozy slug: <.compozy/tasks/<slug>/ or "n/a"> Command: <full verification command> Executed: <timestamp or relative time> Exit code: <0 or non-zero> Output summary: <key pass/fail lines, counts, build result> Warnings: <none or list> Errors: <none or list> Verdict: PASS or FAIL

AUTOMATED COVERAGE ------------------ Support detected: <yes or no> Harness: <playwright, cypress, webdriverio, generic, or none> Canonical command: <full E2E command> or none Required flows:

• <flow name>: <existing-e2e | needs-e2e | manual-only | blocked>
• <flow name>: <existing-e2e | needs-e2e | manual-only | blocked>

Specs added or updated:

• <spec path>: <why this spec changed>
• <spec path>: <why this spec changed>

Commands executed:

• <command> | Exit code: <0 or non-zero> | Summary: <key result>
• <command> | Exit code: <0 or non-zero> | Summary: <key result>

Manual-only or blocked:

• <flow name>: <reason>
• <flow name>: <reason>

TASK IMPLEMENTATION AUDIT ------------------------- Plan sources:

• <task/phase/spec path>

Summary:

• Tasks audited: <count>
• PASS: <count>
• PARTIAL: <count>
• FAIL: <count>
• REOPEN: <count>
• BLOCKED: <count>
• Fixed during audit: <count>

Results:

• Task: <task_NN.md path>

Declared status (frontmatter): <pending | in_progress | completed> Audit verdict: <PASS | PARTIAL | FAIL | REOPEN | BLOCKED> Techspec deliverable: <section in _techspec.md or "none"> Implementation evidence: <files, specs, commands> Verification evidence: <commands and outcomes> Requirement → Test mapping: <covers/weak/missing counts> Gaps: <none or missing requirements/checklist items> AI audit findings: <none | list of red flags from references/ai-implementation-audit.md with verdict> Transcript anomalies: <none | genuine-failure | grader-bug | ambiguous-task | bypass-exploit> Action: <none | fixed | frontmatter reverted to <status> | BUG-NNN filed> Reopened tasks (frontmatter reverted from completed):

• <task_NN.md path>: <reason> | New frontmatter status: <pending | in_progress> | Bug: <BUG-NNN or none>

Memory file written: <.compozy/tasks/<slug>/memory/qa-execution.md or "n/a"> state.yaml: read-only (cy-codex-loop owns mutation via update-state.py)

SUITE HEALTH SNAPSHOT --------------------- Flaky rate (canonical suite): <X.X%> (threshold: <2%) Flaky events this run: <count>

• <test name>: <attempts> attempts, retry outcome: <pass | fail>, category: <async-wait | concurrency | order-dep | external | non-determinism | other>

Mutation score (when harness exists): <X.X% on <module> | "n/a"> Coverage delta vs baseline: <+X.X% | -X.X% | unchanged> Blocked scenarios: <count> Manual-only items: <count> AI audit findings: <count of FAIL/PARTIAL verdicts from references/ai-implementation-audit.md>

QUALITY GATES -------------

• Flaky rate <2%: PASS | FAIL | N/A
• Zero FAIL from AI test-hygiene audit on P0/P1: PASS | FAIL | N/A
• Zero Critical/High issues open: PASS | FAIL | N/A
• Coverage delta ≥ baseline: PASS | FAIL | N/A
• Zero unresolved flaky-suspect on P0 flows: PASS | FAIL | N/A

Overall: PASS or FAIL (FAIL on any gate blocks unconditional final PASS)

ISSUES FILED ------------- Total: <number of BUG-*.md files created in audit-output-path/audit/issues/> By severity:

• Critical: <count>
• High: <count>
• Medium: <count>
• Low: <count>

Details:

• <BUG-ID>: <short-title> | Severity: <level> | Priority: <P0-P3> | Status: <pending | resolved | invalid | flaky-suspect | quarantined> | Reopens task: <task_NN.md path or "none"> | Red flag: <RF-1..RF-6 or "n/a">

BUG-<num>: <short-title>

Severity: Critical | High | Medium | Low Priority: P0 | P1 | P2 | P3 Type: Functional | Performance | Security | Data | Crash | Hygiene Status: pending | resolved | invalid | flaky-suspect | quarantined Reopens task: <task_NN.md path or "none">

Status values (aligned with cy-codex-loop issue_NNN.md frontmatter):

- pending — issue is open and unresolved

- resolved — fixed during this audit run and verified by re-run

- invalid — triaged as non-actionable (not a defect, duplicate, environmental)

- flaky-suspect — one run failed, retry passed; awaiting confirmation runs per references/flaky-triage.md

- quarantined — confirmed flaky after diagnosis; isolated from merge gate but still monitored (requires named owner and fix-by date)

Environment

• Build: <version or commit>
• OS: <operating system if relevant>
• Compozy slug: <.compozy/tasks/<slug>/ or "n/a">

Summary

<Describe the observable failure (or audit finding) in one short paragraph. For RF-* findings, name the red flag ID.>

Reproduction

<exact command, scan, or sequence>

Observed before the fix:

• <observable result>

Expected

<Describe the correct behavior or the Success Criterion that was not met.>

Root cause

<Describe the actual source of the failure, not the symptom. For RF-2, name the original strict assertion that was weakened.>

Fix

<Describe the production change (or test restoration) that fixed the root cause.>

Verification

• <narrow reproduction rerun>
• <broader regression or full gate rerun>
• <Requirement → Test mapping table updated, if applicable>

Impact

• Users Affected: <all / subset / specific role>
• Frequency: <always / sometimes / rarely>
• Workaround: <describe or "none">

Automation Follow-up

• Red Flag ID: <RF-1..RF-6 or "n/a">
• Verdict: FAIL | PARTIAL | PASS
• Required: Yes | No
• Status: Added | Pending | Blocked | N/A
• Spec / Command: <path, suite, or command>
• Notes: <rationale or blocker>

Flake Evidence (when Status is flaky-suspect or quarantined)

• Failure Pattern: consistent | intermittent | order-dependent | env-only
• Reproducibility Rate: <e.g. 3/10 runs>
• Suspected Category: async-wait | concurrency | order-dep | external | non-determinism | other
• Owner: <named person, not team>
• Fix-by Date: <YYYY-MM-DD>

Transcript Anomaly (when applicable)

• Classification: genuine-failure | grader-bug | ambiguous-task | bypass-exploit
• Evidence path: <.compozy/tasks/<slug>/memory/<phase>.md line or section>

Related

• Task: <task_NN.md path>
• Memory: <memory/qa-execution.md section>

AI Implementation Audit

This reference catalogs the observable signals qa-execution Step 4A uses when auditing test code produced by an AI agent. It defines what to scan for, how to classify findings, and how to record them. It does not explain why each pattern is wrong or how to fix it — that belongs to test-pattern/anti-pattern skills (test-antipatterns exists today; a positive test-patterns skill may be added later).

Contents

• When to apply
• Red Flag Scanners (RF-1..RF-6)
• Requirement → Test Mapping
• Verdict Matrix
• Recording findings
• Sources

When to apply

• Any task with declared_status: completed whose implementation includes new or modified test files.
• Any commit that touches both production code and its sibling test in the same change set.
• Any cy-codex-loop Compozy slug under .compozy/tasks/<slug>/ where the implementing agent self-reported success.

Red Flag Scanners

For each red flag, run the listed scan against the test diff since the task baseline. Use git log --follow <test_file> to recover the baseline. Emit the listed verdict when the flag fires.

RF-1 Skipped or disabled tests added

git diff <baseline_sha>..HEAD -- '*test*' '*spec*' \
  | rg -nP '^\+.*(\.skip\(|\.only\(|xit\(|xdescribe\(|t\.Skip\(|@pytest\.mark\.skip|@Ignore|fdescribe|fit\()'

Verdict: FAIL. File BUG-<num>.md with Type Functional and Status pending.

RF-2 Weakened assertions

Detect replacements from strict equality (toBe, toEqual, toStrictEqual) to permissive matchers in the same commit that flipped status: completed.

git diff <baseline_sha>..HEAD -- '*test*' '*spec*' \
  | rg -nP '^-.*(toBe|toEqual|toStrictEqual)\(' \
  | rg -nP '^\+.*(toBeDefined|toBeTruthy|toBeFalsy|toBeNull|toBeUndefined|toMatch\b|expect\.anything|expect\.any\b|assert\.NotNil|require\.NotEmpty)'

Also flag changes from numeric/string equality to .toContain(...) or .toMatchObject(...) with only one expected field.

Verdict: FAIL when the weakened assertion covers a P0/P1 Success Criterion. PARTIAL when it covers only edge case checks. Always require a BUG-<num>.md naming the original strict assertion in Root cause.

RF-3 Mocks inserted in tests classified as Integration or E2E

The TC declares Automation Target: Integration or E2E and lists an External Dependencies set — but the test file mocks one of those dependencies.

rg -nP '(jest\.mock\(|vi\.mock\(|nock\(|gomock\.|httpmock|patch\.object\(|patch\(.*Mock|sinon\.stub\(|MockBean\b)' <test_file>

Cross-reference each match against the TC External Dependencies list. Any mock targeting a dependency that the TC declared real is a violation.

Verdict: FAIL. Tag the BUG with mock-hides-integration. The fix must either remove the mock (preferred) or downgrade the TC to Automation Target: Manual-only with a documented reason.

RF-4 Snapshot or gold-file drift

git diff --name-only <baseline_sha>..HEAD \
  | rg -nP '(__snapshots__/.*\.snap$|testdata/golden/|/__fixtures__/|\.golden$|/fixtures/.*\.(json|yaml|yml)$)'

When any path matches, open each file and verify the change is justified by an explicit requirement. A snapshot updated without a corresponding requirement change is drift.

Verdict: FAIL when the snapshot covers a P0/P1 Success Criterion. PARTIAL elsewhere.

RF-5 Happy-path-only coverage

The TC is P0/P1 and the implementation has only positive-path assertions: no failure row in it.each/test.each, no expect(...).toThrow, no 4xx/5xx assertion, no empty/null/undefined input, no permission-denied case.

Verdict: PARTIAL. File BUG-<num>.md requesting the missing negative paths. Do not REOPEN unless External Dependencies make negative paths trivial.

RF-6 Test-implementation symbiosis

git log --oneline --name-only <baseline_sha>..HEAD \
  | awk '/^[a-f0-9]/ {commit=$0; next} {print commit, $0}' \
  | rg -nP '(\.test\.|\.spec\.|_test\.go|/test_)' \
  | sort -u

Group by commit and flag commits where both implementation and test sibling appear together without a third commit message that names the requirement. Then apply the Requirement → Test Mapping below.

Verdict: PARTIAL until the mapping below resolves to covers for every criterion. Otherwise FAIL.

Requirement → Test Mapping

For every Success Criterion in task_NN.md (frontmatter or body) and every linked bullet in _techspec.md, build the table:

Verdict definitions:

• covers — A specific assertion in the matched test references the literal value, behavior, or contract the criterion describes. The assertion is strict (equality, status code, error type, exact text).
• weak — A test exists in the criterion area but uses a permissive matcher, checks the wrong layer (internal state instead of public outcome), or only checks the happy path of a multi-path criterion.
• missing — No test references the criterion area, or every candidate test is .skip/.only-fenced, mocked away, or asserts unrelated state.

A weak row blocks PASS on a P0/P1 task. A missing row blocks PASS on any task. Record the table in the Task Implementation Matrix column ai_audit_findings and in verification-report.md under the per-task block.

Verdict Matrix

When multiple flags fire on the same task, take the strictest verdict.

Recording findings

Record findings in three places:

1. verification-report.md → TASK IMPLEMENTATION AUDIT block → per-task AI audit findings: field (list red flag IDs that fired with their verdicts). 2. verification-report.md → SUITE HEALTH SNAPSHOT → AI audit findings: count. 3. Compozy mode only: .compozy/tasks/<slug>/memory/qa-execution.md → Errors / Corrections section, before any frontmatter status flip (memory-precedes-status invariant).

Sources

• Anthropic — Demystifying Evals for AI Agents: independent evaluator principle.
• Florian Bruniaux — Claude Code Ultimate Guide: TDD with Claude: Verification Gap.
• Autonoma — Vibe Coding Best Practices: The Testing Checklist: Stanford/UIUC finding on vulnerability rates with AI-assistant trust.

Agent Output Audit Checklist

Mark every item as complete before claiming the audit is done.

Contract Discovery

• [ ] Root instructions and repository docs were read
• [ ] The canonical verify gate was identified or an explicit fallback was chosen
• [ ] The audit-output directory was resolved and audit/ subdir created
• [ ] Compozy mode was detected (yes/no with .compozy/tasks/<slug>/ path or "none")
• [ ] When in Compozy mode, state.yaml, _techspec.md, _tasks.md were read (state.yaml read-only)
• [ ] E2E support was determined (supported, manual-only, or blocked with evidence)

Baseline Verification Gate

• [ ] Dependencies were installed with the repository-preferred command
• [ ] The baseline verification gate was run before any audit work
• [ ] Verification order followed fastest-first: lint, build, unit tests, integration tests
• [ ] Any pre-existing failures were isolated with evidence
• [ ] E2E command planning was recorded explicitly when it is separate from the umbrella gate

Flaky Detection (Baseline)

• [ ] Each baseline failure was run in isolation 3-5 times on the same SHA before classification
• [ ] No PASS verdict was promoted from a single-retry rerun
• [ ] All flaky-suspect events were recorded in SUITE HEALTH SNAPSHOT with timestamp, attempts, retry outcome, and suspected category
• [ ] Flaky rate in the canonical suite is <2% (or documented as a known blocker)

Task Implementation Audit

Skip this section only if no task, phase, PRD, tech spec, or implementation-plan artifacts exist.

• [ ] Task/phase/spec artifacts were discovered and listed
• [ ] Every task marked completed or claimed complete was compared against actual implementation files
• [ ] Every material requirement, subtask, deliverable, and success criterion was mapped to evidence
• [ ] Checked boxes and status fields were treated as claims, not proof
• [ ] Public behavior or automated tests were executed for each material completed task
• [ ] Incomplete completed tasks were marked REOPEN or linked to a BUG-* issue
• [ ] Large missing features were not silently passed as audit success
• [ ] The audit report includes a Task Implementation Audit section with per-task verdicts
• [ ] Task frontmatter status: was used as the declared status; state.yaml was read but not written
• [ ] When running in .compozy/tasks/<slug>/, memory/qa-execution.md was written with canonical sections before any frontmatter status was flipped (memory-precedes-status invariant)

Independent Evaluator Stance

• [ ] The implementing agent's memory/<phase>.md artifacts were read before judging the task
• [ ] Transcript anomalies were classified (genuine-failure / grader-bug / ambiguous-task / bypass-exploit) in memory/qa-execution.md → Errors / Corrections
• [ ] No self-report (transcript success, [x] checkbox, memory done, frontmatter status: completed, PR description) was accepted as evidence

AI Test-Hygiene Scan (RF-1..RF-6)

• [ ] Test diff scanned for .skip / .only / xit / t.Skip since the task baseline (RF-1)
• [ ] Assertions in modified test files verified to not weaken existing checks (RF-2)
• [ ] Mocks in Integration/E2E classified tests audited against TC External Dependencies (RF-3)
• [ ] Snapshot or gold-file changes justified by a documented requirement change (RF-4)
• [ ] Happy-path-only coverage flagged on P0/P1 tasks (RF-5)
• [ ] Test-implementation symbiosis bisected against Requirement → Test mapping (RF-6)
• [ ] Requirement → Test mapping table produced (covers / weak / missing) for every REOPEN candidate

Final Verification

• [ ] The full verification gate was rerun after the last code change made during the audit
• [ ] Narrow E2E specs were rerun after the final code change when they were added or updated
• [ ] The canonical E2E command or covering subset was rerun when the repository supported E2E
• [ ] An audit report was produced from fresh evidence
• [ ] Blocked scenarios or missing prerequisites were disclosed explicitly

Quality Gates

• [ ] Flaky rate <2% in canonical suite
• [ ] Zero FAIL from AI test-hygiene audit on P0/P1 tasks
• [ ] Zero Critical / High issues open
• [ ] Coverage delta ≥ baseline (no regression)
• [ ] Zero unresolved flaky-suspect on P0 flows
• [ ] Suite Health Snapshot populated in audit report
• [ ] All Quality Gates evaluated PASS/FAIL/N/A; a FAIL on any gate blocks unconditional final PASS

E2E Coverage Guide

Read this reference when deciding whether the repository already supports automated end-to-end coverage and when that coverage must be added or updated.

What Counts as E2E

Treat E2E as regression coverage that exercises a public user or operator interface:

• Browser flows through the actual UI
• HTTP API flows through real entrypoints
• CLI or worker flows triggered through documented commands

Do not treat isolated unit or private helper tests as E2E proof.

High-Confidence Support Signals

Treat the repository as E2E-capable when at least one runnable command exists and at least one of these signals confirms the harness:

• Explicit commands such as e2e, test:e2e, playwright, cypress, or acceptance
• Framework configs such as playwright.config.*, cypress.config.*, or wdio.conf.*
• Existing spec directories such as e2e/, tests/e2e/, test/e2e/, or cypress/e2e/
• CI workflows that run the same E2E command

If only a weak signal exists, do not overclaim support. Record the ambiguity and confirm manually from repository docs or CI.

Flow Classification

Classify each changed or regression-critical public flow as one of:

• existing-e2e: matching automated coverage already exists and remains valid
• needs-e2e: repository supports E2E but the flow lacks adequate coverage
• manual-only: coverage intentionally stays manual because automation is the wrong tool
• blocked: repository supports E2E but required credentials, data, services, or environment are missing

Balanced Enforcement Policy

Require new or updated E2E coverage when repository support already exists and any of these are true:

• The flow is P0 or P1
• The flow is release-critical smoke coverage
• A bug fix restores a public regression
• A browser, HTTP, or CLI flow is exercised manually as part of the QA proof and no equivalent automated regression exists

Keep flows manual-only when they are primarily exploratory, usability-focused, or visual-design judgments that do not fit stable automation.

Evidence Rules

When E2E support exists:

• Record the canonical E2E command and any narrower spec command used for the fix
• Record each required flow and its classification
• List spec paths that were added or updated
• Re-run the narrow spec plus the canonical E2E command, or the smallest repository-defined subset that covers the touched critical flows

When support does not exist:

• Do not bootstrap a new framework during QA
• Keep live manual evidence
• Report the gap explicitly as manual-only or blocked

Flaky Test Triage

This reference defines vocabulary, diagnosis protocol, and quarantine policy for flaky tests encountered during qa-execution. It exists because retrying a failing test until it passes is the most common way real bugs reach production — and the qa-execution skill must not silently promote a flake to PASS via retry. It does not document how to fix each cause; that is a test-pattern concern.

Vocabulary

• flaky-suspect — A test failed once, passed on retry without any code change. Awaiting confirmation runs. Cannot be promoted to PASS without further evidence.
• quarantined — Confirmed flaky after isolation runs. Isolated from the merge gate but still executed for monitoring. Requires a named owner (a person, not a team) and a fix-by date.
• flake rate — Percentage of tests in the canonical suite that produced inconsistent verdicts across runs in the current window.

Cause classification

When labeling a flake event in the verification report, pick one category:

• async-wait — failure varies with timing of async I/O or DOM updates.
• concurrency — failure varies with parallel execution or shared mutable state.
• order-dep — failure depends on the order tests run (or the contents of beforeAll/beforeEach state).
• external — failure depends on an external resource (network, clock, FS, 3rd-party API).
• non-determinism — failure varies with intrinsic randomness (RNG without seed, LLM temperature, model output).
• orphan-code / fragile-locator — failure varies with DOM structure, selector instability, or dead code paths.

The category is recorded for triage; the fix belongs to the implementing engineer (and to test-pattern skills), not to this QA flow.

Diagnosis protocol

When a test fails in baseline or in a re-run, do not classify it as pre-existing or new until this protocol completes.

1. Isolate: Run the single failing test 3 to 5 times on the same SHA, in a clean working tree, with no other tests scheduled. Record each outcome. 2. Stress order: If isolation passes ≥ 1 time, run the surrounding describe block in randomized order 3 to 5 times. 3. Stress concurrency: If the suite supports parallel mode, run the affected file with the project's parallelization flag. 4. Bisect: If the test is new or recently modified, git bisect to identify the first commit that introduced the flake. 5. Classify using the cause categories above. Record Suspected Category in the issue.

Retry policy

• PROHIBITED: Promoting a FAIL to PASS because a single retry passed. This includes CI-level "retry on failure" features when their outcomes are not surfaced in the verification report.
• Allowed: Running the diagnosis protocol above and classifying the test as flaky-suspect or quarantined.
• Required: Every flake event (failure → retry → pass) is recorded in verification-report.md under SUITE HEALTH SNAPSHOT → Flaky events this run with test name, attempts, retry outcome, and suspected category.
• Threshold: Flake rate >= 2% in the canonical suite is a FAIL on the Flaky rate <2% Quality Gate. Above this, the QA run cannot conclude with an unconditional PASS.

Quarantine workflow

When a flaky-suspect is confirmed flaky after the diagnosis protocol (i.e., the test cannot be made reliable within the QA window), move it to quarantined:

1. Assign a named owner within 24 hours. Not a team. A person. Without an owner, the test is auto-removed from the suite after one sprint. 2. Set a fix-by date. Maximum two sprints from quarantine. Past that date, the test is removed and a BUG-<num> is filed under Type: Functional with Priority: P1. 3. Isolate from the merge gate. Quarantined tests must still run in CI but their result must not block merges. 4. Monitor: Each qa-execution run reports the quarantine count in SUITE HEALTH SNAPSHOT. 5. Re-entry gate: A quarantined test returns to the main suite only after 10 consecutive clean runs across CI and local. Document the 10 runs in the BUG-<num>.md resolution evidence.

Compozy mode interaction

When the failing test is associated with a task whose declared_status: completed and the task lives under .compozy/tasks/<slug>/:

• If the failure is flaky-suspect on a P0/P1 flow proving the task: degrade qa_verdict to PARTIAL, file BUG-<num>.md with Status flaky-suspect, and do not promote the task. Write the finding to memory/qa-execution.md → Errors / Corrections before flipping any frontmatter status (memory-precedes-status invariant).
• If the failure is flaky-suspect on a non-critical flow: record in the SUITE HEALTH SNAPSHOT, file BUG-<num>.md, but do not degrade the task verdict.
• A flaky-on-completion P0 task never passes the gate until the BUG is resolved or the flake is confirmed invalid.

Sources

• Trunk — The Ultimate Guide to Flaky Tests: retry-as-PASS is "kicking a can down the road"; ~45% of flakes are async-wait related per Luo et al.
• Gradle — A Pragmatist's Guide to Flaky Test Management: record every retry; a stable test over flaky product code looks like a flaky test.
• Harness — Flaky Tests: How to Find, Fix, and Prevent Them: healthy suites <1-2% flake, >5% indicates structural problems.
• Rainforest QA — A Practical Guide to Reducing the Burden of Flaky Tests: fix / re-run / disable / quarantine / delete spectrum.
• ThinkSys — Reduce Flaky Tests: A Practical Guide for QA Teams: 75% of flakes fail in correlated clusters; pipeline consequences for unowned flakes.

Independent Evaluator Protocol

This reference codifies the stance qa-execution Step 4A takes when auditing AI-implemented work. It exists because the agent that wrote the code interprets its own output charitably — and a self-report is not evidence.

Principle

The agent that wrote the implementation and the agent doing this audit are distinct evaluation contexts. The auditor never accepts the implementer's self-report as evidence.

This is not about distrust of the model. It is about how context affects evaluation: an agent that just spent two hours building a feature reads ambiguous output as success. An auditor reading the exit code, the diff, and the test file at its current state does not.

What counts as evidence

• Fresh re-execution of the smallest public proof (CLI invocation, HTTP request, browser flow, worker job) against the current state of the repository.
• Direct read of the test file at its current commit, matched against the literal acceptance criterion using the Requirement → Test mapping defined by the AI Implementation Audit reference (loaded separately from SKILL.md, not via this file).
• Static analysis of the diff since the task baseline (git log --follow <test_file>, git diff <baseline_sha>..HEAD -- <test_file>).
• Successful exit codes from canonical commands recorded with timestamps in verification-report.md.

What does NOT count as evidence

• The implementing agent's transcript or chat log claiming success.
• A "done" or "all green" message in memory/<phase>.md.
• A checkbox marked [x] in task_NN.md body or _techspec.md.
• The frontmatter status: completed field by itself.
• A test run whose output was discarded or summarized by the implementing agent.
• A PR description or commit message asserting that tests pass.

Sequence

When auditing a task under .compozy/tasks/<slug>/:

1. Read the implementer's artifacts before forming a judgment. Open every memory/<phase>.md file the implementing agent wrote during cy-codex-loop. Read for: tools the agent used to bypass blockers, errors it "fixed" by deleting an assertion, fallback paths it took when the real path failed, ambiguity it resolved unilaterally. 2. Classify anomalies found in the transcript into one of:

• genuine-failure — the agent encountered a real problem and did not resolve it.
• grader-bug — the agent encountered a test or check that was wrong; the resolution may be legitimate.
• ambiguous-task — the requirement was unclear and the agent picked an interpretation.
• bypass-exploit — the agent found a path that satisfies the literal test but not the requirement (e.g., hardcoded an expected value, skipped a step the test did not enforce).

3. Record classifications in `memory/qa-execution.md` → `Errors / Corrections` section. This write happens before any frontmatter status: flip (memory-precedes-status invariant from cy-codex-loop). 4. Then apply Step 4A's normal verification — re-execute the smallest proof, read the diff, and run the AI test-hygiene Red Flag scans (RF-1..RF-6) defined in the AI Implementation Audit reference, which Step 4A.3b loads directly from SKILL.md. 5. Then decide the qa_verdict. If transcript classification surfaced a bypass-exploit or genuine-failure not addressed by the implementation, the verdict cannot be PASS regardless of green tests.

Why this matters in Compozy mode

cy-codex-loop is built around the premise that an agent implements a task and self-reports completion via state.yaml and task_NN.md frontmatter. qa-execution is the independent evaluator in that loop. If the auditor accepts the implementer's framing, the loop has no real verification — only ceremonial verification — and task_NN.md status: completed becomes a coordination signal, not a quality signal.

Sources

• Anthropic — Demystifying Evals for AI Agents: "you won't know if your graders are working well unless you read the transcripts and grades from many trials… when a task fails, the transcript tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution."
• Florian Bruniaux — Claude Code Ultimate Guide: TDD with Claude: "the agent that writes the code must not be the same invocation that certifies it done. This is not about distrust of the model; it is about how context affects evaluation."
• InfoQ — Evaluating AI Agents in Practice: Benchmarks, Frameworks, and Lessons Learned: "An agent that works perfectly in a sandbox but silently misreports a failed refund in production hasn't passed any evaluation that counts."

Project Signal Guide

Use this guide when repository instructions do not already define the canonical QA contract.

Priority Order

1. Root instructions such as AGENTS.md, CLAUDE.md, or repository-specific agent docs 2. Dedicated umbrella commands in Makefile, Justfile, task runners, or CI wrapper scripts 3. CI workflows under .github/workflows/ 4. Ecosystem-native manifests such as package.json, go.mod, pyproject.toml, or Cargo.toml 5. Language-default commands as a last resort

Common Signals

Makefile or Justfile

Treat verify, check, ci, test, lint, build, start, run, and dev as high-confidence targets.

package.json

Prefer explicit scripts in this order:

1. verify, check, ci 2. test, test:ci, test:e2e, test:integration 3. lint, typecheck 4. build 5. start, dev, serve, preview

E2E support

High-confidence E2E signals include:

1. Runnable commands such as e2e, test:e2e, playwright, cypress, or acceptance 2. Framework configs such as playwright.config.*, cypress.config.*, or wdio.conf.* 3. Existing spec locations such as e2e/, tests/e2e/, test/e2e/, or cypress/e2e/ 4. CI workflows that clearly run the same E2E command

Treat the repository as E2E-capable only when a runnable command exists and at least one other signal confirms the harness.

Go modules

If no umbrella command exists, treat go test ./..., go build ./..., and repository formatting/lint commands as the minimum baseline. Prefer repository wrappers over direct Go commands when both exist.

Python projects

Look for pytest, tox, nox, ruff, mypy, python -m build, and any scripts declared in pyproject.toml.

Rust projects

Treat cargo test, cargo build, cargo fmt --check, and cargo clippy --all-targets --all-features -- -D warnings as strong defaults when the repository does not define wrappers.

Mixed Repositories

When multiple ecosystems exist, identify the product entrypoint first. Do not assume every manifest is part of the same runtime surface.

Scenario Selection Rules

Always cover:

1. A baseline verification gate 2. The workflows directly touched by the change 3. At least one adjacent regression-critical workflow 4. Startup or readiness if the change can affect bootstrapping 5. A realistic fixture path if the feature consumes external projects, repos, files, or APIs 6. An automation classification for each changed or regression-critical public flow when E2E support exists

E2E policy

When the repository already supports E2E, require new or updated automated coverage for:

1. Changed P0 or P1 flows 2. Release-critical smoke paths 3. Bug fixes that restore public regressions

Keep flows manual-only only when automation is the wrong tool, and mark flows blocked when the harness exists but credentials, data, or runtime prerequisites are missing.

Evidence Rules

Capture exact commands, inputs, outputs, and artifact paths. Prefer observable outcomes over interpretation.