Aep Gen Eval

Generator/Evaluator Pattern

A reusable design pattern for honest evaluation of agent-produced artifacts. Separates the agent that creates work (generator) from the agent that evaluates it (evaluator), because agents consistently praise their own work.

"When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre."

— Anthropic, "Harness Design for Long-Running Application Development"

This skill is both a utility library and a standalone skill:

• As a library: Other skills reference its references/ files for scoring, prompts, protocol, and findings format.
• As a standalone skill: Invoke directly to run a gen/eval loop on any artifact.

---

How Other Skills Use This

Cross-skill reference paths

After sync with aep- prefix, reference files are at:

.claude/skills/aep-gen-eval/references/scoring-framework.md
.claude/skills/aep-gen-eval/references/agent-contracts.md
.claude/skills/aep-gen-eval/references/eval-protocol.md
.claude/skills/aep-gen-eval/references/findings-format.md

---

The Core Principle

Generator and evaluator must be separate agents. This is not optional — it is the single most impactful quality improvement in agentic workflows.

Why:

1. Agents cannot honestly evaluate their own work (demonstrated by Anthropic research) 2. Self-evaluation produces inflated scores and rationalized problems 3. Separate evaluation catches issues the generator is blind to 4. The cost of a second agent is trivial compared to shipping broken work

Scaling up: generator/evaluator is the canonical instance of _adversarial

verification_. When one task produces many findings/claims that each need an

independent check, `/aep-workflow` generalizes this to a

fan-out of N verifiers/refuters — reusing this skill's scoring framework and

findings format per finding.

---

Reference Files

Read these files for detailed specifications. Each file is self-contained.

---

Standalone Usage

When invoked directly, this skill runs a gen/eval loop on any artifact.

Step 1: Identify the artifact

What is being evaluated? Options:

• A document (product context, architecture, design doc)
• Code changes (implementation, PR diff)
• An OpenSpec change (proposal, design, specs, tasks)
• A structured file (YAML, JSON config, migration plan)

Step 2: Choose execution mode

Step 3: Configure dimensions

Read references/scoring-framework.md and select the appropriate preset:

• Code: Completeness, Correctness, UX Quality, Security, Code Quality
• Product/design: Completeness, Consistency, Implementability, Security, Downstream Compatibility
• Documents: Accuracy, Executability, Completeness

Or define custom dimensions for the specific artifact.

Step 4: Spawn agents

Read references/agent-contracts.md for prompt templates. Customize the templates with:

• The artifact content
• The technical constraints
• The verification checklist (what the evaluator should check against the codebase)

Step 5: Process results

Read references/findings-format.md for how to consolidate, categorize, and present findings. Apply fixes to the artifact.

---

Design Decisions

Why a utility skill, not just reference files:

• A utility skill can be invoked directly (/aep-gen-eval) for ad-hoc validation
• It appears in the skill list, making the pattern discoverable
• It has its own description for triggering, so agents use it when appropriate
• The references/ directory is still accessible to other skills via path

Why not merge with `/aep-validate`:

• /aep-validate is a product-context skill with 4 specific modes (product, design, code, document)
• The gen/eval pattern is more general — it applies to any evaluation scenario
• /aep-validate consumes the gen/eval pattern; it is not the pattern itself

Why not keep in `/aep-launch`:

• Launch only sets up criteria; it doesn't run the pattern
• The scoring framework is consumed by build, validate, AND launch
• Keeping it in launch creates a confusing ownership model

---

Next Step

After running gen/eval, proceed based on what was evaluated:

• Product context → /aep-dispatch
• Design artifacts → /aep-launch
• Code → create PR or continue /aep-build
• Documents → publish or share

Agent Contracts

Role definitions and prompt templates for generator and evaluator agents. The core contract: the agent that produces work must never be the agent that evaluates it.

---

Table of Contents

1. Role Separation Principle 2. Generator Role 3. Evaluator Role 4. Protocol Checker Role 5. Context Assembly Rules 6. Prompt Templates

---

Role Separation Principle

---

Generator Role

Responsibility

The generator produces or validates an artifact by attempting to use it. In different contexts:

Generator constraints

• CAN identify issues it notices during its own work
• CAN fix issues between evaluation rounds (in loop mode)
• CANNOT modify verification_steps or passes in feature-verification.json
• CANNOT score its own work on the evaluation dimensions
• CANNOT override or dismiss evaluator findings

Generator output format

The generator produces a structured artifact or a findings list:

## Assessment of [item]

**Can implement?** Yes/No
**Missing details:**

- [specific gap that would cause guesswork]
  **Dependency gaps:**
- [what this item needs but doesn't declare]
  **Assumption mismatches:**
- [implicit assumption that could be wrong]

---

Evaluator Role

Responsibility

The evaluator independently assesses work against specifications. It has NO knowledge of the generator's internal reasoning or self-assessment.

Evaluator constraints

• MUST read the original spec/requirements (not the generator's interpretation)
• MUST score against the dimension scale definitions (not gut feel)
• MUST apply hard failure thresholds strictly
• MUST provide actionable fix suggestions for every finding
• MUST, for UI work, receive screenshot(s) of the running app (captured host-aware per executor/references/dogfood-validation.md) and score the Visual Design dimension against the project's calibration/<type>.yaml / design-system spec using its multimodal vision (Claude natively; Codex GPT-5.4)
• MUST NOT rationalize problems away ("this is probably fine because...")
• MUST NOT implement fixes or capture the screenshot itself (generator ≠ evaluator — the dogfood/capture step produces the image; the evaluator only judges it)
• CAN update passes, evaluated_by, round in feature-verification.json

Evaluator output format

# Evaluation Round <N>

## Findings

### [PASS/FAIL]: [Finding title] ([Dimension]: [Score])

- Steps to reproduce: [concrete steps]
- Expected: [what should happen]
- Actual: [what actually happens]
- Impact: [why this matters]
- Fix: [specific, actionable suggestion]

## Scores

- [Dimension 1]: [Score] — [justification referencing scale definition]
- [Dimension 2]: [Score] — [justification]
  ...

## Result: PASS / FAIL

[If FAIL: which thresholds were violated, what must be fixed]

## Verification Updates

[Which items in feature-verification.json were updated]

---

Protocol Checker Role

Responsibility

A specialized evaluator that checks whether an artifact is compatible with the downstream system that will consume it. Only used when validating structured artifacts (product context, configs).

Protocol Checker constraints

• MUST have the downstream protocol specification (not just the artifact)
• MUST check every required field exists
• MUST validate structural constraints (DAG validity, no cycles, valid references)
• MUST NOT evaluate quality (that's the evaluator's job)
• Focuses on: format compliance, field presence, structural validity

Protocol Checker output format

# Protocol Compatibility Report

## Required fields check

- [field]: present / MISSING
- [field]: present / MISSING (required by [downstream skill])

## Structural validation

- DAG validity: PASS / FAIL ([details])
- Cross-references: PASS / FAIL ([broken refs])
- Scoring compatibility: PASS / FAIL ([missing inputs])

## File conflict analysis

- [file]: modified by [story A] and [story B] in same slice

## Summary

[N] required fixes, [M] warnings

---

Context Assembly Rules

What each agent receives determines the quality of evaluation. Too much context degrades performance. Too little causes missed issues.

Generator context

Include:

1. The artifact being validated — full content 2. The artifact's purpose — what downstream consumer uses it 3. Technical constraints — stack, conventions, existing patterns 4. Dependencies — what this artifact builds on

Exclude:

• Full codebase (evaluator's job)
• History of how the artifact was created
• Other artifacts not directly consumed
• Evaluator's findings (agents work independently)

Evaluator context

Include:

1. The artifact being validated — full content 2. The original spec/requirements — NOT the generator's interpretation 3. Read access to the codebase — package.json, schemas, configs, source 4. The specific claims to verify — file paths, versions, API signatures

Exclude:

• Generator's self-assessment or findings
• Product vision or business context (unless evaluating product artifacts)
• Other evaluator's findings (if running multiple evaluators)

Protocol Checker context

Include:

1. The artifact — specifically the section being checked 2. The downstream protocol specification — exact field requirements, format rules 3. Structural constraints — DAG rules, naming conventions

Exclude:

• The codebase (not relevant for protocol checking)
• Quality dimensions (not its role)
• Business context

---

Prompt Templates

Generator Prompt (Artifact Validation)

You are a GENERATOR agent performing a dry-run validation. Your job is to mentally
walk through using this artifact and identify gaps that would cause problems for
the downstream consumer.

## The Artifact
{artifact_content}

## Downstream Consumer
This artifact will be consumed by: {consumer_description}

## Technical Constraints
{technical_constraints}

## Your Task
For each item in this artifact, attempt to mentally execute it and report:
1. Can it be done? Yes/No
2. Missing details — anything vague that would cause guesswork
3. Dependency gaps — does this item have everything it needs?
4. Assumption mismatches — any implicit assumptions that could be wrong?

Focus on PROBLEMS ONLY. At the end, produce a consolidated list of ALL changes needed.

Evaluator Prompt (Codebase Verification)

You are an EVALUATOR agent. Your job is to compare this artifact against the
ACTUAL state of the codebase and find mismatches. Read real files and verify claims.

## The Artifact
{artifact_content}

## What to Verify
{verification_checklist}

## Your Task
Read the actual files referenced in this artifact. For each claim, check:
1. Does the referenced file/function/type exist?
2. Does it have the signature/shape the artifact assumes?
3. Are version numbers and dependency versions correct?
4. Do import paths resolve correctly?

Report ALL mismatches. Be specific — include file paths and line numbers.
End with a severity-ranked list of required fixes.

Evaluator Prompt (Code Quality)

You are an EVALUATOR agent. Begin evaluation immediately.

Read these files:
1. {criteria_file} (scoring calibration)
2. {eval_request_file} (what to evaluate)
3. All spec files in {spec_directory}
4. {contracts_file} (if exists)
5. {verification_file} (if exists)

Then:
1. Review code changes
2. Test the running application if possible
3. Score each dimension per your criteria
4. Write structured feedback to {eval_response_file}

CRITICAL: Score honestly. Do not rationalize problems away.
Apply hard failure thresholds strictly.
Never modify verification_steps in feature-verification.json.

Product Design Evaluator Prompt

You are a PRODUCT DESIGN EVALUATOR. Your job is to review this product context
against user story mapping principles and the product vision. You are NOT checking
technical correctness — you are checking whether the RIGHT thing is being built.

## The Product Context
{product_context_yaml}

## Your Task — evaluate these dimensions:

1. WALKING SKELETON VALIDITY
   - Is Layer 0 the thinnest possible end-to-end user journey?
   - Can a user complete the crudest possible journey with ONLY Layer 0 stories?
   - Are there gold-plated features hiding in Layer 0 that belong in Layer 1+?
   - Are there infrastructure-only stories with no user-facing change?

2. LAYER ORDERING
   - Does each layer add a meaningful new user capability?
   - Is the ordering optimal — highest-value capabilities earliest?
   - Could any layer be reordered for better incremental delivery?

3. VISION ALIGNMENT
   - Does every story trace back to the opportunity brief?
   - Are there orphan stories that serve no stated user need?
   - Has scope crept beyond the MVP contract?
   - Do the stories serve the JTBD (jobs to be done)?

4. INVEST COMPLIANCE
   - Independent: Can stories run without hidden coupling?
   - Negotiable: Are stories outcomes, not implementation prescriptions?
   - Valuable: Does each story deliver observable user value?
   - Estimable: Is each story clearly scoped with known complexity?
   - Small: Are L-complexity stories actually multiple stories bundled?
   - Testable: Does each story have verifiable acceptance criteria?

5. DEPENDENCY GRAPH QUALITY
   - Do dependencies reflect real value delivery order?
   - Are there artificial dependencies (sequencing that isn't necessary)?
   - Can more stories run in parallel with fewer dependencies?

Score each dimension 1-5 using the Product & Design scales.
Apply story mapping hard failure thresholds.
For each issue, suggest a specific fix (reorder, split, defer, remove).

Protocol Checker Prompt

You are a PROTOCOL CHECKER. Your job is to verify this artifact is compatible
with the downstream protocol that will consume it.

## The Artifact
{artifact_content}

## The Downstream Protocol
{protocol_specification}

## Your Task
1. Are all required fields present on every item?
2. Is the dependency graph a valid DAG (no cycles, no missing references)?
3. Can the scoring/ranking algorithm be computed with available fields?
4. Are there file-level conflicts between parallel items?
5. Can the downstream system create its required artifacts from this data?

Produce a compatibility report with specific fixes needed.

Evaluation Protocol

The request→response→fix loop for running generator/evaluator cycles. Covers execution contexts, signal files, verification tracking, and convergence rules.

---

Table of Contents

1. Execution Contexts 2. The Eval Loop 3. Signal Files 4. Feature Verification JSON 5. Convergence Rules

---

Execution Contexts

The gen/eval pattern can execute in three different contexts. The protocol is the same; the mechanics differ. When /aep-build runs the loop, the context tracks the executor mode in play (see aep-executor/references/backends.md): legacy (pinned tmux) → Context A; native-bg-subagent / claude-bg (foreground Task subagent) → Context B mechanism, worktree-bound; codex-subagent / codex-exec (codex exec --cd with the aep-evaluator role) and workflow (verify stage) → Context C mechanism, in-host. For the native modes the evaluator prompt is delivered at spawn time — no readiness wait, no separate send, no teardown.

Context A: Tmux Split Panes (Workspace — used by /aep-build under legacy/pinned-tmux)

Generator runs in the top tmux pane. Evaluator is spawned as a separate agent instance in the bottom pane.

# Generator spawns evaluator in bottom pane. $EXECUTOR is the INTERACTIVE session command
# from detect() (claude → "claude --dangerously-skip-permissions";
# codex → "codex --dangerously-bypass-approvals-and-sandbox"); defaults to the claude form when unset.
tmux split-window -v -c "$(pwd)" "${EXECUTOR:-claude --dangerously-skip-permissions}"

# Generator returns focus to top pane
tmux select-pane -t :.0

# Generator waits for evaluator to initialize, then sends bootstrap prompt.
# Use -l for the (multi-line) prompt, then a single Enter to submit once.
sleep 10
tmux send-keys -t :.1 -l -- "<evaluator prompt>"
tmux send-keys -t :.1 Enter

# Generator polls for response file
while [ ! -f .dev-workflow/signals/eval-response-<N>.md ]; do sleep 15; done

# Generator reads response and kills evaluator pane
tmux kill-pane -t :.1

When to use: Autonomous workspace implementation. The evaluator needs to test a running application (agent-browser, curl, etc.).

Note: Use tmux split-window, not cmux split. The generator runs inside tmux but was not spawned by cmux, so it cannot use cmux socket commands.

Context B: Parallel Agent Tool Calls (Main Session — used by /aep-validate)

Generator and evaluator are spawned as parallel agents using the Agent tool. They work independently and their results are consolidated after both complete.

Launch in parallel:
  Agent(subagent_type="Plan", prompt="<generator prompt>")
  Agent(subagent_type="Plan", prompt="<evaluator prompt>")
  Agent(subagent_type="Plan", prompt="<protocol checker prompt>")  # optional

Wait for all agents to return.
Consolidate findings.

When to use: Validating artifacts on the main branch. No running application needed. Agents need read access to the codebase but don't modify it.

Context B mechanism in /aep-build (native-bg-subagent / claude-bg): the generator

spawns a single foreground Task subagent with the evaluator prompt; it

inherits the worktree cwd and returns on completion. Same mechanism as above,

but worktree-bound and sequential — not the main-session read-only use.

Context C: Subagent Spawning (CI/Automation)

Generator and evaluator are spawned via the Claude API or SDK as separate conversations.

When to use: Automated pipelines, CI checks, scheduled validation.

Context C mechanism in /aep-build (codex-subagent / codex-exec / workflow):

a bounded codex exec --cd <worktree> one-shot with the aep-evaluator

role, or the workflow's worktree-isolated verify stage. Programmatic spawn,

but worktree-bound review — not an API/SDK CI job.

---

The Eval Loop

Single-pass mode (validate, design review)

1. Assemble context for each agent role
2. Spawn all agents in parallel
3. Wait for all agents to complete
4. Consolidate findings (see findings-format.md)
5. Present findings to user
6. Apply approved fixes
7. Done

Multi-round mode (build, code review)

Round 1:
  1. Generator writes eval-request.md
  2. Evaluator reads request + artifacts + criteria
  3. Evaluator scores, writes eval-response-1.md
  4. Generator reads response
  5. If PASS → done
  6. If FAIL → generator fixes issues

Round 2:
  1. Generator writes updated eval-request.md (notes what was fixed)
  2. Evaluator re-evaluates
  3. If PASS → done
  4. If FAIL → generator fixes again

...repeat up to max_rounds (default 5)...

If not converged → escalate to human

Choosing the mode

---

Signal Files

Used in multi-round mode (workspace context). Files live in .dev-workflow/signals/.

eval-request.md (generator writes)

# Evaluation Request — Round <N>

## What to evaluate

- [summary of implementation state]
- [which tasks are complete]

## Changes since last round

- [what was fixed since previous evaluation, or "first evaluation"]

## Known issues

- [anything the generator is aware of but hasn't fixed yet]

## Files changed

[output of git diff --stat "$BASE"...HEAD (integration branch; see git-ref)]

eval-response-N.md (evaluator writes)

# Evaluation Round <N>

## Findings

### [PASS/FAIL]: [Finding title] ([Dimension]: [Score])

- Steps to reproduce: [concrete steps]
- Expected: [what should happen]
- Actual: [what actually happens]
- Impact: [why this matters]
- Fix: [specific, actionable suggestion]

## Scores

- Completeness: [1-5] — [justification]
- Correctness: [1-5] — [justification]
- UX Quality: [1-5] — [justification]
- Security: [1-5] — [justification]
- Code Quality: [1-5] — [justification]

## Result: PASS / FAIL

[If FAIL: which hard failure thresholds were violated]

## Verification Updates

[Which items in feature-verification.json were updated, with new pass/fail status]

status.json (generator updates at phase boundaries)

{
  "phase": 5,
  "phase_name": "code-review",
  "eval_round": 2,
  "eval_result": "fail",
  "recovery_rung": "reground",
  "completion_pct": 75,
  "updated_at": "2026-03-30T12:00:00Z"
}

recovery_rung (optional) tracks which rung of the recovery ladder the generator is on when eval_result keeps failing — same_fix | reground | fresh_generator | decompose. The autopilot tick reads it to know the ladder is being climbed before it emits an eval_not_converging escalation.

needs-human.md (worker writes — the human-gate record)

When the loop cannot converge (or any decision needs the human), the worker appends to .dev-workflow/signals/needs-human.md and sets "blocked_on": "human" in status.json:

## <ISO8601> — Phase 5 (eval round <N>)

**Question:** <the decision needed, with options considered>
**Context:** <why the generator/evaluator pair can't resolve it>

After acting on the answer the worker appends resolved: <summary> and clears blocked_on. How the question reaches the human is per launch mode — see the Human-Gate Protocol in aep-executor/references/backends.md.

---

Feature Verification JSON

Task-level tracking for code evaluation. Format is intentionally JSON — models tamper with JSON less than Markdown.

Schema

[
  {
    "task": "string — task description from tasks.md",
    "commit_sha": "string — git short SHA (8 chars), null until task is committed",
    "verification_steps": [
      "string — concrete, executable verification step",
      "string — another step"
    ],
    "passes": false,
    "evaluated_by": null,
    "round": null,
    "notes": null
  }
]

Field ownership

Critical rule: The generator MUST NOT modify verification_steps, passes, evaluated_by, round, or notes. The generator may write commit_sha after each Phase 4 task commit. Only the evaluator or a human can update the verification fields. This ensures the generator cannot mark its own work as passing.

Example (real-world, round 1)

[
  {
    "task": "feat: expose WORKSPACE_CONTAINER DO binding in wrangler config",
    "commit_sha": "a1b2c3d4",
    "verification_steps": [
      "wrangler.jsonc includes workspace_container durable_object binding",
      "WorkspaceContainer class is exported from the entrypoint",
      "wrangler dev starts without binding errors"
    ],
    "passes": true,
    "evaluated_by": "evaluator-round-1",
    "round": 1,
    "notes": "All three steps verified. Binding present, class exported, dev server starts clean."
  },
  {
    "task": "feat: add source_type column to marketplace_plugins table",
    "commit_sha": "e5f6g7h8",
    "verification_steps": [
      "Drizzle schema includes source_type column with enum constraint",
      "Migration generated and applies cleanly",
      "Existing rows default to 'github'"
    ],
    "passes": false,
    "evaluated_by": "evaluator-round-1",
    "round": 1,
    "notes": "Schema column added but migration not generated (db:generate not run). Deployment will fail."
  }
]

---

Convergence Rules

When to stop the loop

Escalation format

# Escalation — Eval Loop Not Converging

## Round history

- Round 1: FAIL (Correctness 2, Security 2) — 4 findings
- Round 2: FAIL (Correctness 3, Security 2) — 3 findings (1 fixed, 0 new)
- Round 3: FAIL (Security 2) — 2 findings (1 fixed, 0 new)

## Persistent issues

1. [Issue that generator cannot fix — explain why]
2. [Issue that requires architectural decision]

## Recommendation

[What the human should decide or do]

Round budgets

Findings Format

How to consolidate, categorize, and present findings from generator/evaluator agents. Used when multiple agents produce findings that must be merged into a single action list.

---

Severity Categories

Every finding falls into one of three categories:

Classification heuristics

---

Deduplication

Multiple agents often find the same issue from different angles. Merge duplicates:

How to detect duplicates

Two findings are duplicates if they: 1. Reference the same item (story, field, file, endpoint) 2. Describe the same root cause (even if symptoms differ) 3. Would be fixed by the same change

How to merge

When merging duplicate findings:

• Keep the higher severity classification
• Combine evidence from both agents
• Note which agents found it: "found by Generator + Evaluator"
• Keep the most actionable fix suggestion

Example

Generator found: "Story api-storage-router accepts creatorId from client — implementer needs clarification on ownership model"

Evaluator found: "Story api-storage-router has a security issue — creatorId should come from session, not client input, to prevent cross-user access"

Merged finding:

BLOCKING: creatorId must come from session, not client input
- Found by: Generator (missing detail) + Evaluator (security issue)
- Impact: Any user could upload to another user's creator profile
- Fix: Derive creatorId from session.user.id in all storage procedures

---

Presentation Format

Summary line

Validation complete: {N} blocking, {M} important, {K} minor issues found.

Findings list (grouped by severity)

## Blocking ({N} issues)

1. **[Short title]** — found by [agent(s)]
   - Impact: [what breaks if not fixed]
   - Fix: [specific, actionable change]

2. **[Short title]** — found by [agent(s)]
   - Impact: [what breaks]
   - Fix: [what to do]

## Important ({M} issues)

3. **[Short title]** — found by [agent(s)]
   - Impact: [what friction this causes]
   - Fix: [suggestion]

## Minor ({K} issues)

4. **[Short title]** — found by [agent(s)]
   - Impact: [cosmetic/optional]
   - Fix: [suggestion]

Table format (for large finding sets)

When there are 10+ findings, a table is more scannable:

| # | Severity | Issue | Found by | Fix |
|---|----------|-------|----------|-----|
| 1 | Blocking | creatorId from client = security hole | Gen + Eval | Derive from session |
| 2 | Blocking | Missing dispatch_epoch field | Protocol | Add top-level field |
| 3 | Important | Zod v4 incompatibility | Evaluator | Hand-write schemas |
| ... | | | | |

---

Changelog Entry Format

After applying fixes, append a changelog entry to the artifact:

- date: <ISO 8601 date>
  author: aep-gen-eval
  summary: >
    Generator/evaluator validation ({mode}). Found {N} blocking, {M} important,
    {K} minor issues. Fixed: [brief list of key fixes applied].

Mode labels for changelog

---

Rules for Applying Fixes

1. Only modify the artifact being validated — never create new files or modify other artifacts as a side effect 2. Preserve the artifact's existing structure — don't reorganize sections unless the fix requires it 3. Add, don't replace — when adding missing fields, don't remove existing fields 4. Mark decisions as open questions — if a fix requires a judgment call the agent can't make, add it as an open_question with a default assumption and revisit trigger 5. Present findings before fixing — always show the user the findings and let them approve before applying changes. They may disagree with some findings or want to prioritize differently. 6. Fix blocking issues first — if time is limited, blocking issues take priority over important, which take priority over minor

---

Open Questions Format

When a finding requires a decision the agent can't make:

open_questions:
  - id: question-id
    question: "What is the correct approach for X?"
    default_assumption: "Do Y (most common pattern)"
    revisit_trigger: "If Z happens, reconsider"
    raised_by: gen-eval validation
    date: <ISO 8601>

This pattern ensures the artifact is usable with the default assumption, but the decision point is visible for human review.

Change-Strategy Recovery Ladder

When the Phase 5 gen/eval loop FAILs, the default behavior is for the same generator to retry the same way — fix the FAIL items, re-request evaluation, repeat. After max_rounds (default 5) it escalates to a human. The failure mode this guards against is strategy stagnation: the generator keeps applying the approach that already failed, burning rounds without exploring a genuinely different path.

This reference defines an escalating recovery ladder. Each rung tries something structurally different from the last, so the system exhausts real strategy changes before a human gate — not five copies of the same attempt.

The evaluator never climbs this ladder. Generator≠evaluator separation still holds: the evaluator scores; the generator (or a fresh generator) is the only role that "tries a new approach." A re-grounded read, a fresh generator, and a decomposition are all generator-side moves.

---

Table of Contents

1. The Ladder 2. When to Skip the Ladder 3. State Tracking 4. Spawning a Fresh Generator (Rung 4) 5. Cross-References

---

The Ladder

Round numbers are tunable per project; the shape is what matters — each rung is a strictly larger change of strategy than the one below it.

Round 1–2 — Same fix (current behavior)

The generator reads the latest eval-response-<N>.md, fixes the FAIL items in place, updates eval-request.md, and re-requests evaluation. This is the cheapest rung and resolves most failures (typical convergence is 2–3 rounds). No strategy change is warranted yet — the first couple of FAILs are usually ordinary bugs, not a stuck approach.

Round 3 — Re-ground

Context may have rotted: the generator has been editing for several rounds and its working memory of the spec has drifted. Before fixing again, the generator re-reads the full source of truth from scratch — the spec, the design doc, and the contracts — rather than reasoning from its in-context summary. It then re-attempts the FAIL items against that fresh reading. This catches the common case where the FAIL persists because the generator has been solving the wrong problem.

Round 4 — Different approach (fresh generator)

Re-grounding didn't converge, which suggests the generator is anchored on a design path that cannot satisfy the spec. The stuck generator cannot reliably unstick itself — it will keep returning to the same solution. So spawn a fresh generator that has none of the prior context except an explicit framing:

The previous approach failed on X (cite the persistent FAIL findings). Do not continue that approach. Re-read the spec/design/contracts and take a different design path.

The fresh generator works in the existing worktree (the prior commits remain; it can revert or rework them). See Spawning a Fresh Generator for the host-agnostic spawn contract.

Round 5 — Decompose

If even a fresh approach FAILs, the story is likely too large to land as one unit. The generator (fresh or original) proposes a split into smaller sub-stories / sub-tasks and attempts the smallest viable slice — the thinnest piece that can PASS on its own. The proposed split is surfaced, not silently applied: write it to eval-request.md and the human-gate record so the human (and the autopilot) can see the story has been re-shaped. Landing one slice and deferring the rest is a legitimate outcome of this rung.

After Round 5 — Human gate

Only once every rung has been tried does the loop escalate. This is the eval_not_converging escalation (needs-human.md + blocked_on: human in status.json; see eval-protocol.md → needs-human gate record). The escalation should record the ladder history — which rungs were attempted and why each failed — so the human inherits a genuinely-explored problem, not five identical attempts.

---

When to Skip the Ladder

The ladder is for convergence failures — the generator can't get the work to PASS. Some FAILs are not convergence problems and escalate immediately, skipping all rungs:

• Hard-failure / security FAIL that needs human judgment — e.g. an auth-model gap, a data-exposure risk, or any finding whose fix requires a product/security decision the agent is not authorized to make. Trying "a different approach" on a security boundary is worse than asking. Escalate on the first such FAIL.
• Spec contradiction — the FAIL is caused by the spec itself being internally inconsistent or wrong. No generator strategy can fix a contradictory spec; this needs a human to amend the spec.
• Missing external dependency / access — the work cannot proceed without something outside the worktree (a credential, an unbuilt upstream service). Decomposing won't help.

In these cases, escalate with the appropriate type immediately and note that the ladder was deliberately skipped.

---

State Tracking

Which rung we're on is derived, not free-standing — it follows the eval round count plus an explicit marker so a recovering agent (after a context reset) lands on the right rung:

• `eval_round` in .dev-workflow/signals/status.json is the primary driver (round 3 ⇒ re-ground, round 4 ⇒ fresh generator, etc.).
• `recovery_rung` in status.json records the rung explicitly — one of same_fix | reground | fresh_generator | decompose — so the rung is unambiguous even if rounds and rungs are re-tuned, and so the autopilot can read intent without re-deriving it. A fresh generator (rung 4) reads recovery_rung to learn it must take a different path rather than resume the stuck one.

{
  "phase": 5,
  "eval_round": 4,
  "recovery_rung": "fresh_generator",
  "eval_result": "fail",
  "blocked_on": null,
  "updated_at": "2026-06-16T12:00:00Z"
}

The workspace owns this state and advances its own rung — the autopilot only observes it and nudges (see Cross-References). The autopilot does not climb the ladder on the workspace's behalf.

---

Spawning a Fresh Generator (Rung 4)

The v1.8.0 spawn contract for the fresh generator (host-agnostic; same rules as any executor spawn):

1. Mode: native-bg-subagent — spawned via the Agent tool with run_in_background: true, no team. It runs as an in-process background subagent. 2. Worktree: it inherits the EXISTING worktree (.feature-workspaces/<name>). The prior generator's commits are present; the fresh generator may revert, rework, or build on them — but its prompt forbids resuming the stuck approach. 3. Liveness: it MUST pass the post-spawn liveness probe — skills/patterns/executor/scripts/spawn-liveness-probe.sh <ws> <agent_id>. A spawn call returning is not evidence the worker started; the probe confirms worktree activity, and the caller separately confirms the subagent process exists (TaskList shows <agent_id>). If the probe fails, tear down and re-spawn. 4. Gate-and-park: like any generator, the fresh generator gates and parks for human input when it hits a decision it can't resolve — it does not invent product/security answers.

The fresh generator is still a generator: the evaluator role is untouched, and the generator≠evaluator boundary is preserved across the swap.

---

Cross-References

Scoring Framework

Calibration document for evaluator agents. The evaluator is a separate agent from the generator — this separation is critical because agents consistently praise their own work.

"When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre."

— Anthropic, "Harness Design for Long-Running Application Development"

---

Table of Contents

1. Default Dimensions (Code) 2. Hard Failure Thresholds 3. Dimension Presets 4. Product & Design Dimensions 5. Document Dimensions 6. Few-Shot Examples 7. Anti-Patterns 8. Customization Guide

---

Default Dimensions (Code)

Evaluate each dimension on a 1–5 scale. Score honestly — the value of evaluation comes from catching problems the generator missed.

Customize per project: These dimensions are defaults. Anthropic's research found that scoring dimensions should be task-specific and weighted toward areas where the model falls short. Adjust based on where you observe the generator producing mediocre output.

1. Completeness (1–5)

Does the implementation cover all tasks and specs?

2. Correctness (1–5)

Does the implementation work as specified? Are edge cases handled?

3. UX Quality (1–5)

Is the interface intuitive, responsive, and accessible?

4. Security (1–5)

Input validation, auth checks, data exposure?

5. Code Quality (1–5)

Conventions, maintainability, performance?

6. Visual Design (1–5)

Does a screenshot of the running UI match the project's design system? Evaluated by feeding a screenshot of the running app to the multimodal evaluator, scored against the project's design-system / calibration spec (calibration/<type>.yaml, e.g. calibration/visual-design.yaml) — spacing rhythm, visual hierarchy, brand/token consistency, alignment, and overall polish.

Hard failure: Visual Design < 3 for the .5 polish layer — a screenshot that does not match the design system blocks the polish layer from passing.

>

Host-aware capture: The screenshot is captured per executor/references/dogfood-validation.md — /agent-browser:dogfood on Claude, native in-app browser / computer-use on Codex (GPT-5.4 multimodal), or a Playwright script — and the resulting image is fed to the in-host multimodal evaluator (Claude natively; Codex GPT-5.4). This keeps the visual judgment in-host and removes the human dependency for routine design-system checks.

---

Hard Failure Thresholds

Any of these conditions means the evaluation FAILS and the generator must fix before re-evaluation:

• Completeness below 4 — Missing features are not acceptable
• Correctness below 3 — Broken flows must be fixed
• Security below 3 — Security gaps must be addressed
• Any single dimension below 2 — Critical deficiency

Overall pass: All dimensions >= 3 AND Completeness >= 4 AND no dimension at 1.

---

Dimension Presets

Select the preset matching the artifact type, then adjust with the user during evaluator setup.

UI-heavy (forms, dashboards, layouts)

Dimensions:  Completeness, Correctness, UX Quality, Visual Design, Originality, Accessibility
Weight:      UX Quality (high), Visual Design (high), Originality (high)
De-weight:   Code Quality (still check but don't hard-fail)
Add:         Visual Design — score a screenshot against calibration/visual-design.yaml (multimodal)
             Originality — penalize generic "AI slop" (purple gradients, card layouts)
             Accessibility — WCAG AA compliance, keyboard navigation, screen readers
Hard fail:   UX Quality < 3, Visual Design < 3, Completeness < 4

API-only (endpoints, services, integrations)

Dimensions:  Completeness, Correctness, API Design, Security, Performance
Weight:      Correctness (high), Security (high)
Drop:        UX Quality (no frontend)
Add:         API Design — consistent naming, proper status codes, pagination, error format
             Performance — response times, query efficiency, no N+1
Hard fail:   Correctness < 3, Security < 3

Security-sensitive (auth, payments, data handling)

Dimensions:  Completeness, Correctness, Security, Data Privacy, Code Quality
Weight:      Security (high), Data Privacy (high)
Drop:        UX Quality (unless auth UI is involved)
Add:         Data Privacy — PII handling, encryption at rest, audit logging
Hard fail:   Security < 4, Data Privacy < 4

Data pipeline (ETL, migrations, batch processing)

Dimensions:  Completeness, Correctness, Performance, Data Integrity, Error Recovery
Weight:      Data Integrity (high), Performance (high)
Drop:        UX Quality, Security (unless processing sensitive data)
Add:         Data Integrity — no data loss, idempotent operations, schema validation
             Error Recovery — partial failure handling, retry logic, dead letter queues
Hard fail:   Data Integrity < 4, Completeness < 4

Mixed / Full-stack

Dimensions:  Completeness, Correctness, UX Quality, Visual Design, Security, Code Quality
Weight:      All equal (default)
Add:         Visual Design — when the feature ships UI, score a screenshot against
             calibration/visual-design.yaml (multimodal); omit for non-UI slices
Adjust:      Weight toward the area the user identifies as highest risk
Hard fail:   Default thresholds (any < 3, Completeness < 4); Visual Design < 3 on UI/.5 layers

---

Product & Design Dimensions

When evaluating product context, architecture, or design artifacts (not code):

Completeness (1–5)

Consistency (1–5)

Implementability (1–5)

Security (1–5)

Downstream Compatibility (1–5)

Walking Skeleton Validity (1–5)

Does Layer 0 represent the thinnest possible end-to-end user journey?

"Build a skeleton that can walk before building a perfect leg." — Jeff Patton

Layer Ordering (1–5)

Does each layer add meaningful new user capability in the right order?

Vision Alignment (1–5)

Do all stories trace back to the opportunity brief and product vision?

INVEST Compliance (1–5)

Do stories follow the INVEST criteria (Independent, Negotiable, Valuable, Estimable, Small, Testable)?

Story Mapping Hard Failure Thresholds

• Walking Skeleton Validity < 3 — Layer 0 is not minimal enough
• Vision Alignment < 3 — Stories have drifted from the product vision
• INVEST Compliance < 3 — Stories are not actionable by an autonomous agent

---

Document Dimensions

When evaluating structured documents (RFCs, migration plans, runbooks):

Accuracy (1–5)

Executability (1–5)

Completeness (1–5)

Same as code completeness scale — are all required sections present?

---

Few-Shot Examples

Example 1: Lenient Evaluator (BAD)

This is how an uncalibrated evaluator behaves — finding real problems then rationalizing them away:

EVALUATOR OUTPUT (UNCALIBRATED):

I tested the login form. Clicking "Submit" with empty fields shows no error message
and the form just silently fails. However, this is a minor UX issue since users
would typically fill in the fields before submitting. The core login flow works
when valid credentials are provided.

I also noticed the delete endpoint doesn't check if the requesting user owns the
resource. But since this is an internal tool, the risk is probably low.

Score: Completeness 4, Correctness 4, UX 4, Security 3, Code Quality 4
Result: PASS

Why this is wrong: The evaluator identified two legitimate problems (no form validation feedback, missing authorization check) but rationalized both away. The security issue alone should fail the evaluation.

Example 2: Calibrated Evaluator (GOOD)

This is how a properly calibrated evaluator handles the same findings:

EVALUATOR OUTPUT (CALIBRATED):

## Findings

### FAIL: Missing form validation feedback (Correctness: 2, UX: 2)
- Steps: Navigate to /login → Click Submit with empty fields
- Expected: Error messages showing which fields are required
- Actual: Form silently fails, no visual feedback
- Impact: Users have no way to know what went wrong
- Fix: Add client-side validation with inline error messages

### FAIL: Missing authorization on delete endpoint (Security: 2)
- Steps: POST /api/items/42/delete with a different user's auth token
- Expected: 403 Forbidden
- Actual: 200 OK — item deleted regardless of ownership
- Impact: Any authenticated user can delete any other user's data
- Fix: Add ownership check before delete operation

## Scores
- Completeness: 4 — All features implemented
- Correctness: 2 — Login flow broken for empty input case
- UX: 2 — No validation feedback on primary form
- Security: 2 — Authorization bypass on delete endpoint
- Code Quality: 4 — Clean code, good conventions

## Result: FAIL
Correctness (2) and Security (2) are below threshold.
Generator must fix both issues before re-evaluation.

Why this is correct: Same findings, scored honestly against scale definitions, hard failure thresholds applied. Actionable fix descriptions included.

---

Anti-Patterns

These are common evaluator failure modes — watch for them:

---

Customization Guide

How to use presets

1. During evaluator setup, identify the artifact type 2. Select the matching preset from the dimension presets section 3. Present to the user for customization 4. Write the final criteria to .dev-workflow/evaluator-criteria.md (for workspace evaluation) or use directly in agent prompts (for ad-hoc validation) 5. The evaluator reads the per-workspace file instead of this default reference

Adding project-specific dimensions

Create a validation-criteria.md in your project's .dev-workflow/ directory:

# Project Validation Criteria

## Additional dimensions

- API Design: Check for consistent naming, proper status codes, error format
- Data Privacy: Verify PII handling, encryption, deletion cascade

## Project-specific hard failures

- Any endpoint missing Zod validation → Security FAIL
- Any database change missing migration → Completeness FAIL