Aep Validate

Validate

Run a generator/evaluator pattern against any artifact produced by the AEP workflow. The generator attempts to use the artifact (dry-run), the evaluator checks it against reality (codebase, constraints, downstream protocols), and the results are consolidated into fixes applied to the artifact itself.

"When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre."

— Anthropic, "Harness Design for Long-Running Application Development"

Why separate agents: The agent that produced an artifact cannot honestly evaluate it. The generator/evaluator separation is the single most impactful quality improvement in agentic workflows. This skill applies the gen/eval pattern to product artifacts.

Uses the gen/eval utility pattern. Read these reference files for the underlying framework:

• Scoring: .claude/skills/aep-gen-eval/references/scoring-framework.md
• Agent contracts: .claude/skills/aep-gen-eval/references/agent-contracts.md
• Eval protocol: .claude/skills/aep-gen-eval/references/eval-protocol.md
• Findings format: .claude/skills/aep-gen-eval/references/findings-format.md

Where this fits:

/aep-envision → /aep-map → /aep-model (UI-facing) → /aep-validate → /aep-dispatch → /aep-design → /aep-launch → /aep-build → /aep-wrap
                    ▲ you are here

Also usable after any phase:
  /aep-envision → /aep-validate   (validate product context)
  /aep-map      → /aep-validate   (validate architecture + stories)
  /aep-design   → /aep-validate   (validate specs before launch)
  /aep-build    → /aep-validate   (already built into Phase 5 — use that instead)

Session: Main, can be autonomous or interactive Input: Any AEP artifact (product/index.yaml + product-context.yaml in split mode, or product-context.yaml alone in v1 mode, or OpenSpec change, design doc, code) Output: The same artifact, with issues fixed. A validation report appended to changelog.

---

Before Starting

Identify what is being validated:

# Check for product context
ls product-context.yaml 2>/dev/null

# Check for OpenSpec changes
ls openspec/changes/ 2>/dev/null

# Check for design artifacts
ls .dev-workflow/ 2>/dev/null

If the user doesn't specify what to validate, auto-detect based on the most recently modified artifact.

File Resolution:

ls product/index.yaml 2>/dev/null && echo "SPLIT MODE" || echo "V1 MODE"

• Split mode: Validate both files. Check cross-file consistency.
• V1 mode: Validate product-context.yaml only.

---

Step 1: Determine Validation Mode

The skill operates in one of four modes based on the artifact type. Each mode configures which agents to spawn and what they check.

Mode A: Product Context Validation

When: After /aep-envision or /aep-map — validating product-context.yaml (and product/index.yaml in split mode)

Split-mode cross-file checks:

• stories[].layer values must exist in product/index.yaml product.layers[]
• stories[].activity values must exist in product/index.yaml product.activities[]
• calibration.plan[].dimensions[] must reference product/index.yaml product.quality_dimensions[]
• No opportunity or product section should exist in product-context.yaml when split mode is active
• product/index.yaml must have personas, capabilities, and product sections
• If object-model is a declared quality dimension: every UI-facing capability has a product/maps/<capability>/object-map.yaml; each stories[].object_model_refs entry points to an existing object-map path + object id; object names are consistent with architecture.domain_model and docs/glossary.md

Mode A runs two passes — product design quality first, then technical correctness:

Pass 1: Product Design Evaluation ("Are we building the right thing?")

Agents: Product Design Evaluator + Vision Alignment Checker

Read the Product Design Evaluator prompt from .claude/skills/aep-gen-eval/references/agent-contracts.md. Score using the story mapping dimensions from .claude/skills/aep-gen-eval/references/scoring-framework.md (Walking Skeleton Validity, Layer Ordering, Vision Alignment, INVEST Compliance).

Pass 1 hard failures:

• Walking Skeleton Validity < 3 — Layer 0 is not minimal enough
• Vision Alignment < 3 — Stories have drifted from the product vision
• INVEST Compliance < 3 — Stories are not actionable by an autonomous agent

Pass 1 activity checks (if product.activities exists):

• Activity Coverage: Every activity with layer_introduced: 0 has at least one Layer 0 story. An activity with no stories is a gap in the walking skeleton.
• Activity Consistency: Every story with a non-null activity references a valid activity id from product.activities.
• Narrative Coherence: Read activities left-to-right by order — they should form a coherent user narrative: "User [activity 1], then [activity 2], then..." If it doesn't flow, the backbone needs restructuring.
• Infrastructure Ratio: If more than 60% of Layer 0 stories have null activity, the decomposition may be too technical — consider reframing stories around user capabilities.

Pass 1 Object Map checks (if any product/maps/*/object-map.yaml exists):

• CTA Coverage: Every UI story's verbs map to a CTA on some object in its capability's object-map coverage. An uncovered UI story means a missing object or a hidden task-flow.
• Object Home: Every object in primary_objects has at least one screens[] entry (collection or detail). A primary object with no home screen is unreachable.
• Anchor Present: Each capability object-map declares a navigation.anchor_object.
• Task-Flow Justified: Every interaction_modes entry with mode: task_oriented has a non-empty reason (object-first is the default; deviations must be justified).
• Approval State: No UI-facing story is dispatch-ready while its capability object-map is status: draft or stale — it must be approved first (run /aep-model).
• Ref Resolution: Each stories[].object_model_refs entry (<path>#<object-id>) resolves — the path exists and <object-id> appears in that map's primary_objects/supporting_objects and in object-model.yaml objects[].
• Noun Coverage: Every noun foraged from product.activities maps to an object or is justified as implementation-only.

Pass 2: Technical Validation ("Can we build it correctly?")

Agents: Generator + Evaluator + Protocol Checker

Why two passes: Pass 1 catches product design problems (wrong stories, bad layering, vision drift). Pass 2 catches technical problems (missing fields, broken references, codebase mismatches). Both are required before dispatching to autonomous agents — the agents will faithfully build whatever you give them, right or wrong.

Mode B: Design Validation

When: After /aep-design — validating OpenSpec artifacts (proposal, design, specs, tasks) Agents: Generator + Evaluator

Mode C: Code Validation

When: After implementation — validating code changes Agents: Generator + Evaluator (same as /aep-build Phase 5)

Note: For code validation in a workspace, prefer /aep-build Phase 5 which has the full evaluator loop (spawned worktree-bound via executor.spawn_evaluator). Use this skill for code review on the integration branch or for lighter validation.

Mode D: Document Validation

When: Validating any structured document (architecture doc, RFC, migration plan) Agents: Generator + Evaluator

---

Step 2: Assemble Validation Context

For each agent, prepare a focused context package. Irrelevant context degrades evaluation quality.

Generator Context

The generator needs:

1. The artifact being validated — full content 2. The artifact's purpose — what downstream consumer will use it (e.g., "dispatch will read stories", "an implementer agent will follow these tasks") 3. Constraints — technical stack, project conventions, existing patterns

The generator does NOT need:

• The full codebase (that's the evaluator's job)
• History of how the artifact was created
• Other artifacts not directly consumed

Evaluator Context

The evaluator needs:

1. The artifact being validated — full content 2. Read access to the codebase — package.json files, existing schemas, config files, source code 3. The specific claims to verify — file paths, import statements, version numbers, API signatures

The evaluator does NOT need:

• Product vision or business context
• The generator's findings (agents work independently)

Protocol Checker Context (Mode A only)

The protocol checker needs:

1. The artifact being validated — specifically the stories section 2. The downstream protocol specification — e.g., the dispatch skill's requirements for story fields, scoring formula, DAG validation rules 3. The topology and layer gate definitions

---

Step 3: Spawn Agents

Launch all agents in parallel. Each agent works independently — they do not see each other's output.

Generator Agent Prompt Template

You are a GENERATOR agent performing a dry-run validation. Your job is to mentally
walk through using this artifact and identify gaps that would cause problems.

## The Artifact
{artifact_content}

## Your Task
For each item in this artifact, attempt to mentally execute it and report:
1. Can it be done? Yes/No
2. Missing details — anything vague or ambiguous that would cause guesswork
3. Dependency gaps — does this item have everything it needs?
4. Assumption mismatches — any implicit assumptions that could be wrong?

## Constraints
{technical_constraints}

## Output Format
For each item, output a brief assessment. Focus on PROBLEMS ONLY — don't describe
what's fine. At the end, produce a consolidated list of ALL changes needed.

Evaluator Agent Prompt Template

You are an EVALUATOR agent. Your job is to compare this artifact against the
ACTUAL state of the codebase and find mismatches. Read real files and verify claims.

## The Artifact
{artifact_content}

## What to Verify
{verification_checklist}

## Your Task
Read the actual files referenced in this artifact. For each claim, check:
1. Does the referenced file/function/type exist?
2. Does it have the signature/shape the artifact assumes?
3. Are version numbers and dependency versions correct?
4. Do import paths resolve correctly?

## Output Format
Report ALL mismatches between the artifact and reality. Be specific — include
file paths and line numbers. End with a severity-ranked list of required fixes.

Protocol Checker Agent Prompt Template (Mode A)

You are a PROTOCOL CHECKER. Your job is to verify this artifact is compatible
with the downstream protocol that will consume it.

## The Artifact
{artifact_content}

## The Downstream Protocol
{protocol_specification}

## Your Task
1. Are all required fields present on every item?
2. Is the dependency graph a valid DAG (no cycles, no missing references)?
3. Can the scoring/ranking algorithm be computed with the available fields?
4. Are there file-level conflicts between parallel items?
5. Can the downstream system create its required artifacts from this data?

## Output Format
Produce a compatibility report with specific fixes needed in the artifact.

---

Step 4: Consolidate Findings

After all agents return, consolidate their findings into a single action list.

Categorize by severity

Deduplicate

Multiple agents may find the same issue from different angles. Merge these into a single finding with the combined evidence.

Present to user

Show the consolidated findings with counts:

Validation complete: {N} blocking, {M} important, {K} minor issues found.

Blocking:
  1. [issue] — found by Generator + Evaluator
  2. [issue] — found by Protocol Checker

Important:
  3. [issue] — found by Generator
  ...

---

Step 5: Apply Fixes

Apply all blocking and important fixes to the artifact. Minor fixes are optional.

Rules for fixes:

• Only modify the artifact being validated — never create new files or modify other artifacts
• Preserve the artifact's existing structure and conventions
• Add a changelog entry recording what was validated and what changed
• If a fix requires a decision the agent can't make (architectural choice, business priority), mark it as an open_question with a default assumption

Changelog entry format

- date: <today>
  author: aep-validate
  summary: >
    Generator/evaluator validation. Found {N} blocking, {M} important, {K} minor issues.
    Fixed: [brief list of key fixes].

---

Step 6: Commit

# Resolve $BASE (integration branch) — see git-ref "Integration Branch" (override → develop → main)
BASE=$(git config --get aep.integration-branch 2>/dev/null || true)
[ -z "$BASE" ] && { git show-ref --verify --quiet refs/heads/develop \
  || git show-ref --verify --quiet refs/remotes/origin/develop; } && BASE=develop
BASE=${BASE:-main}

git pull --ff-only origin "$BASE"
git add <validated-files>
git commit -m "fix: validate {artifact-name} — {N} issues found and fixed"
git push origin "$BASE"

---

Validation Dimensions

When the agents evaluate, they should consider these dimensions (adapted from the evaluator criteria in /aep-build). Not all dimensions apply to every mode.

For Product Context (Mode A)

For Design Artifacts (Mode B)

For Code (Mode C)

See references/evaluator-criteria.md for the full 5-dimension scoring framework (Completeness, Correctness, UX Quality, Security, Code Quality).

For Documents (Mode D)

---

When NOT to Use This Skill

• During `/aep-build` Phase 5 — use the built-in evaluator loop instead (it has executor.spawn_evaluator, verification JSON, scoring framework)
• For subjective quality — this skill validates factual correctness and completeness, not aesthetic judgment
• For tiny changes — single-file edits or typo fixes don't need a 3-agent validation

---

Customization

Adding domain-specific checks

Create a validation-criteria.md file in your project's .dev-workflow/ directory to add project-specific validation checks. The agents will read this file if it exists.

# Project Validation Criteria

## Additional checks for Mode A (Product Context)

- All stories must have `business_value` field (required by our dispatch)
- Complexity must use S/M/L format (not small/medium/large)
- All file paths must be verified against the actual filesystem

## Additional checks for Mode B (Design)

- All API endpoints must include Zod validation schemas
- Database schema changes must include migration plan

Adjusting agent count

By default, Mode A spawns 3 agents, Modes B-D spawn 2. You can adjust:

• Lighter validation (1 agent): Use when the artifact is small or low-risk. The single agent combines generator + evaluator roles.
• Heavier validation (4+ agents): Add domain-specific agents for complex artifacts. Examples: a "security reviewer" for auth-related designs, a "performance reviewer" for data pipeline architectures.

---

Anti-Patterns

• Generator evaluating its own work — The agent that created an artifact must never be the one validating it. Use separate agent invocations.
• Evaluator without codebase access — An evaluator that can't read the actual code is just guessing. Always give the evaluator read access to relevant files.
• Applying fixes without presenting findings — Always show the user what was found before applying fixes. They may disagree with some findings or want to prioritize differently.
• Validating against stale state — Always read the current file/codebase state, not cached versions. Files may have changed since the artifact was created.
• Over-validating — Not every artifact needs 3-agent validation. Use judgment about the artifact's risk and complexity.

---

Next Step

After validation, proceed to the appropriate downstream skill:

Product context validated → /aep-dispatch
Design validated          → /aep-launch
Code validated            → create PR (or /aep-build Phase 9)
Document validated        → publish/share

Downstream Protocol Specifications

Reference document for the Protocol Checker agent. Defines the exact requirements that downstream consumers impose on product-context.yaml.

---

Dispatch Protocol Requirements

The /aep-dispatch skill reads stories from product-context.yaml and computes dispatch_score for ranking. Every story MUST have these fields for dispatch to function.

Required Story Fields

- id: string # kebab-case unique identifier
  title: string # short descriptive title
  module: string # package name (e.g., packages/db) or "integration"
  layer: integer # 0 = walking skeleton, 1+ = enrichment
  slice: integer # execution slice number (parallel batch)
  status: string # pending | ready | in_progress | review | done | blocked | failed
  business_value: string # critical | high | medium | low
  complexity: string # S | M | L
  description: string # what changes when complete
  acceptance_criteria: list # testable conditions (minimum 3 for well-specified stories)
  dependencies: list # story IDs this depends on (empty list if none)
  files_affected: list # file paths modified/created
  attempt_count: integer # starts at 0, incremented on retry
  failure_logs: list # empty list initially, populated on failure

Scoring Formula

dispatch_score = (business_value + unblock_potential + critical_path_urgency + reuse_leverage) / (complexity_cost + ambiguity_penalty + interface_risk)

Where:

• business_value (1-10): from story field if set, else derived from priority (critical=10, high=7, medium=4, low=1)
• unblock_potential (0-10): min(10, count_of_direct_dependents \* 2)
• critical_path_urgency (0-10): Stories on the longest dependency chain get 10
• reuse_leverage (0-10): min(10, modules_depending_on_output \* 3) for shared enablers
• complexity_cost (denominator): S=1, M=2, L=4
• ambiguity_penalty (0-5): +2 if <3 criteria, +1 each for missing interfaces/files/questions
• interface_risk (0-3): +1 per interface contract touched

DAG Validation Rules

1. Dependencies must form a directed acyclic graph (no cycles) 2. Every ID in a dependencies list must reference an existing story ID 3. Stories in the same slice should have no mutual dependencies 4. Integration stories (module: "integration") should depend on all stories they test

File Conflict Detection

Stories with overlapping files_affected must not be dispatched in parallel. Exceptions:

• Files documented as "append-only" (e.g., router index files) via conflict_note field
• Files in files_affected: [] (no files — cannot conflict)

Top-Level Fields

dispatch_epoch: integer # starts at 0, incremented each dispatch run

Layer Gate Fields

layer_gates:
  - layer: integer
    name: string
    status: string # pending | passed | failed
    description: string
    tests: list
    pass_criteria: string
    blocks: string

---

Design Protocol Requirements (/aep-design → /aep-launch)

OpenSpec changes created by /aep-dispatch and consumed by /aep-design must include:

Story Spec Completeness

For /aep-design to skip straight to /aep-launch (well-specified path):

• 3+ specific, testable acceptance criteria
• Interface obligations defined (if touching module boundaries)
• Files affected identified
• Complexity S or M

For /aep-design to refine first (ambiguous path):

• Fewer than 3 acceptance criteria
• Missing interface details
• Complexity L
• Open questions relevant to this story

Context Package Structure

openspec/changes/<story-id>/
├── proposal.md          # story description + why + business value
├── design.md            # module definition + interface contracts
├── specs/<module>.md    # acceptance criteria + interface obligations
├── tasks.md             # implementation tasks
└── .context/            # pre-assembled context
    ├── stable-prefix.md # shared product/architecture context
    ├── dependencies.md  # public APIs from completed dependencies
    └── retrieval.md     # what to explore at runtime

---

Build Protocol Requirements (/aep-build)

Feature Verification JSON

If the build agent uses verification tracking:

[
  {
    "task": "string — task description",
    "commit_sha": "string — git short SHA (8 chars)",
    "verification_steps": ["step 1", "step 2"],
    "passes": false,
    "evaluated_by": null,
    "round": null,
    "notes": null
  }
]

Rules:

• Generator MUST NOT modify verification_steps or passes
• Only evaluator or human updates these fields

Signal Protocol

Workspace agents communicate via .dev-workflow/signals/:

• status.json — phase progress, completion %, PR URL, cost
• eval-request.md — generator requests evaluation
• eval-response-<N>.md — evaluator returns findings

---

Topology Requirements

topology:
  roles:
    - name: string
      responsibility: string
      does_not: string
      input: string
      output: string
      context_window: list
      cost_budget: string

  handoff_contracts:
    - from: string
      to: string
      trigger: string
      payload: string
      validation: string

  routing:
    dispatch_policy: string
    concurrency_limit: integer # default 5
    conflict_detection: string
    retry_routing: string

---

Common Protocol Violations

These are the most frequently caught issues (ranked by frequency from real validations):

1. Missing `business_value` field — map skill doesn't produce it, dispatch requires it 2. `estimated_size` vs `complexity` — map uses small/medium/large, dispatch expects S/M/L 3. Missing `dispatch_epoch` — top-level field required for idempotent dispatch runs 4. Missing `attempt_count` and `failure_logs` — required for retry routing 5. Layer gate missing `status` field — dispatch checks this for layer advancement 6. Incomplete `files_affected` — missing package.json updates, file deletions, config files 7. DAG violations — stories referencing non-existent dependency IDs 8. Undeclared file conflicts — multiple stories in the same slice modifying the same file