Skill Creator

Name: Skill Creator
Author: luongnv89

luongnv89/asm

2 installs
784 repo stars
Updated August 5, 2026
luongnv89/asm

skill-creator is a Claude Code skill for ai & agent building.

About

Creates, improves, evaluates, and benchmarks Claude skills through a draft-test-eval-revise loop with quantitative evals. A developer uses it to author a new skill or iterate an existing one against test cases and description optimization.

Two paths: create from scratch or improve an existing skill via eval feedback
Description budget rules and negative-trigger clauses for accurate triggering

Skill Creator by the numbers

2 all-time installs (skills.sh)
+1 installs in the week ending Aug 2, 2026 (Skillselion tracking)
Ranked #609 of 782 Skill Development skills by installs in the Skillselion catalog
Data as of Aug 5, 2026 (Skillselion catalog sync)

npx skills add https://github.com/luongnv89/asm --skill skill-creator

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/luongnv89/asm/skill-creator.svg)](https://skillselion.com/skills/luongnv89/asm/skill-creator)

Installs	2
repo stars	★ 784
Last updated	August 5, 2026
Repository	luongnv89/asm ↗

How do I helps with ai & agent building tasks.?

Creates, improves, evaluates, and benchmarks Claude skills through a draft-test-eval-revise loop with test cases and evals.

Who is it for?

A solo builder working on ai & agent building tasks who needs structured help with skill creator.

Skip if: Teams with no ai & agent building needs, or anyone wanting a generic chat assistant without this specific workflow.

When should I use this skill?

When you need to helps with ai & agent building tasks., or when skill-creator is a claude code skill for ai & agent building.

What you get

Structured output aligned to skill-creator: skill-creator, AI & Agent Building.

Files

agents/
assets/
docs/
eval-viewer/
references/
scripts/

SKILL.mdMarkdownGitHub ↗

Skill Creator

A skill for creating new skills and iteratively improving them. The agent's context budget is the primary constraint, so this SKILL.md links out to focused reference files instead of inlining their content.

The core loop:

1. Decide what the skill should do and how it should do it 2. Write a draft 3. Run test prompts against claude-with-access-to-the-skill 4. Evaluate results with the user (qualitative review via eval-viewer/generate_review.py, plus quantitative evals) 5. Revise the skill based on feedback and benchmarks 6. Repeat until satisfied; expand the test set and try again at scale

Identify where the user is in this loop and jump in there. New skill from scratch → start at step 1. Existing draft → jump to step 3 or 4. User wants to vibe-iterate without formal evals → support that. After the skill stabilizes, optionally run the description improver to optimize triggering.

Two entry paths

The skill supports two distinct workflows. Identify which one the user is on before you do anything else — they don't share a starting step.

Path A — Create a new skill from scratch. The user wants to capture a workflow, codify a pattern, or build a new capability. Start at "Creating a skill" below (Capture Intent → Interview → Write SKILL.md → Test → Eval).
Path B — Improve an existing skill. The user points to a skill that already exists and wants it brought up to standard, fixed, optimized, or iterated based on eval feedback. Do not start with Capture Intent — the intent is already encoded in the existing SKILL.md. Start at "Improving an existing skill" below.

If the user's request is ambiguous ("can you look at this skill?"), assume Path B and ask them to confirm before interviewing them as if it were a new skill. Path B is also the one that fires when the user invokes /skill-creator while pointing at a skill directory or file.

Both paths share the mandatory rules below: Repo Sync Before Edits, Version Management, YAML Frontmatter Safety, and Frontmatter Audit on Review/Evaluation. Apply them in either path.

Step Completion Reports

After completing each major step, output a status report in this format:

◆ [Step Name] ([step N of M] — [context])
··································································
  [Check 1]:          √ pass
  [Check 2]:          √ pass (note if relevant)
  [Check 3]:          × fail — [reason]
  [Check 4]:          √ pass
  [Criteria]:         √ N/M met
  ____________________________
  Result:             PASS | FAIL | PARTIAL

Adapt the check names to match what the step actually validates. Use √ for pass, × for fail, and — to add brief context. The "Criteria" line summarizes how many acceptance criteria were met. The "Result" line gives the overall verdict.

Intent Capture phase checks: Goal defined, Triggers identified, Output format agreed

Skill Writing phase checks: SKILL.md written, README generated, Subagents designed

Testing phase checks: Evals created, Runs completed, Viewer launched

Iteration phase checks: Feedback incorporated, Benchmarks improved, Description optimized

Communicating with the user

Users span a wide range of technical familiarity. Match jargon to context cues: "evaluation" and "benchmark" are borderline-fine; "JSON" and "assertion" need clear cues that the user knows the term before you use it without explaining. Briefly define terms when in doubt.

---

Mandatory Rule for Repo-Mutating Skills

When creating or updating any skill that changes files in a git repository (code, docs, config, commits, publishing), include this rule in that skill's SKILL.md:

Add a "Repo Sync Before Edits (mandatory)" section near the top.
Require pulling latest remote branch before modifications:
branch="$(git rev-parse --abbrev-ref HEAD)"
git fetch origin
git pull --rebase origin "$branch"
If working tree is dirty: stash, sync, then pop.
If origin is missing or conflicts occur: stop and ask the user before continuing.

Do not ship repo-mutating skills without this pre-sync guardrail.

Frontmatter rules (mandatory)

Read references/frontmatter-rules.md for the full mandatory rules:

Version Management — set metadata.version: 1.0.0 on creation; bump patch/minor/major on every edit.
YAML Frontmatter Safety — quote any string value containing :, #, -, <, >, |, {, }, [, ], ,, &, *, ?, =, !, %, @, or ` ``.
Frontmatter Audit on Review/Evaluation — required-field check, name/dir match, allowed top-level keys, metadata.version, metadata.author, YAML safety, and consistency with docs/README.md. Run python scripts/quick_validate.py <skill-path> first; it catches mechanical issues without LLM reasoning.

These rules apply on every write. Always confirm them before saving.

Creating a skill

Capture Intent

Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user fills the gaps and confirms before proceeding.

1. What should this skill enable Claude to do? 2. When should this skill trigger? (what user phrases/contexts) 3. What's the expected output format? 4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't. Suggest the appropriate default based on the skill type, but let the user decide. 5. Should this skill use subagents? Read references/subagent-patterns.md for the full guide. Key signals:

Will the skill read many files or scan large codebases? → Explorer subagent
Can parts of the work run in parallel? → Parallel worker subagents
Does the skill need independent quality review? → Review loop with fresh subagents
Will the skill produce large artifacts that require focused reasoning? → Executor subagent

If any apply, design the skill with a main-agent-as-orchestrator architecture so subagents handle the heavy lifting and the main conversation context stays clean.

Interview and Research

Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until this part is ironed out. Check available MCPs — research in parallel via subagents if available, otherwise inline.

Write the SKILL.md

Based on the user interview, fill in:

name: 1-64 chars, lowercase letters/digits/hyphens, no consecutive hyphens, exactly matches parent directory. Enforced by scripts/quick_validate.py.
description: When to trigger and what it does. Primary triggering mechanism. Single line, no newlines. Claude tends to _undertrigger_ — make descriptions a little "pushy", with negative triggers.
effort (optional): low | medium | high | xhigh | max. Defaults to high.
metadata.version: Semver string (see frontmatter rules).
compatibility: Required tools or dependencies (rare).

Writing a good description: pushy + negative triggers

A description has two jobs: pull in the queries that _should_ trigger the skill, and push away the queries from adjacent domains that _shouldn't_. Most authors do the first part well and forget the second, producing false-positive triggers — a Tailwind skill running on a Vue project, a Python skill firing on shell-script questions.

The fix is a "Don't use for ..." clause. Name adjacent domains that share keywords or intent but are the wrong fit.

Positive only: Creates React components using Tailwind CSS.
With negatives: Creates React components using Tailwind CSS. Use whenever the user asks for a new React component, UI element, or styled layout. Don't use for Vue, Svelte, vanilla CSS, or plain HTML projects.

Write positive and negative halves as one continuous sentence or two back-to-back sentences — not a structured list. scripts/quick_validate.py warns (non-fatal) when the negative-trigger clause appears missing.

Description length budget

Three limits, in order of which one bites first:

1. 250 chars — Claude Code's /skills listing cap. Anything beyond is truncated tail-first, chopping the negative-trigger clause. This is the limit that actually shapes triggering behavior. 2. ~2% of context window (~16k chars total, ~109 chars overhead per skill) — the shared available_skills budget. When it overflows, extra skills become invisible to the agent. 3. 1024 chars — the API spec ceiling, hard error.

Rule: target ≤250 characters. Treat 1024 as a hard error, not a goal. Lead with verbs, drop hedge words ("helps", "allows you to"), collapse synonyms, keep the negative half to two or three adjacent domains.

Skill Writing Guide

Read references/writing-guide.md for the full guide. It covers:

Anatomy of a skill — directory layout, where agents/, references/, scripts/, assets/, docs/ go.
Progressive disclosure — three-level loading, the 500-line SKILL.md cap, when to split into references/.
Principle of Lack of Surprise — no malware, no misleading skills.
Writing patterns — imperative voice, output-format templates, examples patterns.
Bundled scripts and error messages — scripts must print descriptive errors before exiting so the agent can self-correct.
Step Completion Reports — every skill emits one after each major phase.
Writing style — explain _why_ in lieu of heavy MUSTs.
Generate README.md — docs/README.md only, with AI-skip notice; see references/readme-template.md.
Test Cases — 2-3 realistic prompts saved to evals/evals.json; see references/schemas.md.
Optional pre-eval LLM validation — see references/validation-prompts.md.

Running and evaluating test cases

Read references/eval-loop.md for the full 5-step sequence (spawn runs, draft assertions, capture timing, grade/aggregate/view, read feedback). It covers the with-skill + baseline subagent pattern, the eval_metadata.json and timing.json formats, the generate_review.py invocation, and reading feedback.json.

Do NOT use /skill-test or any other testing skill — the flow in references/eval-loop.md is the one this skill expects.

Improving an existing skill

This is Path B from the entry-paths block at the top. Two distinct subpaths — pick based on what the user is asking for.

Subpath B1 — Retrofit an existing skill to the standard

Use this when the user says "update this skill to match the standard," "fix this skill," "review and improve," or invokes /skill-creator on a published skill that hasn't been touched in a while. The goal is mechanical conformance, not behavioral redesign. Do not interview the user about purpose, triggers, or output format — those are encoded in the existing SKILL.md.

Sequence:

1. Read the existing SKILL.md and surrounding directory. Note current frontmatter, body length, references, scripts, version. Skim docs/README.md for human-facing claims. 2. Run python scripts/quick_validate.py <skill-path>. Validates allowed keys, name format, description length, missing negative trigger, broken YAML. 3. Run the Frontmatter Audit described in references/frontmatter-rules.md. Cover every checklist item, not just what quick_validate.py flagged. 4. Inspect the body against the standards in this skill:

SKILL.md under 500 lines (split to references/ if not).
Step Completion Reports section present.
"Repo Sync Before Edits" section if the skill mutates a git repo.
Bundled scripts print descriptive errors before exiting.
Progressive disclosure used appropriately; references one level deep.

5. Decide fix vs. review-only mode. If fixing, apply edits and bump `metadata.version` — patch for frontmatter-only fixes, minor for new sections, major for restructuring. If reviewing only, surface findings as before/after suggestions and don't silently edit. 6. Re-run quick_validate.py to confirm clean. Output a Step Completion Report with a Frontmatter valid check. 7. Optional: offer description optimization (see below). Don't run it automatically — it costs eval tokens.

This subpath does not require running evals. Skip to subpath B2 only if body changes are substantive enough that the user wants verification.

Subpath B2 — Iterate on a skill based on eval feedback

Use this when the user has eval results (or wants to run evals) and wants the skill revised based on what the evals show. The opening move is the eval loop, not interviewing.

1. If evals already exist, read the latest results and the user's feedback.json. If not, run them per "Running and evaluating test cases" above. 2. Read references/iteration.md for the five principles of revision (generalize, stay lean, explain the why, spot repeated work, consider subagents) and the iteration loop (apply → rerun → review → repeat). 3. Run the Frontmatter Audit alongside content revision — a polished body on top of broken frontmatter still fails validation. 4. Bump metadata.version per Version Management — minor for new capabilities or expanded triggers, patch for wording fixes. 5. Re-run evals into a new iteration-<N+1>/ directory and let the user compare.

references/iteration.md also documents the optional blind A/B comparison system.

Description Optimization

The description field is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.

Read references/description-optimization.md for the full 4-step flow: generate trigger eval queries, review with the user via the HTML template, run the optimization loop with run_loop.py, apply the best description.

Package and Present (only if `present_files` tool is available)

Check whether you have access to the present_files tool. If not, skip. If yes, package the skill and present the .skill file:

python -m scripts.package_skill <path/to/skill-folder>

After packaging, direct the user to the resulting .skill file path so they can install it.

Environment-specific notes

If you're on Claude.ai (no subagents) or in Cowork (subagents but no browser), some mechanics change. Read references/environment-modes.md for the adapted flow. The core loop (draft → test → review → improve) is the same everywhere — only execution mechanics shift.

---

Reference files

The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.

agents/grader.md — How to evaluate assertions against outputs
agents/comparator.md — How to do blind A/B comparison between two outputs
agents/analyzer.md — How to analyze why one version beat another

The references/ directory has additional documentation:

references/frontmatter-rules.md — Version Management, YAML Safety, and Frontmatter Audit (mandatory).
references/writing-guide.md — Anatomy, progressive disclosure, writing patterns, error messages, test cases.
references/schemas.md — JSON structures for evals.json, grading.json, etc.
references/subagent-patterns.md — When and how to design skills that use the Agent tool.
references/workflows.md — Workflow patterns for structuring skill instructions.
references/output-patterns.md — Output format and file-writing patterns.
references/validation-prompts.md — Optional 4-phase LLM validation pass for a draft skill.
references/eval-loop.md — Full 5-step eval run / grade / viewer flow.
references/iteration.md — Principles for improving a skill based on feedback; blind comparison.
references/description-optimization.md — 4-step description-tuning workflow.
references/environment-modes.md — Claude.ai and Cowork-specific adaptations.
references/readme-template.md — AI-skip notice, template, and rules for docs/README.md.

---

If you maintain a task list, include "Create evals JSON and run eval-viewer/generate_review.py so human can review test cases" — especially in Cowork, where it's easy to skip.

Post-hoc Analyzer Agent

Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.

Role

After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?

Inputs

You receive these parameters in your prompt:

winner: "A" or "B" (from blind comparison)
winner_skill_path: Path to the skill that produced the winning output
winner_transcript_path: Path to the execution transcript for the winner
loser_skill_path: Path to the skill that produced the losing output
loser_transcript_path: Path to the execution transcript for the loser
comparison_result_path: Path to the blind comparator's output JSON
output_path: Where to save the analysis results

Process

Step 1: Read Comparison Result

1. Read the blind comparator's output at comparison_result_path 2. Note the winning side (A or B), the reasoning, and any scores 3. Understand what the comparator valued in the winning output

Step 2: Read Both Skills

1. Read the winner skill's SKILL.md and key referenced files 2. Read the loser skill's SKILL.md and key referenced files 3. Identify structural differences:

Instructions clarity and specificity
Script/tool usage patterns
Example coverage
Edge case handling

Step 3: Read Both Transcripts

1. Read the winner's transcript 2. Read the loser's transcript 3. Compare execution patterns:

How closely did each follow their skill's instructions?
What tools were used differently?
Where did the loser diverge from optimal behavior?
Did either encounter errors or make recovery attempts?

Step 4: Analyze Instruction Following

For each transcript, evaluate:

Did the agent follow the skill's explicit instructions?
Did the agent use the skill's provided tools/scripts?
Were there missed opportunities to leverage skill content?
Did the agent add unnecessary steps not in the skill?

Score instruction following 1-10 and note specific issues.

Step 5: Identify Winner Strengths

Determine what made the winner better:

Clearer instructions that led to better behavior?
Better scripts/tools that produced better output?
More comprehensive examples that guided edge cases?
Better error handling guidance?

Be specific. Quote from skills/transcripts where relevant.

Step 6: Identify Loser Weaknesses

Determine what held the loser back:

Ambiguous instructions that led to suboptimal choices?
Missing tools/scripts that forced workarounds?
Gaps in edge case coverage?
Poor error handling that caused failures?

Step 7: Generate Improvement Suggestions

Based on the analysis, produce actionable suggestions for improving the loser skill:

Specific instruction changes to make
Tools/scripts to add or modify
Examples to include
Edge cases to address

Prioritize by impact. Focus on changes that would have changed the outcome.

Step 8: Write Analysis Results

Save structured analysis to {output_path}.

Output Format

Write a JSON file with this structure:

{
  "comparison_summary": {
    "winner": "A",
    "winner_skill": "path/to/winner/skill",
    "loser_skill": "path/to/loser/skill",
    "comparator_reasoning": "Brief summary of why comparator chose winner"
  },
  "winner_strengths": [
    "Clear step-by-step instructions for handling multi-page documents",
    "Included validation script that caught formatting errors",
    "Explicit guidance on fallback behavior when OCR fails"
  ],
  "loser_weaknesses": [
    "Vague instruction 'process the document appropriately' led to inconsistent behavior",
    "No script for validation, agent had to improvise and made errors",
    "No guidance on OCR failure, agent gave up instead of trying alternatives"
  ],
  "instruction_following": {
    "winner": {
      "score": 9,
      "issues": ["Minor: skipped optional logging step"]
    },
    "loser": {
      "score": 6,
      "issues": [
        "Did not use the skill's formatting template",
        "Invented own approach instead of following step 3",
        "Missed the 'always validate output' instruction"
      ]
    }
  },
  "improvement_suggestions": [
    {
      "priority": "high",
      "category": "instructions",
      "suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
      "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
    },
    {
      "priority": "high",
      "category": "tools",
      "suggestion": "Add validate_output.py script similar to winner skill's validation approach",
      "expected_impact": "Would catch formatting errors before final output"
    },
    {
      "priority": "medium",
      "category": "error_handling",
      "suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
      "expected_impact": "Would prevent early failure on difficult documents"
    }
  ],
  "transcript_insights": {
    "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
    "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
  }
}

Guidelines

Be specific: Quote from skills and transcripts, don't just say "instructions were unclear"
Be actionable: Suggestions should be concrete changes, not vague advice
Focus on skill improvements: The goal is to improve the losing skill, not critique the agent
Prioritize by impact: Which changes would most likely have changed the outcome?
Consider causation: Did the skill weakness actually cause the worse output, or is it incidental?
Stay objective: Analyze what happened, don't editorialize
Think about generalization: Would this improvement help on other evals too?

Categories for Suggestions

Use these categories to organize improvement suggestions:

Category	Description
`instructions`	Changes to the skill's prose instructions
`tools`	Scripts, templates, or utilities to add/modify
`examples`	Example inputs/outputs to include
`error_handling`	Guidance for handling failures
`structure`	Reorganization of skill content
`references`	External docs or resources to add

Priority Levels

high: Would likely change the outcome of this comparison
medium: Would improve quality but may not change win/loss
low: Nice to have, marginal improvement

---

Analyzing Benchmark Results

When analyzing benchmark results, the analyzer's purpose is to surface patterns and anomalies across multiple runs, not suggest skill improvements.

Role

Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.

Inputs

You receive these parameters in your prompt:

benchmark_data_path: Path to the in-progress benchmark.json with all run results
skill_path: Path to the skill being benchmarked
output_path: Where to save the notes (as JSON array of strings)

Process

Step 1: Read Benchmark Data

1. Read the benchmark.json containing all run results 2. Note the configurations tested (with_skill, without_skill) 3. Understand the run_summary aggregates already calculated

Step 2: Analyze Per-Assertion Patterns

For each expectation across all runs:

Does it always pass in both configurations? (may not differentiate skill value)
Does it always fail in both configurations? (may be broken or beyond capability)
Does it always pass with skill but fail without? (skill clearly adds value here)
Does it always fail with skill but pass without? (skill may be hurting)
Is it highly variable? (flaky expectation or non-deterministic behavior)

Step 3: Analyze Cross-Eval Patterns

Look for patterns across evals:

Are certain eval types consistently harder/easier?
Do some evals show high variance while others are stable?
Are there surprising results that contradict expectations?

Step 4: Analyze Metrics Patterns

Look at time_seconds, tokens, tool_calls:

Does the skill significantly increase execution time?
Is there high variance in resource usage?
Are there outlier runs that skew the aggregates?

Step 5: Generate Notes

Write freeform observations as a list of strings. Each note should:

State a specific observation
Be grounded in the data (not speculation)
Help the user understand something the aggregate metrics don't show

Examples:

"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"
"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky"
"Without-skill runs consistently fail on table extraction expectations (0% pass rate)"
"Skill adds 13s average execution time but improves pass rate by 50%"
"Token usage is 80% higher with skill, primarily due to script output parsing"
"All 3 without-skill runs for eval 1 produced empty output"

Step 6: Write Notes

Save notes to {output_path} as a JSON array of strings:

[
  "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
  "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",
  "Without-skill runs consistently fail on table extraction expectations",
  "Skill adds 13s average execution time but improves pass rate by 50%"
]

Guidelines

DO:

Report what you observe in the data
Be specific about which evals, expectations, or runs you're referring to
Note patterns that aggregate metrics would hide
Provide context that helps interpret the numbers

DO NOT:

Suggest improvements to the skill (that's for the improvement step, not benchmarking)
Make subjective quality judgments ("the output was good/bad")
Speculate about causes without evidence
Repeat information already in the run_summary aggregates

Blind Comparator Agent

Compare two outputs WITHOUT knowing which skill produced them.

Role

The Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.

Your judgment is based purely on output quality and task completion.

Inputs

You receive these parameters in your prompt:

output_a_path: Path to the first output file or directory
output_b_path: Path to the second output file or directory
eval_prompt: The original task/prompt that was executed
expectations: List of expectations to check (optional - may be empty)

Process

Step 1: Read Both Outputs

1. Examine output A (file or directory) 2. Examine output B (file or directory) 3. Note the type, structure, and content of each 4. If outputs are directories, examine all relevant files inside

Step 2: Understand the Task

1. Read the eval_prompt carefully 2. Identify what the task requires:

What should be produced?
What qualities matter (accuracy, completeness, format)?
What would distinguish a good output from a poor one?

Step 3: Generate Evaluation Rubric

Based on the task, generate a rubric with two dimensions:

Content Rubric (what the output contains):

Criterion	1 (Poor)	3 (Acceptable)	5 (Excellent)
Correctness	Major errors	Minor errors	Fully correct
Completeness	Missing key elements	Mostly complete	All elements present
Accuracy	Significant inaccuracies	Minor inaccuracies	Accurate throughout

Structure Rubric (how the output is organized):

Criterion	1 (Poor)	3 (Acceptable)	5 (Excellent)
Organization	Disorganized	Reasonably organized	Clear, logical structure
Formatting	Inconsistent/broken	Mostly consistent	Professional, polished
Usability	Difficult to use	Usable with effort	Easy to use

Adapt criteria to the specific task. For example:

PDF form → "Field alignment", "Text readability", "Data placement"
Document → "Section structure", "Heading hierarchy", "Paragraph flow"
Data output → "Schema correctness", "Data types", "Completeness"

Step 4: Evaluate Each Output Against the Rubric

For each output (A and B):

1. Score each criterion on the rubric (1-5 scale) 2. Calculate dimension totals: Content score, Structure score 3. Calculate overall score: Average of dimension scores, scaled to 1-10

Step 5: Check Assertions (if provided)

If expectations are provided:

1. Check each expectation against output A 2. Check each expectation against output B 3. Count pass rates for each output 4. Use expectation scores as secondary evidence (not the primary decision factor)

Step 6: Determine the Winner

Compare A and B based on (in priority order):

1. Primary: Overall rubric score (content + structure) 2. Secondary: Assertion pass rates (if applicable) 3. Tiebreaker: If truly equal, declare a TIE

Be decisive - ties should be rare. One output is usually better, even if marginally.

Step 7: Write Comparison Results

Save results to a JSON file at the path specified (or comparison.json if not specified).

Output Format

Write a JSON file with this structure:

{
  "winner": "A",
  "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
  "rubric": {
    "A": {
      "content": {
        "correctness": 5,
        "completeness": 5,
        "accuracy": 4
      },
      "structure": {
        "organization": 4,
        "formatting": 5,
        "usability": 4
      },
      "content_score": 4.7,
      "structure_score": 4.3,
      "overall_score": 9.0
    },
    "B": {
      "content": {
        "correctness": 3,
        "completeness": 2,
        "accuracy": 3
      },
      "structure": {
        "organization": 3,
        "formatting": 2,
        "usability": 3
      },
      "content_score": 2.7,
      "structure_score": 2.7,
      "overall_score": 5.4
    }
  },
  "output_quality": {
    "A": {
      "score": 9,
      "strengths": [
        "Complete solution",
        "Well-formatted",
        "All fields present"
      ],
      "weaknesses": ["Minor style inconsistency in header"]
    },
    "B": {
      "score": 5,
      "strengths": ["Readable output", "Correct basic structure"],
      "weaknesses": [
        "Missing date field",
        "Formatting inconsistencies",
        "Partial data extraction"
      ]
    }
  },
  "expectation_results": {
    "A": {
      "passed": 4,
      "total": 5,
      "pass_rate": 0.8,
      "details": [
        { "text": "Output includes name", "passed": true },
        { "text": "Output includes date", "passed": true },
        { "text": "Format is PDF", "passed": true },
        { "text": "Contains signature", "passed": false },
        { "text": "Readable text", "passed": true }
      ]
    },
    "B": {
      "passed": 3,
      "total": 5,
      "pass_rate": 0.6,
      "details": [
        { "text": "Output includes name", "passed": true },
        { "text": "Output includes date", "passed": false },
        { "text": "Format is PDF", "passed": true },
        { "text": "Contains signature", "passed": false },
        { "text": "Readable text", "passed": true }
      ]
    }
  }
}

If no expectations were provided, omit the expectation_results field entirely.

Field Descriptions

winner: "A", "B", or "TIE"
reasoning: Clear explanation of why the winner was chosen (or why it's a tie)
rubric: Structured rubric evaluation for each output
content: Scores for content criteria (correctness, completeness, accuracy)
structure: Scores for structure criteria (organization, formatting, usability)
content_score: Average of content criteria (1-5)
structure_score: Average of structure criteria (1-5)
overall_score: Combined score scaled to 1-10
output_quality: Summary quality assessment
score: 1-10 rating (should match rubric overall_score)
strengths: List of positive aspects
weaknesses: List of issues or shortcomings
expectation_results: (Only if expectations provided)
passed: Number of expectations that passed
total: Total number of expectations
pass_rate: Fraction passed (0.0 to 1.0)
details: Individual expectation results

Guidelines

Stay blind: DO NOT try to infer which skill produced which output. Judge purely on output quality.
Be specific: Cite specific examples when explaining strengths and weaknesses.
Be decisive: Choose a winner unless outputs are genuinely equivalent.
Output quality first: Assertion scores are secondary to overall task completion.
Be objective: Don't favor outputs based on style preferences; focus on correctness and completeness.
Explain your reasoning: The reasoning field should make it clear why you chose the winner.
Handle edge cases: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.

Grader Agent

Evaluate expectations against an execution transcript and outputs.

Role

The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.

You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.

Inputs

You receive these parameters in your prompt:

expectations: List of expectations to evaluate (strings)
transcript_path: Path to the execution transcript (markdown file)
outputs_dir: Directory containing output files from execution

Process

Step 1: Read the Transcript

1. Read the transcript file completely 2. Note the eval prompt, execution steps, and final result 3. Identify any issues or errors documented

Step 2: Examine Output Files

1. List files in outputs_dir 2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced. 3. Note contents, structure, and quality

Step 3: Evaluate Each Assertion

For each expectation:

1. Search for evidence in the transcript and outputs 2. Determine verdict:

PASS: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance
FAIL: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)

3. Cite the evidence: Quote the specific text or describe what you found

Step 4: Extract and Verify Claims

Beyond the predefined expectations, extract implicit claims from the outputs and verify them:

1. Extract claims from the transcript and outputs:

Factual statements ("The form has 12 fields")
Process claims ("Used pypdf to fill the form")
Quality claims ("All fields were filled correctly")

2. Verify each claim:

Factual claims: Can be checked against the outputs or external sources
Process claims: Can be verified from the transcript
Quality claims: Evaluate whether the claim is justified

3. Flag unverifiable claims: Note claims that cannot be verified with available information

This catches issues that predefined expectations might miss.

Step 5: Read User Notes

If {outputs_dir}/user_notes.md exists:

1. Read it and note any uncertainties or issues flagged by the executor 2. Include relevant concerns in the grading output 3. These may reveal problems even when expectations pass

Step 6: Critique the Evals

After grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.

Good suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion _discriminating_: it passes when the skill genuinely succeeds and fails when it doesn't.

Suggestions worth raising:

An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)
An important outcome you observed — good or bad — that no assertion covers at all
An assertion that can't actually be verified from the available outputs

Keep the bar high. The goal is to flag things the eval author would say "good catch" about, not to nitpick every assertion.

Step 7: Write Grading Results

Save results to {outputs_dir}/../grading.json (sibling to outputs_dir).

Grading Criteria

PASS when:

The transcript or outputs clearly demonstrate the expectation is true
Specific evidence can be cited
The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)

FAIL when:

No evidence found for the expectation
Evidence contradicts the expectation
The expectation cannot be verified from available information
The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete
The output appears to meet the assertion by coincidence rather than by actually doing the work

When uncertain: The burden of proof to pass is on the expectation.

Step 8: Read Executor Metrics and Timing

1. If {outputs_dir}/metrics.json exists, read it and include in grading output 2. If {outputs_dir}/../timing.json exists, read it and include timing data

Output Format

Write a JSON file with this structure:

{
  "expectations": [
    {
      "text": "The output includes the name 'John Smith'",
      "passed": true,
      "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
    },
    {
      "text": "The spreadsheet has a SUM formula in cell B10",
      "passed": false,
      "evidence": "No spreadsheet was created. The output was a text file."
    },
    {
      "text": "The assistant used the skill's OCR script",
      "passed": true,
      "evidence": "Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'"
    }
  ],
  "summary": {
    "passed": 2,
    "failed": 1,
    "total": 3,
    "pass_rate": 0.67
  },
  "execution_metrics": {
    "tool_calls": {
      "Read": 5,
      "Write": 2,
      "Bash": 8
    },
    "total_tool_calls": 15,
    "total_steps": 6,
    "errors_encountered": 0,
    "output_chars": 12450,
    "transcript_chars": 3200
  },
  "timing": {
    "executor_duration_seconds": 165.0,
    "grader_duration_seconds": 26.0,
    "total_duration_seconds": 191.0
  },
  "claims": [
    {
      "claim": "The form has 12 fillable fields",
      "type": "factual",
      "verified": true,
      "evidence": "Counted 12 fields in field_info.json"
    },
    {
      "claim": "All required fields were populated",
      "type": "quality",
      "verified": false,
      "evidence": "Reference section was left blank despite data being available"
    }
  ],
  "user_notes_summary": {
    "uncertainties": ["Used 2023 data, may be stale"],
    "needs_review": [],
    "workarounds": ["Fell back to text overlay for non-fillable fields"]
  },
  "eval_feedback": {
    "suggestions": [
      {
        "assertion": "The output includes the name 'John Smith'",
        "reason": "A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input"
      },
      {
        "reason": "No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught"
      }
    ],
    "overall": "Assertions check presence but not correctness. Consider adding content verification."
  }
}

Field Descriptions

expectations: Array of graded expectations
text: The original expectation text
passed: Boolean - true if expectation passes
evidence: Specific quote or description supporting the verdict
summary: Aggregate statistics
passed: Count of passed expectations
failed: Count of failed expectations
total: Total expectations evaluated
pass_rate: Fraction passed (0.0 to 1.0)
execution_metrics: Copied from executor's metrics.json (if available)
output_chars: Total character count of output files (proxy for tokens)
transcript_chars: Character count of transcript
timing: Wall clock timing from timing.json (if available)
executor_duration_seconds: Time spent in executor subagent
total_duration_seconds: Total elapsed time for the run
claims: Extracted and verified claims from the output
claim: The statement being verified
type: "factual", "process", or "quality"
verified: Boolean - whether the claim holds
evidence: Supporting or contradicting evidence
user_notes_summary: Issues flagged by the executor
uncertainties: Things the executor wasn't sure about
needs_review: Items requiring human attention
workarounds: Places where the skill didn't work as expected
eval_feedback: Improvement suggestions for the evals (only when warranted)
suggestions: List of concrete suggestions, each with a reason and optionally an assertion it relates to
overall: Brief assessment — can be "No suggestions, evals look solid" if nothing to flag

Guidelines

Be objective: Base verdicts on evidence, not assumptions
Be specific: Quote the exact text that supports your verdict
Be thorough: Check both transcript and output files
Be consistent: Apply the same standard to each expectation
Explain failures: Make it clear why evidence was insufficient
No partial credit: Each expectation is pass or fail, not partial

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Eval Set Review - __SKILL_NAME_PLACEHOLDER__</title>
    <link rel="preconnect" href="https://fonts.googleapis.com" />
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
    <link
      href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap"
      rel="stylesheet"
    />
    <style>
      * {
        box-sizing: border-box;
        margin: 0;
        padding: 0;
      }
      body {
        font-family: "Lora", Georgia, serif;
        background: #faf9f5;
        padding: 2rem;
        color: #141413;
      }
      h1 {
        font-family: "Poppins", sans-serif;
        margin-bottom: 0.5rem;
        font-size: 1.5rem;
      }
      .description {
        color: #b0aea5;
        margin-bottom: 1.5rem;
        font-style: italic;
        max-width: 900px;
      }
      .controls {
        margin-bottom: 1rem;
        display: flex;
        gap: 0.5rem;
      }
      .btn {
        font-family: "Poppins", sans-serif;
        padding: 0.5rem 1rem;
        border: none;
        border-radius: 6px;
        cursor: pointer;
        font-size: 0.875rem;
        font-weight: 500;
      }
      .btn-add {
        background: #6a9bcc;
        color: white;
      }
      .btn-add:hover {
        background: #5889b8;
      }
      .btn-export {
        background: #d97757;
        color: white;
      }
      .btn-export:hover {
        background: #c4613f;
      }
      table {
        width: 100%;
        max-width: 1100px;
        border-collapse: collapse;
        background: white;
        border-radius: 6px;
        overflow: hidden;
        box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
      }
      th {
        font-family: "Poppins", sans-serif;
        background: #141413;
        color: #faf9f5;
        padding: 0.75rem 1rem;
        text-align: left;
        font-size: 0.875rem;
      }
      td {
        padding: 0.75rem 1rem;
        border-bottom: 1px solid #e8e6dc;
        vertical-align: top;
      }
      tr:nth-child(even) td {
        background: #faf9f5;
      }
      tr:hover td {
        background: #f3f1ea;
      }
      .section-header td {
        background: #e8e6dc;
        font-family: "Poppins", sans-serif;
        font-weight: 500;
        font-size: 0.8rem;
        color: #141413;
        text-transform: uppercase;
        letter-spacing: 0.05em;
      }
      .query-input {
        width: 100%;
        padding: 0.4rem;
        border: 1px solid #e8e6dc;
        border-radius: 4px;
        font-size: 0.875rem;
        font-family: "Lora", Georgia, serif;
        resize: vertical;
        min-height: 60px;
      }
      .query-input:focus {
        outline: none;
        border-color: #d97757;
        box-shadow: 0 0 0 2px rgba(217, 119, 87, 0.15);
      }
      .toggle {
        position: relative;
        display: inline-block;
        width: 44px;
        height: 24px;
      }
      .toggle input {
        opacity: 0;
        width: 0;
        height: 0;
      }
      .toggle .slider {
        position: absolute;
        inset: 0;
        background: #b0aea5;
        border-radius: 24px;
        cursor: pointer;
        transition: 0.2s;
      }
      .toggle .slider::before {
        content: "";
        position: absolute;
        width: 18px;
        height: 18px;
        left: 3px;
        bottom: 3px;
        background: white;
        border-radius: 50%;
        transition: 0.2s;
      }
      .toggle input:checked + .slider {
        background: #d97757;
      }
      .toggle input:checked + .slider::before {
        transform: translateX(20px);
      }
      .btn-delete {
        background: #c44;
        color: white;
        padding: 0.3rem 0.6rem;
        border: none;
        border-radius: 4px;
        cursor: pointer;
        font-size: 0.75rem;
        font-family: "Poppins", sans-serif;
      }
      .btn-delete:hover {
        background: #a33;
      }
      .summary {
        margin-top: 1rem;
        color: #b0aea5;
        font-size: 0.875rem;
      }
    </style>
  </head>
  <body>
    <h1>
      Eval Set Review: <span id="skill-name">__SKILL_NAME_PLACEHOLDER__</span>
    </h1>
    <p class="description">
      Current description:
      <span id="skill-desc">__SKILL_DESCRIPTION_PLACEHOLDER__</span>
    </p>

    <div class="controls">
      <button class="btn btn-add" onclick="addRow()">+ Add Query</button>
      <button class="btn btn-export" onclick="exportEvalSet()">
        Export Eval Set
      </button>
    </div>

    <table>
      <thead>
        <tr>
          <th style="width: 65%">Query</th>
          <th style="width: 18%">Should Trigger</th>
          <th style="width: 10%">Actions</th>
        </tr>
      </thead>
      <tbody id="eval-body"></tbody>
    </table>

    <p class="summary" id="summary"></p>

    <script>
      const EVAL_DATA = __EVAL_DATA_PLACEHOLDER__;

      let evalItems = [...EVAL_DATA];

      function render() {
        const tbody = document.getElementById("eval-body");
        tbody.innerHTML = "";

        // Sort: should-trigger first, then should-not-trigger
        const sorted = evalItems
          .map((item, origIdx) => ({ ...item, origIdx }))
          .sort(
            (a, b) => (b.should_trigger ? 1 : 0) - (a.should_trigger ? 1 : 0),
          );

        let lastGroup = null;
        sorted.forEach((item) => {
          const group = item.should_trigger ? "trigger" : "no-trigger";
          if (group !== lastGroup) {
            const headerRow = document.createElement("tr");
            headerRow.className = "section-header";
            headerRow.innerHTML = `<td colspan="3">${item.should_trigger ? "Should Trigger" : "Should NOT Trigger"}</td>`;
            tbody.appendChild(headerRow);
            lastGroup = group;
          }

          const idx = item.origIdx;
          const tr = document.createElement("tr");
          tr.innerHTML = `
          <td><textarea class="query-input" onchange="updateQuery(${idx}, this.value)">${escapeHtml(item.query)}</textarea></td>
          <td>
            <label class="toggle">
              <input type="checkbox" ${item.should_trigger ? "checked" : ""} onchange="updateTrigger(${idx}, this.checked)">
              <span class="slider"></span>
            </label>
            <span style="margin-left:8px;font-size:0.8rem;color:#b0aea5">${item.should_trigger ? "Yes" : "No"}</span>
          </td>
          <td><button class="btn-delete" onclick="deleteRow(${idx})">Delete</button></td>
        `;
          tbody.appendChild(tr);
        });
        updateSummary();
      }

      function escapeHtml(text) {
        const div = document.createElement("div");
        div.textContent = text;
        return div.innerHTML;
      }

      function updateQuery(idx, value) {
        evalItems[idx].query = value;
        updateSummary();
      }
      function updateTrigger(idx, value) {
        evalItems[idx].should_trigger = value;
        render();
      }
      function deleteRow(idx) {
        evalItems.splice(idx, 1);
        render();
      }

      function addRow() {
        evalItems.push({ query: "", should_trigger: true });
        render();
        const inputs = document.querySelectorAll(".query-input");
        inputs[inputs.length - 1].focus();
      }

      function updateSummary() {
        const trigger = evalItems.filter((i) => i.should_trigger).length;
        const noTrigger = evalItems.filter((i) => !i.should_trigger).length;
        document.getElementById("summary").textContent =
          `${evalItems.length} queries total: ${trigger} should trigger, ${noTrigger} should not trigger`;
      }

      function exportEvalSet() {
        const valid = evalItems.filter((i) => i.query.trim() !== "");
        const data = valid.map((i) => ({
          query: i.query.trim(),
          should_trigger: i.should_trigger,
        }));
        const blob = new Blob([JSON.stringify(data, null, 2)], {
          type: "application/json",
        });
        const url = URL.createObjectURL(blob);
        const a = document.createElement("a");
        a.href = url;
        a.download = "eval_set.json";
        document.body.appendChild(a);
        a.click();
        document.body.removeChild(a);
        URL.revokeObjectURL(url);
      }

      render();
    </script>
  </body>
</html>

Skill Creator

Create, evaluate, benchmark, and iteratively improve agent skills.

Highlights

Iterative skill loop: draft, test prompts, evaluate, refine
Subagent architecture guidance: design skills that delegate heavy work to subagents, keeping the main agent lean
Quantitative + qualitative eval workflow with baseline comparison
Benchmark aggregation, variance analysis, and report tooling
Description optimization flow to improve triggering accuracy
Dedicated eval viewer and grading agents for structured review

When to Use

Say this...	Skill will...
"Create a skill for X"	Interview you, draft SKILL.md + README.md, run test cases
"Improve this skill"	Run evals, collect feedback, suggest subagent restructuring, iterate
"Run evals for my skill"	Execute test prompts, grade results, show benchmark
"Optimize skill triggering"	Generate trigger eval queries, run optimization loop
"This skill is too slow / bloated"	Analyze for subagent refactoring opportunities

How It Works

graph TD
    A["Capture Intent & Interview"] --> B["Assess Subagent Architecture"]
    B --> C["Draft SKILL.md + agents/ + README.md"]
    C --> D["Run Test Cases & Baselines"]
    D --> E["Evaluate: Viewer + Benchmarks"]
    E --> F["Iterate & Refactor"]
    F --> D
    F --> G["Optimize Description & Package"]
    style A fill:#4CAF50,color:#fff
    style G fill:#2196F3,color:#fff

Installation

Install via npx (Vercel):

npx skills add https://github.com/luongnv89/skills --skill skill-creator

Or via agent-skill-manager (asm):

asm install github:luongnv89/skills:skills/skill-creator

Usage

/skill-creator

Resources

Path	Description
`scripts/`	Eval loop, benchmarking, packaging, validation utilities
`references/`	Evals schema, subagent patterns, workflow patterns
`eval-viewer/`	Generate/view review pages for eval results
`agents/`	Analyzer, comparator, and grader agent prompts
`assets/`	Viewer template assets

Output

Produces complete skill packages (SKILL.md + docs/README.md + agents/), eval results with benchmark reports, subagent restructuring recommendations, and optimized skill descriptions for accurate triggering.

#!/usr/bin/env python3
"""Generate and serve a review page for eval results.

Reads the workspace directory, discovers runs (directories with outputs/),
embeds all output data into a self-contained HTML page, and serves it via
a tiny HTTP server. Feedback auto-saves to feedback.json in the workspace.

Usage:
    python generate_review.py <workspace-path> [--port PORT] [--skill-name NAME]
    python generate_review.py <workspace-path> --previous-feedback /path/to/old/feedback.json

No dependencies beyond the Python stdlib are required.
"""

import argparse
import base64
import json
import mimetypes
import os
import re
import signal
import subprocess
import sys
import time
import webbrowser
from functools import partial
from http.server import HTTPServer, BaseHTTPRequestHandler
from pathlib import Path

# Files to exclude from output listings
METADATA_FILES = {"transcript.md", "user_notes.md", "metrics.json"}

# Extensions we render as inline text
TEXT_EXTENSIONS = {
    ".txt", ".md", ".json", ".csv", ".py", ".js", ".ts", ".tsx", ".jsx",
    ".yaml", ".yml", ".xml", ".html", ".css", ".sh", ".rb", ".go", ".rs",
    ".java", ".c", ".cpp", ".h", ".hpp", ".sql", ".r", ".toml",
}

# Extensions we render as inline images
IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"}

# MIME type overrides for common types
MIME_OVERRIDES = {
    ".svg": "image/svg+xml",
    ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
    ".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
}


def get_mime_type(path: Path) -> str:
    ext = path.suffix.lower()
    if ext in MIME_OVERRIDES:
        return MIME_OVERRIDES[ext]
    mime, _ = mimetypes.guess_type(str(path))
    return mime or "application/octet-stream"


def find_runs(workspace: Path) -> list[dict]:
    """Recursively find directories that contain an outputs/ subdirectory."""
    runs: list[dict] = []
    _find_runs_recursive(workspace, workspace, runs)
    runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"]))
    return runs


def _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None:
    if not current.is_dir():
        return

    outputs_dir = current / "outputs"
    if outputs_dir.is_dir():
        run = build_run(root, current)
        if run:
            runs.append(run)
        return

    skip = {"node_modules", ".git", "__pycache__", "skill", "inputs"}
    for child in sorted(current.iterdir()):
        if child.is_dir() and child.name not in skip:
            _find_runs_recursive(root, child, runs)


def build_run(root: Path, run_dir: Path) -> dict | None:
    """Build a run dict with prompt, outputs, and grading data."""
    prompt = ""
    eval_id = None

    # Try eval_metadata.json
    for candidate in [run_dir / "eval_metadata.json", run_dir.parent / "eval_metadata.json"]:
        if candidate.exists():
            try:
                metadata = json.loads(candidate.read_text())
                prompt = metadata.get("prompt", "")
                eval_id = metadata.get("eval_id")
            except (json.JSONDecodeError, OSError):
                pass
            if prompt:
                break

    # Fall back to transcript.md
    if not prompt:
        for candidate in [run_dir / "transcript.md", run_dir / "outputs" / "transcript.md"]:
            if candidate.exists():
                try:
                    text = candidate.read_text()
                    match = re.search(r"## Eval Prompt\n\n([\s\S]*?)(?=\n##|$)", text)
                    if match:
                        prompt = match.group(1).strip()
                except OSError:
                    pass
                if prompt:
                    break

    if not prompt:
        prompt = "(No prompt found)"

    run_id = str(run_dir.relative_to(root)).replace("/", "-").replace("\\", "-")

    # Collect output files
    outputs_dir = run_dir / "outputs"
    output_files: list[dict] = []
    if outputs_dir.is_dir():
        for f in sorted(outputs_dir.iterdir()):
            if f.is_file() and f.name not in METADATA_FILES:
                output_files.append(embed_file(f))

    # Load grading if present
    grading = None
    for candidate in [run_dir / "grading.json", run_dir.parent / "grading.json"]:
        if candidate.exists():
            try:
                grading = json.loads(candidate.read_text())
            except (json.JSONDecodeError, OSError):
                pass
            if grading:
                break

    return {
        "id": run_id,
        "prompt": prompt,
        "eval_id": eval_id,
        "outputs": output_files,
        "grading": grading,
    }


def embed_file(path: Path) -> dict:
    """Read a file and return an embedded representation."""
    ext = path.suffix.lower()
    mime = get_mime_type(path)

    if ext in TEXT_EXTENSIONS:
        try:
            content = path.read_text(errors="replace")
        except OSError:
            content = "(Error reading file)"
        return {
            "name": path.name,
            "type": "text",
            "content": content,
        }
    elif ext in IMAGE_EXTENSIONS:
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "image",
            "mime": mime,
            "data_uri": f"data:{mime};base64,{b64}",
        }
    elif ext == ".pdf":
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "pdf",
            "data_uri": f"data:{mime};base64,{b64}",
        }
    elif ext == ".xlsx":
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "xlsx",
            "data_b64": b64,
        }
    else:
        # Binary / unknown — base64 download link
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "binary",
            "mime": mime,
            "data_uri": f"data:{mime};base64,{b64}",
        }


def load_previous_iteration(workspace: Path) -> dict[str, dict]:
    """Load previous iteration's feedback and outputs.

    Returns a map of run_id -> {"feedback": str, "outputs": list[dict]}.
    """
    result: dict[str, dict] = {}

    # Load feedback
    feedback_map: dict[str, str] = {}
    feedback_path = workspace / "feedback.json"
    if feedback_path.exists():
        try:
            data = json.loads(feedback_path.read_text())
            feedback_map = {
                r["run_id"]: r["feedback"]
                for r in data.get("reviews", [])
                if r.get("feedback", "").strip()
            }
        except (json.JSONDecodeError, OSError, KeyError):
            pass

    # Load runs (to get outputs)
    prev_runs = find_runs(workspace)
    for run in prev_runs:
        result[run["id"]] = {
            "feedback": feedback_map.get(run["id"], ""),
            "outputs": run.get("outputs", []),
        }

    # Also add feedback for run_ids that had feedback but no matching run
    for run_id, fb in feedback_map.items():
        if run_id not in result:
            result[run_id] = {"feedback": fb, "outputs": []}

    return result


def generate_html(
    runs: list[dict],
    skill_name: str,
    previous: dict[str, dict] | None = None,
    benchmark: dict | None = None,
) -> str:
    """Generate the complete standalone HTML page with embedded data."""
    template_path = Path(__file__).parent / "viewer.html"
    template = template_path.read_text()

    # Build previous_feedback and previous_outputs maps for the template
    previous_feedback: dict[str, str] = {}
    previous_outputs: dict[str, list[dict]] = {}
    if previous:
        for run_id, data in previous.items():
            if data.get("feedback"):
                previous_feedback[run_id] = data["feedback"]
            if data.get("outputs"):
                previous_outputs[run_id] = data["outputs"]

    embedded = {
        "skill_name": skill_name,
        "runs": runs,
        "previous_feedback": previous_feedback,
        "previous_outputs": previous_outputs,
    }
    if benchmark:
        embedded["benchmark"] = benchmark

    data_json = json.dumps(embedded)

    return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};")


# ---------------------------------------------------------------------------
# HTTP server (stdlib only, zero dependencies)
# ---------------------------------------------------------------------------

def _kill_port(port: int) -> None:
    """Kill any process listening on the given port."""
    try:
        result = subprocess.run(
            ["lsof", "-ti", f":{port}"],
            capture_output=True, text=True, timeout=5,
        )
        for pid_str in result.stdout.strip().split("\n"):
            if pid_str.strip():
                try:
                    os.kill(int(pid_str.strip()), signal.SIGTERM)
                except (ProcessLookupError, ValueError):
                    pass
        if result.stdout.strip():
            time.sleep(0.5)
    except subprocess.TimeoutExpired:
        pass
    except FileNotFoundError:
        print("Note: lsof not found, cannot check if port is in use", file=sys.stderr)

class ReviewHandler(BaseHTTPRequestHandler):
    """Serves the review HTML and handles feedback saves.

    Regenerates the HTML on each page load so that refreshing the browser
    picks up new eval outputs without restarting the server.
    """

    def __init__(
        self,
        workspace: Path,
        skill_name: str,
        feedback_path: Path,
        previous: dict[str, dict],
        benchmark_path: Path | None,
        *args,
        **kwargs,
    ):
        self.workspace = workspace
        self.skill_name = skill_name
        self.feedback_path = feedback_path
        self.previous = previous
        self.benchmark_path = benchmark_path
        super().__init__(*args, **kwargs)

    def do_GET(self) -> None:
        if self.path == "/" or self.path == "/index.html":
            # Regenerate HTML on each request (re-scans workspace for new outputs)
            runs = find_runs(self.workspace)
            benchmark = None
            if self.benchmark_path and self.benchmark_path.exists():
                try:
                    benchmark = json.loads(self.benchmark_path.read_text())
                except (json.JSONDecodeError, OSError):
                    pass
            html = generate_html(runs, self.skill_name, self.previous, benchmark)
            content = html.encode("utf-8")
            self.send_response(200)
            self.send_header("Content-Type", "text/html; charset=utf-8")
            self.send_header("Content-Length", str(len(content)))
            self.end_headers()
            self.wfile.write(content)
        elif self.path == "/api/feedback":
            data = b"{}"
            if self.feedback_path.exists():
                data = self.feedback_path.read_bytes()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(data)))
            self.end_headers()
            self.wfile.write(data)
        else:
            self.send_error(404)

    def do_POST(self) -> None:
        if self.path == "/api/feedback":
            length = int(self.headers.get("Content-Length", 0))
            body = self.rfile.read(length)
            try:
                data = json.loads(body)
                if not isinstance(data, dict) or "reviews" not in data:
                    raise ValueError("Expected JSON object with 'reviews' key")
                self.feedback_path.write_text(json.dumps(data, indent=2) + "\n")
                resp = b'{"ok":true}'
                self.send_response(200)
            except (json.JSONDecodeError, OSError, ValueError) as e:
                resp = json.dumps({"error": str(e)}).encode()
                self.send_response(500)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(resp)))
            self.end_headers()
            self.wfile.write(resp)
        else:
            self.send_error(404)

    def log_message(self, format: str, *args: object) -> None:
        # Suppress request logging to keep terminal clean
        pass


def main() -> None:
    parser = argparse.ArgumentParser(description="Generate and serve eval review")
    parser.add_argument("workspace", type=Path, help="Path to workspace directory")
    parser.add_argument("--port", "-p", type=int, default=3117, help="Server port (default: 3117)")
    parser.add_argument("--skill-name", "-n", type=str, default=None, help="Skill name for header")
    parser.add_argument(
        "--previous-workspace", type=Path, default=None,
        help="Path to previous iteration's workspace (shows old outputs and feedback as context)",
    )
    parser.add_argument(
        "--benchmark", type=Path, default=None,
        help="Path to benchmark.json to show in the Benchmark tab",
    )
    parser.add_argument(
        "--static", "-s", type=Path, default=None,
        help="Write standalone HTML to this path instead of starting a server",
    )
    args = parser.parse_args()

    workspace = args.workspace.resolve()
    if not workspace.is_dir():
        print(f"Error: {workspace} is not a directory", file=sys.stderr)
        sys.exit(1)

    runs = find_runs(workspace)
    if not runs:
        print(f"No runs found in {workspace}", file=sys.stderr)
        sys.exit(1)

    skill_name = args.skill_name or workspace.name.replace("-workspace", "")
    feedback_path = workspace / "feedback.json"

    previous: dict[str, dict] = {}
    if args.previous_workspace:
        previous = load_previous_iteration(args.previous_workspace.resolve())

    benchmark_path = args.benchmark.resolve() if args.benchmark else None
    benchmark = None
    if benchmark_path and benchmark_path.exists():
        try:
            benchmark = json.loads(benchmark_path.read_text())
        except (json.JSONDecodeError, OSError):
            pass

    if args.static:
        html = generate_html(runs, skill_name, previous, benchmark)
        args.static.parent.mkdir(parents=True, exist_ok=True)
        args.static.write_text(html)
        print(f"\n  Static viewer written to: {args.static}\n")
        sys.exit(0)

    # Kill any existing process on the target port
    port = args.port
    _kill_port(port)
    handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path)
    try:
        server = HTTPServer(("127.0.0.1", port), handler)
    except OSError:
        # Port still in use after kill attempt — find a free one
        server = HTTPServer(("127.0.0.1", 0), handler)
        port = server.server_address[1]

    url = f"http://localhost:{port}"
    print(f"\n  Eval Viewer")
    print(f"  ─────────────────────────────────")
    print(f"  URL:       {url}")
    print(f"  Workspace: {workspace}")
    print(f"  Feedback:  {feedback_path}")
    if previous:
        print(f"  Previous:  {args.previous_workspace} ({len(previous)} runs)")
    if benchmark_path:
        print(f"  Benchmark: {benchmark_path}")
    print(f"\n  Press Ctrl+C to stop.\n")

    webbrowser.open(url)

    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nStopped.")
        server.server_close()


if __name__ == "__main__":
    main()

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Eval Review</title>
    <link rel="preconnect" href="https://fonts.googleapis.com" />
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
    <link
      href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap"
      rel="stylesheet"
    />
    <script
      src="https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js"
      integrity="sha384-EnyY0/GSHQGSxSgMwaIPzSESbqoOLSexfnSMN2AP+39Ckmn92stwABZynq1JyzdT"
      crossorigin="anonymous"
    ></script>
    <style>
      :root {
        --bg: #faf9f5;
        --surface: #ffffff;
        --border: #e8e6dc;
        --text: #141413;
        --text-muted: #b0aea5;
        --accent: #d97757;
        --accent-hover: #c4613f;
        --green: #788c5d;
        --green-bg: #eef2e8;
        --red: #c44;
        --red-bg: #fceaea;
        --header-bg: #141413;
        --header-text: #faf9f5;
        --radius: 6px;
      }

      * {
        box-sizing: border-box;
        margin: 0;
        padding: 0;
      }

      body {
        font-family: "Lora", Georgia, serif;
        background: var(--bg);
        color: var(--text);
        height: 100vh;
        display: flex;
        flex-direction: column;
      }

      /* ---- Header ---- */
      .header {
        background: var(--header-bg);
        color: var(--header-text);
        padding: 1rem 2rem;
        display: flex;
        justify-content: space-between;
        align-items: center;
        flex-shrink: 0;
      }
      .header h1 {
        font-family: "Poppins", sans-serif;
        font-size: 1.25rem;
        font-weight: 600;
      }
      .header .instructions {
        font-size: 0.8rem;
        opacity: 0.7;
        margin-top: 0.25rem;
      }
      .header .progress {
        font-size: 0.875rem;
        opacity: 0.8;
        text-align: right;
      }

      /* ---- Main content ---- */
      .main {
        flex: 1;
        overflow-y: auto;
        padding: 1.5rem 2rem;
        display: flex;
        flex-direction: column;
        gap: 1.25rem;
      }

      /* ---- Sections ---- */
      .section {
        background: var(--surface);
        border: 1px solid var(--border);
        border-radius: var(--radius);
        flex-shrink: 0;
      }
      .section-header {
        font-family: "Poppins", sans-serif;
        padding: 0.75rem 1rem;
        font-size: 0.75rem;
        font-weight: 500;
        text-transform: uppercase;
        letter-spacing: 0.05em;
        color: var(--text-muted);
        border-bottom: 1px solid var(--border);
        background: var(--bg);
      }
      .section-body {
        padding: 1rem;
      }

      /* ---- Config badge ---- */
      .config-badge {
        display: inline-block;
        padding: 0.2rem 0.625rem;
        border-radius: 9999px;
        font-family: "Poppins", sans-serif;
        font-size: 0.6875rem;
        font-weight: 600;
        text-transform: uppercase;
        letter-spacing: 0.03em;
        margin-left: 0.75rem;
        vertical-align: middle;
      }
      .config-badge.config-primary {
        background: rgba(33, 150, 243, 0.12);
        color: #1976d2;
      }
      .config-badge.config-baseline {
        background: rgba(255, 193, 7, 0.15);
        color: #f57f17;
      }

      /* ---- Prompt ---- */
      .prompt-text {
        white-space: pre-wrap;
        font-size: 0.9375rem;
        line-height: 1.6;
      }

      /* ---- Outputs ---- */
      .output-file {
        border: 1px solid var(--border);
        border-radius: var(--radius);
        overflow: hidden;
      }
      .output-file + .output-file {
        margin-top: 1rem;
      }
      .output-file-header {
        padding: 0.5rem 0.75rem;
        font-size: 0.8rem;
        font-weight: 600;
        color: var(--text-muted);
        background: var(--bg);
        border-bottom: 1px solid var(--border);
        font-family:
          "SF Mono", SFMono-Regular, Consolas, "Liberation Mono", Menlo,
          monospace;
        display: flex;
        justify-content: space-between;
        align-items: center;
      }
      .output-file-header .dl-btn {
        font-size: 0.7rem;
        color: var(--accent);
        text-decoration: none;
        cursor: pointer;
        font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
        font-weight: 500;
        opacity: 0.8;
      }
      .output-file-header .dl-btn:hover {
        opacity: 1;
        text-decoration: underline;
      }
      .output-file-content {
        padding: 0.75rem;
        overflow-x: auto;
      }
      .output-file-content pre {
        font-size: 0.8125rem;
        line-height: 1.5;
        white-space: pre-wrap;
        word-break: break-word;
        font-family:
          "SF Mono", SFMono-Regular, Consolas, "Liberation Mono", Menlo,
          monospace;
      }
      .output-file-content img {
        max-width: 100%;
        height: auto;
        border-radius: 4px;
      }
      .output-file-content iframe {
        width: 100%;
        height: 600px;
        border: none;
      }
      .output-file-content table {
        border-collapse: collapse;
        font-size: 0.8125rem;
        width: 100%;
      }
      .output-file-content table td,
      .output-file-content table th {
        border: 1px solid var(--border);
        padding: 0.375rem 0.5rem;
        text-align: left;
      }
      .output-file-content table th {
        background: var(--bg);
        font-weight: 600;
      }
      .output-file-content .download-link {
        display: inline-flex;
        align-items: center;
        gap: 0.5rem;
        padding: 0.5rem 1rem;
        background: var(--bg);
        border: 1px solid var(--border);
        border-radius: 4px;
        color: var(--accent);
        text-decoration: none;
        font-size: 0.875rem;
        cursor: pointer;
      }
      .output-file-content .download-link:hover {
        background: var(--border);
      }
      .empty-state {
        color: var(--text-muted);
        font-style: italic;
        padding: 2rem;
        text-align: center;
      }

      /* ---- Feedback ---- */
      .prev-feedback {
        background: var(--bg);
        border: 1px solid var(--border);
        border-radius: 4px;
        padding: 0.625rem 0.75rem;
        margin-top: 0.75rem;
        font-size: 0.8125rem;
        color: var(--text-muted);
        line-height: 1.5;
      }
      .prev-feedback-label {
        font-size: 0.7rem;
        font-weight: 600;
        text-transform: uppercase;
        letter-spacing: 0.04em;
        margin-bottom: 0.25rem;
        color: var(--text-muted);
      }
      .feedback-textarea {
        width: 100%;
        min-height: 100px;
        padding: 0.75rem;
        border: 1px solid var(--border);
        border-radius: 4px;
        font-family: inherit;
        font-size: 0.9375rem;
        line-height: 1.5;
        resize: vertical;
        color: var(--text);
      }
      .feedback-textarea:focus {
        outline: none;
        border-color: var(--accent);
        box-shadow: 0 0 0 3px rgba(37, 99, 235, 0.1);
      }
      .feedback-status {
        font-size: 0.75rem;
        color: var(--text-muted);
        margin-top: 0.5rem;
        min-height: 1.1em;
      }

      /* ---- Grades (collapsible) ---- */
      .grades-toggle {
        display: flex;
        align-items: center;
        cursor: pointer;
        user-select: none;
      }
      .grades-toggle:hover {
        color: var(--accent);
      }
      .grades-toggle .arrow {
        margin-right: 0.5rem;
        transition: transform 0.15s;
        font-size: 0.75rem;
      }
      .grades-toggle .arrow.open {
        transform: rotate(90deg);
      }
      .grades-content {
        display: none;
        margin-top: 0.75rem;
      }
      .grades-content.open {
        display: block;
      }
      .grades-summary {
        font-size: 0.875rem;
        margin-bottom: 0.75rem;
        display: flex;
        align-items: center;
        gap: 0.5rem;
      }
      .grade-badge {
        display: inline-block;
        padding: 0.125rem 0.5rem;
        border-radius: 9999px;
        font-size: 0.75rem;
        font-weight: 600;
      }
      .grade-pass {
        background: var(--green-bg);
        color: var(--green);
      }
      .grade-fail {
        background: var(--red-bg);
        color: var(--red);
      }
      .assertion-list {
        list-style: none;
      }
      .assertion-item {
        padding: 0.625rem 0;
        border-bottom: 1px solid var(--border);
        font-size: 0.8125rem;
      }
      .assertion-item:last-child {
        border-bottom: none;
      }
      .assertion-status {
        font-weight: 600;
        margin-right: 0.5rem;
      }
      .assertion-status.pass {
        color: var(--green);
      }
      .assertion-status.fail {
        color: var(--red);
      }
      .assertion-evidence {
        color: var(--text-muted);
        font-size: 0.75rem;
        margin-top: 0.25rem;
        padding-left: 1.5rem;
      }

      /* ---- View tabs ---- */
      .view-tabs {
        display: flex;
        gap: 0;
        padding: 0 2rem;
        background: var(--bg);
        border-bottom: 1px solid var(--border);
        flex-shrink: 0;
      }
      .view-tab {
        font-family: "Poppins", sans-serif;
        padding: 0.625rem 1.25rem;
        font-size: 0.8125rem;
        font-weight: 500;
        cursor: pointer;
        border: none;
        background: none;
        color: var(--text-muted);
        border-bottom: 2px solid transparent;
        transition: all 0.15s;
      }
      .view-tab:hover {
        color: var(--text);
      }
      .view-tab.active {
        color: var(--accent);
        border-bottom-color: var(--accent);
      }
      .view-panel {
        display: none;
      }
      .view-panel.active {
        display: flex;
        flex-direction: column;
        flex: 1;
        overflow: hidden;
      }

      /* ---- Benchmark view ---- */
      .benchmark-view {
        padding: 1.5rem 2rem;
        overflow-y: auto;
        flex: 1;
      }
      .benchmark-table {
        border-collapse: collapse;
        background: var(--surface);
        border: 1px solid var(--border);
        border-radius: var(--radius);
        font-size: 0.8125rem;
        width: 100%;
        margin-bottom: 1.5rem;
      }
      .benchmark-table th,
      .benchmark-table td {
        padding: 0.625rem 0.75rem;
        text-align: left;
        border: 1px solid var(--border);
      }
      .benchmark-table th {
        font-family: "Poppins", sans-serif;
        background: var(--header-bg);
        color: var(--header-text);
        font-weight: 500;
        font-size: 0.75rem;
        text-transform: uppercase;
        letter-spacing: 0.04em;
      }
      .benchmark-table tr:hover {
        background: var(--bg);
      }
      .benchmark-table tr.benchmark-row-with {
        background: rgba(33, 150, 243, 0.06);
      }
      .benchmark-table tr.benchmark-row-without {
        background: rgba(255, 193, 7, 0.06);
      }
      .benchmark-table tr.benchmark-row-with:hover {
        background: rgba(33, 150, 243, 0.12);
      }
      .benchmark-table tr.benchmark-row-without:hover {
        background: rgba(255, 193, 7, 0.12);
      }
      .benchmark-table tr.benchmark-row-avg {
        font-weight: 600;
        border-top: 2px solid var(--border);
      }
      .benchmark-table tr.benchmark-row-avg.benchmark-row-with {
        background: rgba(33, 150, 243, 0.12);
      }
      .benchmark-table tr.benchmark-row-avg.benchmark-row-without {
        background: rgba(255, 193, 7, 0.12);
      }
      .benchmark-delta-positive {
        color: var(--green);
        font-weight: 600;
      }
      .benchmark-delta-negative {
        color: var(--red);
        font-weight: 600;
      }
      .benchmark-notes {
        background: var(--surface);
        border: 1px solid var(--border);
        border-radius: var(--radius);
        padding: 1rem;
      }
      .benchmark-notes h3 {
        font-family: "Poppins", sans-serif;
        font-size: 0.875rem;
        margin-bottom: 0.75rem;
      }
      .benchmark-notes ul {
        list-style: disc;
        padding-left: 1.25rem;
      }
      .benchmark-notes li {
        font-size: 0.8125rem;
        line-height: 1.6;
        margin-bottom: 0.375rem;
      }
      .benchmark-empty {
        color: var(--text-muted);
        font-style: italic;
        text-align: center;
        padding: 3rem;
      }

      /* ---- Navigation ---- */
      .nav {
        display: flex;
        justify-content: space-between;
        align-items: center;
        padding: 1rem 2rem;
        border-top: 1px solid var(--border);
        background: var(--surface);
        flex-shrink: 0;
      }
      .nav-btn {
        font-family: "Poppins", sans-serif;
        padding: 0.5rem 1.25rem;
        border: 1px solid var(--border);
        border-radius: var(--radius);
        background: var(--surface);
        cursor: pointer;
        font-size: 0.875rem;
        font-weight: 500;
        color: var(--text);
        transition: all 0.15s;
      }
      .nav-btn:hover:not(:disabled) {
        background: var(--bg);
        border-color: var(--text-muted);
      }
      .nav-btn:disabled {
        opacity: 0.4;
        cursor: not-allowed;
      }
      .done-btn {
        font-family: "Poppins", sans-serif;
        padding: 0.5rem 1.5rem;
        border: 1px solid var(--border);
        border-radius: var(--radius);
        background: var(--surface);
        color: var(--text);
        cursor: pointer;
        font-size: 0.875rem;
        font-weight: 500;
        transition: all 0.15s;
      }
      .done-btn:hover {
        background: var(--bg);
        border-color: var(--text-muted);
      }
      .done-btn.ready {
        border: none;
        background: var(--accent);
        color: white;
        font-weight: 600;
      }
      .done-btn.ready:hover {
        background: var(--accent-hover);
      }
      /* ---- Done overlay ---- */
      .done-overlay {
        display: none;
        position: fixed;
        inset: 0;
        background: rgba(0, 0, 0, 0.5);
        z-index: 100;
        justify-content: center;
        align-items: center;
      }
      .done-overlay.visible {
        display: flex;
      }
      .done-card {
        background: var(--surface);
        border-radius: 12px;
        padding: 2rem 3rem;
        text-align: center;
        box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
        max-width: 500px;
      }
      .done-card h2 {
        font-size: 1.5rem;
        margin-bottom: 0.5rem;
      }
      .done-card p {
        color: var(--text-muted);
        margin-bottom: 1.5rem;
        line-height: 1.5;
      }
      .done-card .btn-row {
        display: flex;
        gap: 0.5rem;
        justify-content: center;
      }
      .done-card button {
        padding: 0.5rem 1.25rem;
        border: 1px solid var(--border);
        border-radius: var(--radius);
        background: var(--surface);
        cursor: pointer;
        font-size: 0.875rem;
      }
      .done-card button:hover {
        background: var(--bg);
      }
      /* ---- Toast ---- */
      .toast {
        position: fixed;
        bottom: 5rem;
        left: 50%;
        transform: translateX(-50%);
        background: var(--header-bg);
        color: var(--header-text);
        padding: 0.625rem 1.25rem;
        border-radius: var(--radius);
        font-size: 0.875rem;
        opacity: 0;
        transition: opacity 0.3s;
        pointer-events: none;
        z-index: 200;
      }
      .toast.visible {
        opacity: 1;
      }
    </style>
  </head>
  <body>
    <div id="app" style="height: 100vh; display: flex; flex-direction: column">
      <div class="header">
        <div>
          <h1>Eval Review: <span id="skill-name"></span></h1>
          <div class="instructions">
            Review each output and leave feedback below. Navigate with arrow
            keys or buttons. When done, copy feedback and paste into Claude
            Code.
          </div>
        </div>
        <div class="progress" id="progress"></div>
      </div>

      <!-- View tabs (only shown when benchmark data exists) -->
      <div class="view-tabs" id="view-tabs" style="display: none">
        <button class="view-tab active" onclick="switchView('outputs')">
          Outputs
        </button>
        <button class="view-tab" onclick="switchView('benchmark')">
          Benchmark
        </button>
      </div>

      <!-- Outputs panel (qualitative review) -->
      <div class="view-panel active" id="panel-outputs">
        <div class="main">
          <!-- Prompt -->
          <div class="section">
            <div class="section-header">
              Prompt
              <span
                class="config-badge"
                id="config-badge"
                style="display: none"
              ></span>
            </div>
            <div class="section-body">
              <div class="prompt-text" id="prompt-text"></div>
            </div>
          </div>

          <!-- Outputs -->
          <div class="section">
            <div class="section-header">Output</div>
            <div class="section-body" id="outputs-body">
              <div class="empty-state">No output files found</div>
            </div>
          </div>

          <!-- Previous Output (collapsible) -->
          <div class="section" id="prev-outputs-section" style="display: none">
            <div class="section-header">
              <div class="grades-toggle" onclick="togglePrevOutputs()">
                <span class="arrow" id="prev-outputs-arrow">&#9654;</span>
                Previous Output
              </div>
            </div>
            <div class="grades-content" id="prev-outputs-content"></div>
          </div>

          <!-- Grades (collapsible) -->
          <div class="section" id="grades-section" style="display: none">
            <div class="section-header">
              <div class="grades-toggle" onclick="toggleGrades()">
                <span class="arrow" id="grades-arrow">&#9654;</span>
                Formal Grades
              </div>
            </div>
            <div class="grades-content" id="grades-content"></div>
          </div>

          <!-- Feedback -->
          <div class="section">
            <div class="section-header">Your Feedback</div>
            <div class="section-body">
              <textarea
                class="feedback-textarea"
                id="feedback"
                placeholder="What do you think of this output? Any issues, suggestions, or things that look great?"
              ></textarea>
              <div class="feedback-status" id="feedback-status"></div>
              <div
                class="prev-feedback"
                id="prev-feedback"
                style="display: none"
              >
                <div class="prev-feedback-label">Previous feedback</div>
                <div id="prev-feedback-text"></div>
              </div>
            </div>
          </div>
        </div>

        <div class="nav" id="outputs-nav">
          <button class="nav-btn" id="prev-btn" onclick="navigate(-1)">
            &#8592; Previous
          </button>
          <button class="done-btn" id="done-btn" onclick="showDoneDialog()">
            Submit All Reviews
          </button>
          <button class="nav-btn" id="next-btn" onclick="navigate(1)">
            Next &#8594;
          </button>
        </div>
      </div>
      <!-- end panel-outputs -->

      <!-- Benchmark panel (quantitative stats) -->
      <div class="view-panel" id="panel-benchmark">
        <div class="benchmark-view" id="benchmark-content">
          <div class="benchmark-empty">
            No benchmark data available. Run a benchmark to see quantitative
            results here.
          </div>
        </div>
      </div>
    </div>

    <!-- Done overlay -->
    <div class="done-overlay" id="done-overlay">
      <div class="done-card">
        <h2>Review Complete</h2>
        <p>
          Your feedback has been saved. Go back to your Claude Code session and
          tell Claude you're done reviewing.
        </p>
        <div class="btn-row">
          <button onclick="closeDoneDialog()">OK</button>
        </div>
      </div>
    </div>

    <!-- Toast -->
    <div class="toast" id="toast"></div>

    <script>
      // ---- Embedded data (injected by generate_review.py) ----
      /*__EMBEDDED_DATA__*/

      // ---- State ----
      let feedbackMap = {}; // run_id -> feedback text
      let currentIndex = 0;
      let visitedRuns = new Set();

      // ---- Init ----
      async function init() {
        // Load saved feedback from server — but only if this isn't a fresh
        // iteration (indicated by previous_feedback being present). When
        // previous feedback exists, the feedback.json on disk is stale from
        // the prior iteration and should not pre-fill the textareas.
        const hasPrevious =
          Object.keys(EMBEDDED_DATA.previous_feedback || {}).length > 0 ||
          Object.keys(EMBEDDED_DATA.previous_outputs || {}).length > 0;
        if (!hasPrevious) {
          try {
            const resp = await fetch("/api/feedback");
            const data = await resp.json();
            if (data.reviews) {
              for (const r of data.reviews) feedbackMap[r.run_id] = r.feedback;
            }
          } catch {
            /* first run, no feedback yet */
          }
        }

        document.getElementById("skill-name").textContent =
          EMBEDDED_DATA.skill_name;
        showRun(0);

        // Wire up feedback auto-save
        const textarea = document.getElementById("feedback");
        let saveTimeout = null;
        textarea.addEventListener("input", () => {
          clearTimeout(saveTimeout);
          document.getElementById("feedback-status").textContent = "";
          saveTimeout = setTimeout(() => saveCurrentFeedback(), 800);
        });
      }

      // ---- Navigation ----
      function navigate(delta) {
        const newIndex = currentIndex + delta;
        if (newIndex >= 0 && newIndex < EMBEDDED_DATA.runs.length) {
          saveCurrentFeedback();
          showRun(newIndex);
        }
      }

      function updateNavButtons() {
        document.getElementById("prev-btn").disabled = currentIndex === 0;
        document.getElementById("next-btn").disabled =
          currentIndex === EMBEDDED_DATA.runs.length - 1;
      }

      // ---- Show a run ----
      function showRun(index) {
        currentIndex = index;
        const run = EMBEDDED_DATA.runs[index];

        // Progress
        document.getElementById("progress").textContent =
          `${index + 1} of ${EMBEDDED_DATA.runs.length}`;

        // Prompt
        document.getElementById("prompt-text").textContent = run.prompt;

        // Config badge
        const badge = document.getElementById("config-badge");
        const configMatch = run.id.match(
          /(with_skill|without_skill|new_skill|old_skill)/,
        );
        if (configMatch) {
          const config = configMatch[1];
          const isBaseline =
            config === "without_skill" || config === "old_skill";
          badge.textContent = config.replace(/_/g, " ");
          badge.className =
            "config-badge " +
            (isBaseline ? "config-baseline" : "config-primary");
          badge.style.display = "inline-block";
        } else {
          badge.style.display = "none";
        }

        // Outputs
        renderOutputs(run);

        // Previous outputs
        renderPrevOutputs(run);

        // Grades
        renderGrades(run);

        // Previous feedback
        const prevFb = (EMBEDDED_DATA.previous_feedback || {})[run.id];
        const prevEl = document.getElementById("prev-feedback");
        if (prevFb) {
          document.getElementById("prev-feedback-text").textContent = prevFb;
          prevEl.style.display = "block";
        } else {
          prevEl.style.display = "none";
        }

        // Feedback
        document.getElementById("feedback").value = feedbackMap[run.id] || "";
        document.getElementById("feedback-status").textContent = "";

        updateNavButtons();

        // Track visited runs and promote done button when all visited
        visitedRuns.add(index);
        const doneBtn = document.getElementById("done-btn");
        if (visitedRuns.size >= EMBEDDED_DATA.runs.length) {
          doneBtn.classList.add("ready");
        }

        // Scroll main content to top
        document.querySelector(".main").scrollTop = 0;
      }

      // ---- Render outputs ----
      function renderOutputs(run) {
        const container = document.getElementById("outputs-body");
        container.innerHTML = "";

        const outputs = run.outputs || [];
        if (outputs.length === 0) {
          container.innerHTML =
            '<div class="empty-state">No output files</div>';
          return;
        }

        for (const file of outputs) {
          const fileDiv = document.createElement("div");
          fileDiv.className = "output-file";

          // Always show file header with download link
          const header = document.createElement("div");
          header.className = "output-file-header";
          const nameSpan = document.createElement("span");
          nameSpan.textContent = file.name;
          header.appendChild(nameSpan);
          const dlBtn = document.createElement("a");
          dlBtn.className = "dl-btn";
          dlBtn.textContent = "Download";
          dlBtn.download = file.name;
          dlBtn.href = getDownloadUri(file);
          header.appendChild(dlBtn);
          fileDiv.appendChild(header);

          const content = document.createElement("div");
          content.className = "output-file-content";

          if (file.type === "text") {
            const pre = document.createElement("pre");
            pre.textContent = file.content;
            content.appendChild(pre);
          } else if (file.type === "image") {
            const img = document.createElement("img");
            img.src = file.data_uri;
            img.alt = file.name;
            content.appendChild(img);
          } else if (file.type === "pdf") {
            const iframe = document.createElement("iframe");
            iframe.src = file.data_uri;
            content.appendChild(iframe);
          } else if (file.type === "xlsx") {
            renderXlsx(content, file.data_b64);
          } else if (file.type === "binary") {
            const a = document.createElement("a");
            a.className = "download-link";
            a.href = file.data_uri;
            a.download = file.name;
            a.textContent = "Download " + file.name;
            content.appendChild(a);
          } else if (file.type === "error") {
            const pre = document.createElement("pre");
            pre.textContent = file.content;
            pre.style.color = "var(--red)";
            content.appendChild(pre);
          }

          fileDiv.appendChild(content);
          container.appendChild(fileDiv);
        }
      }

      // ---- XLSX rendering via SheetJS ----
      function renderXlsx(container, b64Data) {
        try {
          const raw = Uint8Array.from(atob(b64Data), (c) => c.charCodeAt(0));
          const wb = XLSX.read(raw, { type: "array" });

          for (let i = 0; i < wb.SheetNames.length; i++) {
            const sheetName = wb.SheetNames[i];
            const ws = wb.Sheets[sheetName];

            if (wb.SheetNames.length > 1) {
              const sheetLabel = document.createElement("div");
              sheetLabel.style.cssText =
                "font-weight:600; font-size:0.8rem; color:#b0aea5; margin-top:0.5rem; margin-bottom:0.25rem;";
              sheetLabel.textContent = "Sheet: " + sheetName;
              container.appendChild(sheetLabel);
            }

            const htmlStr = XLSX.utils.sheet_to_html(ws, { editable: false });
            const wrapper = document.createElement("div");
            wrapper.innerHTML = htmlStr;
            container.appendChild(wrapper);
          }
        } catch (err) {
          container.textContent = "Error rendering spreadsheet: " + err.message;
        }
      }

      // ---- Grades ----
      function renderGrades(run) {
        const section = document.getElementById("grades-section");
        const content = document.getElementById("grades-content");

        if (!run.grading) {
          section.style.display = "none";
          return;
        }

        const grading = run.grading;
        section.style.display = "block";
        // Reset to collapsed
        content.classList.remove("open");
        document.getElementById("grades-arrow").classList.remove("open");

        const summary = grading.summary || {};
        const expectations = grading.expectations || [];

        let html = '<div style="padding: 1rem;">';

        // Summary line
        const passRate =
          summary.pass_rate != null
            ? Math.round(summary.pass_rate * 100) + "%"
            : "?";
        const badgeClass =
          summary.pass_rate >= 0.8
            ? "grade-pass"
            : summary.pass_rate >= 0.5
              ? ""
              : "grade-fail";
        html += '<div class="grades-summary">';
        html +=
          '<span class="grade-badge ' +
          badgeClass +
          '">' +
          passRate +
          "</span>";
        html +=
          "<span>" +
          (summary.passed || 0) +
          " passed, " +
          (summary.failed || 0) +
          " failed of " +
          (summary.total || 0) +
          "</span>";
        html += "</div>";

        // Assertions list
        html += '<ul class="assertion-list">';
        for (const exp of expectations) {
          const statusClass = exp.passed ? "pass" : "fail";
          const statusIcon = exp.passed ? "\u2713" : "\u2717";
          html += '<li class="assertion-item">';
          html +=
            '<span class="assertion-status ' +
            statusClass +
            '">' +
            statusIcon +
            "</span>";
          html += "<span>" + escapeHtml(exp.text) + "</span>";
          if (exp.evidence) {
            html +=
              '<div class="assertion-evidence">' +
              escapeHtml(exp.evidence) +
              "</div>";
          }
          html += "</li>";
        }
        html += "</ul>";

        html += "</div>";
        content.innerHTML = html;
      }

      function toggleGrades() {
        const content = document.getElementById("grades-content");
        const arrow = document.getElementById("grades-arrow");
        content.classList.toggle("open");
        arrow.classList.toggle("open");
      }

      // ---- Previous outputs (collapsible) ----
      function renderPrevOutputs(run) {
        const section = document.getElementById("prev-outputs-section");
        const content = document.getElementById("prev-outputs-content");
        const prevOutputs = (EMBEDDED_DATA.previous_outputs || {})[run.id];

        if (!prevOutputs || prevOutputs.length === 0) {
          section.style.display = "none";
          return;
        }

        section.style.display = "block";
        // Reset to collapsed
        content.classList.remove("open");
        document.getElementById("prev-outputs-arrow").classList.remove("open");

        // Render the files into the content area
        content.innerHTML = "";
        const wrapper = document.createElement("div");
        wrapper.style.padding = "1rem";

        for (const file of prevOutputs) {
          const fileDiv = document.createElement("div");
          fileDiv.className = "output-file";

          const header = document.createElement("div");
          header.className = "output-file-header";
          const nameSpan = document.createElement("span");
          nameSpan.textContent = file.name;
          header.appendChild(nameSpan);
          const dlBtn = document.createElement("a");
          dlBtn.className = "dl-btn";
          dlBtn.textContent = "Download";
          dlBtn.download = file.name;
          dlBtn.href = getDownloadUri(file);
          header.appendChild(dlBtn);
          fileDiv.appendChild(header);

          const fc = document.createElement("div");
          fc.className = "output-file-content";

          if (file.type === "text") {
            const pre = document.createElement("pre");
            pre.textContent = file.content;
            fc.appendChild(pre);
          } else if (file.type === "image") {
            const img = document.createElement("img");
            img.src = file.data_uri;
            img.alt = file.name;
            fc.appendChild(img);
          } else if (file.type === "pdf") {
            const iframe = document.createElement("iframe");
            iframe.src = file.data_uri;
            fc.appendChild(iframe);
          } else if (file.type === "xlsx") {
            renderXlsx(fc, file.data_b64);
          } else if (file.type === "binary") {
            const a = document.createElement("a");
            a.className = "download-link";
            a.href = file.data_uri;
            a.download = file.name;
            a.textContent = "Download " + file.name;
            fc.appendChild(a);
          }

          fileDiv.appendChild(fc);
          wrapper.appendChild(fileDiv);
        }

        content.appendChild(wrapper);
      }

      function togglePrevOutputs() {
        const content = document.getElementById("prev-outputs-content");
        const arrow = document.getElementById("prev-outputs-arrow");
        content.classList.toggle("open");
        arrow.classList.toggle("open");
      }

      // ---- Feedback (saved to server -> feedback.json) ----
      function saveCurrentFeedback() {
        const run = EMBEDDED_DATA.runs[currentIndex];
        const text = document.getElementById("feedback").value;

        if (text.trim() === "") {
          delete feedbackMap[run.id];
        } else {
          feedbackMap[run.id] = text;
        }

        // Build reviews array from map
        const reviews = [];
        for (const [run_id, feedback] of Object.entries(feedbackMap)) {
          if (feedback.trim()) {
            reviews.push({
              run_id,
              feedback,
              timestamp: new Date().toISOString(),
            });
          }
        }

        fetch("/api/feedback", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ reviews, status: "in_progress" }),
        })
          .then(() => {
            document.getElementById("feedback-status").textContent = "Saved";
          })
          .catch(() => {
            // Static mode or server unavailable — no-op on auto-save,
            // feedback will be downloaded on final submit
            document.getElementById("feedback-status").textContent =
              "Will download on submit";
          });
      }

      // ---- Done ----
      function showDoneDialog() {
        // Save current textarea to feedbackMap (but don't POST yet)
        const run = EMBEDDED_DATA.runs[currentIndex];
        const text = document.getElementById("feedback").value;
        if (text.trim() === "") {
          delete feedbackMap[run.id];
        } else {
          feedbackMap[run.id] = text;
        }

        // POST once with status: complete — include ALL runs so the model
        // can distinguish "no feedback" (looks good) from "not reviewed"
        const reviews = [];
        const ts = new Date().toISOString();
        for (const r of EMBEDDED_DATA.runs) {
          reviews.push({
            run_id: r.id,
            feedback: feedbackMap[r.id] || "",
            timestamp: ts,
          });
        }
        const payload = JSON.stringify(
          { reviews, status: "complete" },
          null,
          2,
        );
        fetch("/api/feedback", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: payload,
        })
          .then(() => {
            document.getElementById("done-overlay").classList.add("visible");
          })
          .catch(() => {
            // Server not available (static mode) — download as file
            const blob = new Blob([payload], { type: "application/json" });
            const url = URL.createObjectURL(blob);
            const a = document.createElement("a");
            a.href = url;
            a.download = "feedback.json";
            a.click();
            URL.revokeObjectURL(url);
            document.getElementById("done-overlay").classList.add("visible");
          });
      }

      function closeDoneDialog() {
        // Reset status back to in_progress
        saveCurrentFeedback();
        document.getElementById("done-overlay").classList.remove("visible");
      }

      // ---- Toast ----
      function showToast(message) {
        const toast = document.getElementById("toast");
        toast.textContent = message;
        toast.classList.add("visible");
        setTimeout(() => toast.classList.remove("visible"), 2000);
      }

      // ---- Keyboard nav ----
      document.addEventListener("keydown", (e) => {
        // Don't capture when typing in textarea
        if (e.target.tagName === "TEXTAREA") return;

        if (e.key === "ArrowLeft" || e.key === "ArrowUp") {
          e.preventDefault();
          navigate(-1);
        } else if (e.key === "ArrowRight" || e.key === "ArrowDown") {
          e.preventDefault();
          navigate(1);
        }
      });

      // ---- Util ----
      function getDownloadUri(file) {
        if (file.data_uri) return file.data_uri;
        if (file.data_b64)
          return "data:application/octet-stream;base64," + file.data_b64;
        if (file.type === "text")
          return (
            "data:text/plain;charset=utf-8," + encodeURIComponent(file.content)
          );
        return "#";
      }

      function escapeHtml(text) {
        const div = document.createElement("div");
        div.textContent = text;
        return div.innerHTML;
      }

      // ---- View switching ----
      function switchView(view) {
        document
          .querySelectorAll(".view-tab")
          .forEach((t) => t.classList.remove("active"));
        document
          .querySelectorAll(".view-panel")
          .forEach((p) => p.classList.remove("active"));
        document
          .querySelector(`[onclick="switchView('${view}')"]`)
          .classList.add("active");
        document.getElementById("panel-" + view).classList.add("active");
      }

      // ---- Benchmark rendering ----
      function renderBenchmark() {
        const data = EMBEDDED_DATA.benchmark;
        if (!data) return;

        // Show the tabs
        document.getElementById("view-tabs").style.display = "flex";

        const container = document.getElementById("benchmark-content");
        const summary = data.run_summary || {};
        const metadata = data.metadata || {};
        const notes = data.notes || [];

        let html = "";

        // Header
        html +=
          "<h2 style='font-family: Poppins, sans-serif; margin-bottom: 0.5rem;'>Benchmark Results</h2>";
        html +=
          "<p style='color: var(--text-muted); font-size: 0.875rem; margin-bottom: 1.25rem;'>";
        if (metadata.skill_name)
          html +=
            "<strong>" + escapeHtml(metadata.skill_name) + "</strong> &mdash; ";
        if (metadata.timestamp) html += metadata.timestamp + " &mdash; ";
        if (metadata.evals_run)
          html += "Evals: " + metadata.evals_run.join(", ") + " &mdash; ";
        html +=
          (metadata.runs_per_configuration || "?") + " runs per configuration";
        html += "</p>";

        // Summary table
        html += '<table class="benchmark-table">';

        function fmtStat(stat, pct) {
          if (!stat) return "—";
          const suffix = pct ? "%" : "";
          const m = pct ? (stat.mean * 100).toFixed(0) : stat.mean.toFixed(1);
          const s = pct
            ? (stat.stddev * 100).toFixed(0)
            : stat.stddev.toFixed(1);
          return m + suffix + " ± " + s + suffix;
        }

        function deltaClass(val) {
          if (!val) return "";
          const n = parseFloat(val);
          if (n > 0) return "benchmark-delta-positive";
          if (n < 0) return "benchmark-delta-negative";
          return "";
        }

        // Discover config names dynamically (everything except "delta")
        const configs = Object.keys(summary).filter((k) => k !== "delta");
        const configA = configs[0] || "config_a";
        const configB = configs[1] || "config_b";
        const labelA = configA
          .replace(/_/g, " ")
          .replace(/\b\w/g, (c) => c.toUpperCase());
        const labelB = configB
          .replace(/_/g, " ")
          .replace(/\b\w/g, (c) => c.toUpperCase());
        const a = summary[configA] || {};
        const b = summary[configB] || {};
        const delta = summary.delta || {};

        html +=
          "<thead><tr><th>Metric</th><th>" +
          escapeHtml(labelA) +
          "</th><th>" +
          escapeHtml(labelB) +
          "</th><th>Delta</th></tr></thead>";
        html += "<tbody>";

        html += "<tr><td><strong>Pass Rate</strong></td>";
        html += "<td>" + fmtStat(a.pass_rate, true) + "</td>";
        html += "<td>" + fmtStat(b.pass_rate, true) + "</td>";
        html +=
          '<td class="' +
          deltaClass(delta.pass_rate) +
          '">' +
          (delta.pass_rate || "—") +
          "</td></tr>";

        // Time (only show row if data exists)
        if (a.time_seconds || b.time_seconds) {
          html += "<tr><td><strong>Time (s)</strong></td>";
          html += "<td>" + fmtStat(a.time_seconds, false) + "</td>";
          html += "<td>" + fmtStat(b.time_seconds, false) + "</td>";
          html +=
            '<td class="' +
            deltaClass(delta.time_seconds) +
            '">' +
            (delta.time_seconds ? delta.time_seconds + "s" : "—") +
            "</td></tr>";
        }

        // Tokens (only show row if data exists)
        if (a.tokens || b.tokens) {
          html += "<tr><td><strong>Tokens</strong></td>";
          html += "<td>" + fmtStat(a.tokens, false) + "</td>";
          html += "<td>" + fmtStat(b.tokens, false) + "</td>";
          html +=
            '<td class="' +
            deltaClass(delta.tokens) +
            '">' +
            (delta.tokens || "—") +
            "</td></tr>";
        }

        html += "</tbody></table>";

        // Per-eval breakdown (if runs data available)
        const runs = data.runs || [];
        if (runs.length > 0) {
          const evalIds = [...new Set(runs.map((r) => r.eval_id))].sort(
            (a, b) => a - b,
          );

          html +=
            "<h3 style='font-family: Poppins, sans-serif; margin-bottom: 0.75rem;'>Per-Eval Breakdown</h3>";

          const hasTime = runs.some(
            (r) => r.result && r.result.time_seconds != null,
          );
          const hasErrors = runs.some((r) => r.result && r.result.errors > 0);

          for (const evalId of evalIds) {
            const evalRuns = runs.filter((r) => r.eval_id === evalId);
            const evalName =
              evalRuns[0] && evalRuns[0].eval_name
                ? evalRuns[0].eval_name
                : "Eval " + evalId;

            html +=
              "<h4 style='font-family: Poppins, sans-serif; margin: 1rem 0 0.5rem; color: var(--text);'>" +
              escapeHtml(evalName) +
              "</h4>";
            html += '<table class="benchmark-table">';
            html += "<thead><tr><th>Config</th><th>Run</th><th>Pass Rate</th>";
            if (hasTime) html += "<th>Time (s)</th>";
            if (hasErrors) html += "<th>Crashes During Execution</th>";
            html += "</tr></thead>";
            html += "<tbody>";

            // Group by config and render with average rows
            const configGroups = [
              ...new Set(evalRuns.map((r) => r.configuration)),
            ];
            for (let ci = 0; ci < configGroups.length; ci++) {
              const config = configGroups[ci];
              const configRuns = evalRuns.filter(
                (r) => r.configuration === config,
              );
              if (configRuns.length === 0) continue;

              const rowClass =
                ci === 0 ? "benchmark-row-with" : "benchmark-row-without";
              const configLabel = config
                .replace(/_/g, " ")
                .replace(/\b\w/g, (c) => c.toUpperCase());

              for (const run of configRuns) {
                const r = run.result || {};
                const prClass =
                  r.pass_rate >= 0.8
                    ? "benchmark-delta-positive"
                    : r.pass_rate < 0.5
                      ? "benchmark-delta-negative"
                      : "";
                html += '<tr class="' + rowClass + '">';
                html += "<td>" + configLabel + "</td>";
                html += "<td>" + run.run_number + "</td>";
                html +=
                  '<td class="' +
                  prClass +
                  '">' +
                  ((r.pass_rate || 0) * 100).toFixed(0) +
                  "% (" +
                  (r.passed || 0) +
                  "/" +
                  (r.total || 0) +
                  ")</td>";
                if (hasTime)
                  html +=
                    "<td>" +
                    (r.time_seconds != null ? r.time_seconds.toFixed(1) : "—") +
                    "</td>";
                if (hasErrors) html += "<td>" + (r.errors || 0) + "</td>";
                html += "</tr>";
              }

              // Average row
              const rates = configRuns.map(
                (r) => (r.result || {}).pass_rate || 0,
              );
              const avgRate = rates.reduce((a, b) => a + b, 0) / rates.length;
              const avgPrClass =
                avgRate >= 0.8
                  ? "benchmark-delta-positive"
                  : avgRate < 0.5
                    ? "benchmark-delta-negative"
                    : "";
              html += '<tr class="benchmark-row-avg ' + rowClass + '">';
              html += "<td>" + configLabel + "</td>";
              html += "<td>Avg</td>";
              html +=
                '<td class="' +
                avgPrClass +
                '">' +
                (avgRate * 100).toFixed(0) +
                "%</td>";
              if (hasTime) {
                const times = configRuns
                  .map((r) => (r.result || {}).time_seconds)
                  .filter((t) => t != null);
                html +=
                  "<td>" +
                  (times.length
                    ? (times.reduce((a, b) => a + b, 0) / times.length).toFixed(
                        1,
                      )
                    : "—") +
                  "</td>";
              }
              if (hasErrors) html += "<td></td>";
              html += "</tr>";
            }
            html += "</tbody></table>";

            // Per-assertion detail for this eval
            const runsWithExpectations = {};
            for (const config of configGroups) {
              runsWithExpectations[config] = evalRuns.filter(
                (r) =>
                  r.configuration === config &&
                  r.expectations &&
                  r.expectations.length > 0,
              );
            }
            const hasAnyExpectations = Object.values(runsWithExpectations).some(
              (runs) => runs.length > 0,
            );
            if (hasAnyExpectations) {
              // Collect all unique assertion texts across all configs
              const allAssertions = [];
              const seen = new Set();
              for (const config of configGroups) {
                for (const run of runsWithExpectations[config]) {
                  for (const exp of run.expectations || []) {
                    if (!seen.has(exp.text)) {
                      seen.add(exp.text);
                      allAssertions.push(exp.text);
                    }
                  }
                }
              }

              html +=
                '<table class="benchmark-table" style="margin-top: 0.5rem;">';
              html += "<thead><tr><th>Assertion</th>";
              for (const config of configGroups) {
                const label = config
                  .replace(/_/g, " ")
                  .replace(/\b\w/g, (c) => c.toUpperCase());
                html += "<th>" + escapeHtml(label) + "</th>";
              }
              html += "</tr></thead><tbody>";

              for (const assertionText of allAssertions) {
                html += "<tr><td>" + escapeHtml(assertionText) + "</td>";

                for (const config of configGroups) {
                  html += "<td>";
                  for (const run of runsWithExpectations[config]) {
                    const exp = (run.expectations || []).find(
                      (e) => e.text === assertionText,
                    );
                    if (exp) {
                      const cls = exp.passed
                        ? "benchmark-delta-positive"
                        : "benchmark-delta-negative";
                      const icon = exp.passed ? "\u2713" : "\u2717";
                      html +=
                        '<span class="' +
                        cls +
                        '" title="Run ' +
                        run.run_number +
                        ": " +
                        escapeHtml(exp.evidence || "") +
                        '">' +
                        icon +
                        "</span> ";
                    } else {
                      html += "— ";
                    }
                  }
                  html += "</td>";
                }
                html += "</tr>";
              }
              html += "</tbody></table>";
            }
          }
        }

        // Notes
        if (notes.length > 0) {
          html += '<div class="benchmark-notes">';
          html += "<h3>Analysis Notes</h3>";
          html += "<ul>";
          for (const note of notes) {
            html += "<li>" + escapeHtml(note) + "</li>";
          }
          html += "</ul></div>";
        }

        container.innerHTML = html;
      }

      // ---- Start ----
      init();
      renderBenchmark();
    </script>
  </body>
</html>

Description Optimization

The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.

Step 1: Generate trigger eval queries

Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:

[
  { "query": "the user prompt", "should_trigger": true },
  { "query": "another prompt", "should_trigger": false }
]

The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).

Bad: "Format this data", "Extract text from PDF", "Create a chart"

Good: "ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"

For the should-trigger queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.

For the should-not-trigger queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.

The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.

Step 2: Review with user

Present the eval set to the user for review using the HTML template:

1. Read the template from assets/eval_review.html 2. Replace the placeholders:

__EVAL_DATA_PLACEHOLDER__ → the JSON array of eval items (no quotes around it — it's a JS variable assignment)
__SKILL_NAME_PLACEHOLDER__ → the skill's name
__SKILL_DESCRIPTION_PLACEHOLDER__ → the skill's current description

3. Write to a temp file (e.g., /tmp/eval_review_<skill-name>.html) and open it: open /tmp/eval_review_<skill-name>.html 4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set" 5. The file downloads to ~/Downloads/eval_set.json — check the Downloads folder for the most recent version in case there are multiple (e.g., eval_set (1).json)

This step matters — bad eval queries lead to bad descriptions.

Step 3: Run the optimization loop

Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."

Save the eval set to the workspace, then run in the background:

python -m scripts.run_loop \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 \
  --verbose

Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.

While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.

This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude with extended thinking to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with best_description — selected by test score rather than train score to avoid overfitting.

How skill triggering works

Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's available_skills list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.

This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.

Step 4: Apply the result

Take best_description from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.

"""Shared utilities for skill-creator scripts."""

from pathlib import Path



def parse_skill_md(skill_path: Path) -> tuple[str, str, str]:
    """Parse a SKILL.md file, returning (name, description, full_content)."""
    content = (skill_path / "SKILL.md").read_text()
    lines = content.split("\n")

    if lines[0].strip() != "---":
        raise ValueError("SKILL.md missing frontmatter (no opening ---)")

    end_idx = None
    for i, line in enumerate(lines[1:], start=1):
        if line.strip() == "---":
            end_idx = i
            break

    if end_idx is None:
        raise ValueError("SKILL.md missing frontmatter (no closing ---)")

    name = ""
    description = ""
    frontmatter_lines = lines[1:end_idx]
    i = 0
    while i < len(frontmatter_lines):
        line = frontmatter_lines[i]
        if line.startswith("name:"):
            name = line[len("name:"):].strip().strip('"').strip("'")
        elif line.startswith("description:"):
            value = line[len("description:"):].strip()
            # Handle YAML multiline indicators (>, |, >-, |-)
            if value in (">", "|", ">-", "|-"):
                continuation_lines: list[str] = []
                i += 1
                while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith("  ") or frontmatter_lines[i].startswith("\t")):
                    continuation_lines.append(frontmatter_lines[i].strip())
                    i += 1
                description = " ".join(continuation_lines)
                continue
            else:
                description = value.strip('"').strip("'")
        i += 1

    return name, description, content

Related skills

Find SkillsSo their agent can intelligently recommend and install the right modular skills from the open ecosystem when they ask how to do something.3M27.9k

Lark Skill MakerRapidly create custom skills for Lark (Feishu) that can be invoked by their AI agents and automation workflows.513k

Skill CreatorGenerate, test, and iterate on new reusable agent skills directly inside their Claude or Cursor workflow.337k166k

BrainstormingTurn vague product ideas into clear, approved designs and specifications through guided collaborative dialogue before writing any code.307k265k

Writing Great SkillsCreate high-quality, reusable agent skills that other developers can discover and install from the Skillselion catalog.290k200k

Find SkillsInstantly surface the best relevant agent skills, MCP servers, and workflows across multiple registries before starting any new project.226k12

FAQ

What does skill-creator do?

skill-creator is a Claude Code skill for ai & agent building.

When should I use skill-creator?

When you need to helps with ai & agent building tasks., or when skill-creator is a claude code skill for ai & agent building.

What are the main capabilities?

skill-creator; AI & Agent Building; AI-coding skill.

Skill Developmentagentsautomation

About

Skill Creator by the numbers

Add your badge

How do I helps with ai & agent building tasks.?

Who is it for?

When should I use this skill?

What you get

Files

Skill Creator

Two entry paths

Step Completion Reports

Communicating with the user

Mandatory Rule for Repo-Mutating Skills

Frontmatter rules (mandatory)

Creating a skill

Capture Intent

Interview and Research

Write the SKILL.md

Writing a good description: pushy + negative triggers

Description length budget

Skill Writing Guide

Running and evaluating test cases

Improving an existing skill

Subpath B1 — Retrofit an existing skill to the standard

Subpath B2 — Iterate on a skill based on eval feedback

Description Optimization

Package and Present (only if present_files tool is available)

Environment-specific notes

Reference files

Post-hoc Analyzer Agent

Role

Inputs

Process

Step 1: Read Comparison Result

Step 2: Read Both Skills

Step 3: Read Both Transcripts

Step 4: Analyze Instruction Following

Step 5: Identify Winner Strengths

Step 6: Identify Loser Weaknesses

Step 7: Generate Improvement Suggestions

Step 8: Write Analysis Results

Output Format

Guidelines

Categories for Suggestions

Priority Levels

Analyzing Benchmark Results

Role

Inputs

Process

Step 1: Read Benchmark Data

Step 2: Analyze Per-Assertion Patterns

Step 3: Analyze Cross-Eval Patterns

Step 4: Analyze Metrics Patterns

Step 5: Generate Notes

Step 6: Write Notes

Guidelines

Blind Comparator Agent

Role

Inputs

Process

Step 1: Read Both Outputs

Step 2: Understand the Task

Step 3: Generate Evaluation Rubric

Step 4: Evaluate Each Output Against the Rubric

Step 5: Check Assertions (if provided)

Step 6: Determine the Winner

Step 7: Write Comparison Results

Output Format

Field Descriptions

Guidelines

Grader Agent

Role

Inputs

Process

Step 1: Read the Transcript

Step 2: Examine Output Files

Step 3: Evaluate Each Assertion

Step 4: Extract and Verify Claims

Step 5: Read User Notes

Step 6: Critique the Evals

Package and Present (only if `present_files` tool is available)