Skill Master

Name: Skill Master
Author: itechmeat

itechmeat/llm-code

153 installs
22 repo stars
Updated August 1, 2026
itechmeat/llm-code

Author, structure, and maintain high-quality Claude Code skills with consistent triggers, references, and progressive disclosure so agent capabilities stay reliable and reusable.

About

Covers authoring and maintaining Claude Code skills with proper SKILL.md structure, trigger phrases, progressive disclosure, bundled references, and quality conventions so agent capabilities stay reliable and reusable.

SKILL.md structure and conventions
Trigger phrases and progressive disclosure
Bundled references and helper scripts
Reusable agent capability packaging
Quality checks for skill maintainability

Skill Master by the numbers

153 all-time installs (skills.sh)
Ranked #202 of 782 Skill Development skills by installs in the Skillselion catalog
Data as of Aug 2, 2026 (Skillselion catalog sync)

npx skills add https://github.com/itechmeat/llm-code --skill skill-master

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/itechmeat/llm-code/skill-master.svg)](https://skillselion.com/skills/itechmeat/llm-code/skill-master)

Installs	153
repo stars	★ 22
Last updated	August 1, 2026
Repository	itechmeat/llm-code ↗

What it does

Author, structure, and maintain high-quality Claude Code skills with consistent triggers, references, and progressive disclosure so agent capabilities stay reliable and reusable.

Files

agents/
assets/
eval-viewer/
references/
scripts/

SKILL.mdMarkdownGitHub ↗

Skill Master

Create, edit, and validate Agent Skills following the open agentskills.io specification. This skill is the entry point for creating and maintaining Agent Skills.

Language requirement: all skills MUST be authored in English.

Quick Navigation

New to skills? Read: references/specification.md
SKILL.md templates? See: assets/skill-templates.md
Writing effective descriptions & instructions? Read: references/writing-skills.md
Structuring multi-step processes? Read: references/workflows.md
Adding scripts to a skill? Read: references/scripts.md
Adding assets/templates to a skill? Read: references/assets.md
Advanced features (context, agents, hooks)? Read: references/advanced-features.md
Creating from docs? Read: references/docs-ingestion.md
Testing, evals, benchmarking, or iterative improvement? Read: references/testing-troubleshooting.md, references/eval-testing.md, references/iterative-improvement.md
Description optimization? Read: references/description-optimization.md
JSON schemas (evals, grading, benchmark)? Read: references/schemas.md
Eval set review UI and output viewer? See: assets/eval_review.html, eval-viewer/
Validation & packaging? See scripts/

When to Use

Creating a new skill from scratch
Updating an existing skill
Creating a skill by ingesting external documentation
Validating or packaging a skill for distribution
Running the full skill-creator-style loop: interview, draft, eval, benchmark, viewer review, and iteration

skill-master subsumes skill-creator; for the detailed operator playbook, workspace layout, timing capture, benchmark analysis, and viewer handoff, read references/iterative-improvement.md.

Skill Structure (Required)

my-skill/
├── SKILL.md          # Required: instructions + metadata (must be human-readable too)
├── metadata.json     # Optional: extended metadata for publishing
├── references/       # Optional: documentation, guides, API references
├── examples/         # Optional: sample outputs, usage examples
├── scripts/          # Optional: executable code
└── assets/           # Optional: templates, images, data files

_Do NOT include README.md in skill folders. SKILL.md is the single source of truth. If migrating a skill with README.md, merge its content into SKILL.md and delete it._

Folder Purposes (CRITICAL)

Folder	Purpose	Examples
`references/`	Documentation for agents to read	Guides, API docs, concept explanations, troubleshooting
`examples/`	Sample outputs showing expected format	Output examples, usage demonstrations
`assets/`	Static resources to copy/use	Document templates, config templates, images, schemas
`scripts/`	Executable code to run	Python scripts, shell scripts, validators

When to Use Each

references/ — documentation the agent reads and understands
examples/ — sample outputs showing expected format
assets/ — templates, configs, schemas to copy verbatim
scripts/ — executable code the agent runs

IMPORTANT: Templates belong in assets/, examples in examples/, documentation in references/.

Frontmatter Schema

Every SKILL.md MUST start with YAML frontmatter:

---
name: skill-name
description: "What it does. Keywords: term1, term2."
metadata:
  author: your-name
  version: "1.2.3"
  release_date: "2026-01-01"
---

Field order: name → description → license → compatibility → metadata

Required Fields

Field	Constraints
name	1-64 chars, lowercase `a-z0-9-`, no `--`, no leading/trailing `-`, must match folder name
description	1-1024 chars (target: 80-150), describes what skill does + when to use it, include keywords

Optional Fields (Top Level)

Field	Purpose
license	License name or reference to bundled LICENSE file
compatibility	Environment requirements (max 500 chars)
metadata	Object for arbitrary key-value pairs (see below)

metadata Object (Common Fields)

Field	Purpose
author	Author name or organization
version	Upstream product version or skill version
release_date	Last meaningful update date (`YYYY-MM-DD`)
argument-hint	Hint for autocomplete, e.g., `[issue-number]`

Optional Fields (Claude Code / Advanced)

Field	Purpose
disable-model-invocation	`true` = only user can invoke (via `/name`). Default: `false`
user-invocable	`false` = hidden from `/` menu, only agent can load. Default: `true`
allowed-tools	Space-delimited tools agent can use without asking, e.g., `Read Grep Glob`
model	Specific model to use when skill is active
context	Set to `fork` to run in a forked subagent context
agent	Subagent type when `context: fork`, e.g., `Explore`, `Plan`
hooks	Hooks scoped to skill's lifecycle (see agent documentation)

Invocation Control Matrix

Frontmatter	User can invoke	Agent can invoke	Notes
(default)	✅ Yes	✅ Yes	Description in context, loads when used
`disable-model-invocation: true`	✅ Yes	❌ No	For manual workflows with side effects
`user-invocable: false`	❌ No	✅ Yes	Background knowledge, not a command

Variable Substitutions

Available placeholders in skill content:

Variable	Description
`$ARGUMENTS`	All arguments passed when invoking the skill
`${CLAUDE_SESSION_ID}`	Current session ID for logging or session-specific files

If $ARGUMENTS is not in content, arguments are appended as ARGUMENTS: <value>.

Example:

---
name: fix-issue
description: Fix a GitHub issue
disable-model-invocation: true
---
Fix GitHub issue $ARGUMENTS following our coding standards.

Dynamic Context Injection

Use !command`` syntax to run shell commands before skill content is sent to the agent:

## Pull request context

- PR diff: !`gh pr diff`
- Changed files: !`gh pr diff --name-only`

## Your task

Review this pull request...

The command output replaces the placeholder, so the agent receives actual data.

metadata.json (Optional)

For publishing or extended metadata, create metadata.json:

{
  "version": "1.0.0",
  "organization": "Your Org",
  "date": "January 2026",
  "abstract": "Brief description of what this skill provides...",
  "references": ["https://docs.example.com", "https://github.com/org/repo"]
}

Fields:

version — Skill version (semver)
organization — Author or organization
date — Publication date
abstract — Extended description (can be longer than frontmatter)
references — List of source documentation URLs

Name Validation Examples

# Valid
name: pdf-processing
name: data-analysis
name: code-review

# Invalid
name: PDF-Processing  # uppercase not allowed
name: -pdf            # cannot start with hyphen
name: pdf--processing # consecutive hyphens not allowed

Description Rules

Purpose: Tell the LLM what the skill does and when to activate it. Minimize tokens — just enough for activation decision.

Formulas:

For library/reference skills (Claude Code, keyword-based discovery):

[Product] [core function]. Covers [2-3 key topics]. Keywords: [terms].

For workflow/automation skills (Claude.ai, trigger-based activation):

[What it does]. Use when user [specific trigger phrases].

Critical for auto-triggering: include explicit "Use when user says / asks / mentions..." phrases. Without them, Claude may not load the skill automatically. See references/writing-skills.md for good/bad examples and debugging tips.

Constraints:

Target: 80-150 chars
Max: 1024 chars (300 if keeping it minimal)
No marketing ("powerful", "comprehensive", "modern")
No filler ("this skill", "use this for", "helps with")
No XML angle brackets < >
Names with "claude" or "anthropic" are reserved

Good examples:

description: "Turso SQLite database. Covers encryption, sync, agent patterns. Keywords: Turso, libSQL, SQLite."

description: "Base UI unstyled React components. Covers forms, menus, overlays. Keywords: @base-ui/react, render props."

description: "Inworld TTS API. Covers voice cloning, audio markups, timestamps. Keywords: Inworld, TTS, visemes."

Poor examples:

# Too vague
description: "Helps with PDFs."

# Too verbose
description: "Turso embedded SQLite database for modern apps and AI agents. Covers encryption, authorization, sync, partial sync, and agent database patterns."

# Marketing
description: "A powerful solution for all your database needs."

Keywords: product name, package name, 3-5 terms max.

How Skills Work (Progressive Disclosure)

1. Discovery: Agent loads only name + description of each skill (~50-100 tokens) 2. Activation: When task matches, agent reads full SKILL.md into context 3. Execution: Agent follows instructions, loads referenced files as needed

Key rule: Keep SKILL.md under 500 lines. Move details to references/.

Creating a New Skill

_Pro Tip: You can use the skill-creator skill (available in Claude.ai or Claude Code) to interactively generate your first draft, then refine it using the steps below._

Step 1: Scaffold

python scripts/init_skill.py <skill-name>
# Or specify custom directory:
python scripts/init_skill.py <skill-name> --skills-dir skills

Or manually create:

<skills-folder>/<skill-name>/
├── SKILL.md
├── references/   # For documentation, guides
└── assets/       # For templates, static files

Step 2: Write Frontmatter

---
name: <skill-name>
description: "[Purpose] + [Triggers/Keywords]"
---

Step 3: Write Body

Recommended sections:

When to use (triggers, situations)
Quick navigation (router to references and assets)
Steps / Recipes / Checklists
Critical prohibitions
Links

Step 4: Add References (documentation)

For each major topic, create references/<topic>.md with:

Actionable takeaways (5-15 bullets)
Gotchas / prohibitions
Practical examples

Step 5: Add Scripts (if applicable)

Consult references/scripts.md for when scripts are worth writing and how to structure them.

Language selection:

Python — default for CLI wrappers, validators, scaffolding
Go — when the skill's ecosystem is Kubernetes/cloud-native (see k8s-cluster-api)
JS/TS — when the skill's ecosystem is Node/npm

Minimal checklist for a script:

argparse (Python) or flag (Go) for all parameters — no hardcoded values
Check tool availability before use (shutil.which() in Python)
Docstring at top with Usage + Examples
All exceptions caught; exit non-zero with a message to stderr

Step 6: Add Assets (if needed)

Consult references/assets.md for when assets are worth creating and how to format them.

For templates or static resources, create assets/<resource>. Common types:

Config templates — .minimal.yaml / .full.yaml pair; add # yaml-language-server: $schema= header
YAML manifests — use ${VAR_NAME:=default} for variables; include purpose + usage comment at top
Text templates — use #-prefixed comments to explain fields and valid values
Markdown checklists / runbooks — - [ ] checkboxes, inline commands, sign-off section
Prompt / prose templates — grouped by use case, each block self-contained

Naming: <tool>.minimal.yaml, <purpose>-checklist.md, <name>.template, <topic>-prompts.md.

Step 7: Validate

python scripts/quick_validate_skill.py <skill-path>
# Compatibility alias:
python scripts/quick_validate.py <skill-path>

Creating a Skill from Documentation

When building a skill from external docs, use the autonomous ingestion workflow:

Phase 1: Scaffold

1. Create skill folder with SKILL.md skeleton 2. Create plan.md for progress tracking 3. Create references/ directory

Phase 2: Build Queue

For each doc link:

Fetch the page
Extract internal doc links (avoid nav duplicates)
Prioritize: concepts → API → operations → troubleshooting

Phase 3: Ingest Loop

For each page:

1. Fetch one page 2. Create references/<topic>.md with actionable summary 3. Update plan.md checkbox 4. Update SKILL.md if it adds a useful recipe/rule

Do not ask user after each page — continue autonomously.

Phase 4: Finalize

Review SKILL.md for completeness
Ensure practical recipes, not docs mirror
plan.md may be deleted manually after ingestion

Critical Prohibitions

Do NOT copy large verbatim chunks from vendor docs (summarize in own words)
Do NOT write skills in languages other than English
Do NOT include project-specific secrets, paths, or assumptions
Do NOT keep SKILL.md over 500 lines
Do NOT skip name validation (must match folder name)
Do NOT use poor descriptions that lack trigger keywords
Do NOT omit product version when creating skills from documentation

Version Tracking

What `version` Means

In this repository, store version metadata as metadata.version and metadata.release_date to match the existing skills and current Copilot skill-file validation. metadata.version holds the upstream product version the skill was built against (e.g., "1.12.3" for CAPI v1.12.3). For standalone skills not tied to an external product, it holds the skill's own version (e.g., "1.2.4" for skill-master). Use "—" when the product uses continuous deployment with no semantic versioning (e.g., hosted services like Cloudflare Workers).

When to Update

Trigger	Update `metadata.version`	Update `metadata.release_date`
Product released a new version	✅ Yes	✅ Yes
Content changed, no upstream version	❌ No	✅ Yes
Typo or minor fix	❌ No	❌ No

When bumping metadata.version or making significant content changes, also update:

SKILLS_VERSIONS.md — set new version + date, move row to top of table
CHANGELOG.md — prepend a new dated block at the top

Links format:

## Links

- [Documentation](https://example.com/docs)
- [Changelog](https://example.com/changelog)
- [GitHub](https://github.com/org/repo)
- [npm](https://www.npmjs.com/package/name)

Order: Documentation → Changelog/Releases → GitHub → Package registry. Include only applicable links.

Validation Checklist

_Pro Tip: You can use the skill-creator skill (available in Claude.ai or Claude Code) to review your skill and suggest improvements before finalizing._

[ ] name matches folder name, kebab-case, 1-64 chars, no --
[ ] description includes WHAT it does AND WHEN to use it (trigger phrases)
[ ] description has no XML angle brackets < >
[ ] SKILL.md under 500 lines
[ ] Documentation in references/, templates in assets/
[ ] Complex multi-step processes extracted to references/workflows.md
[ ] All text in English
[ ] Skill triggers on obvious test queries
[ ] Skill does NOT trigger on unrelated queries

Full pre-publish checklist: references/testing-troubleshooting.md

Scripts

Script	Purpose
`init_skill.py`	Scaffold new Agent Skill (agentskills.io)
`init_copilot_asset.py`	Scaffold Copilot-specific assets (instructions, agents)
`quick_validate_skill.py`	Validate skill structure
`quick_validate.py`	Compatibility wrapper for legacy validation workflow
`package_skill.py`	Package skill into distributable zip
`run_eval.py`	Run evals via `claude -p` with parallel workers
`run_loop.py`	Eval+improve loop with train/test split
`improve_description.py`	Improve skill description using eval feedback
`aggregate_benchmark.py`	Aggregate grading results into benchmark stats
`generate_report.py`	Generate HTML report from loop iteration data

Agents: agents/grader.md (grade expectations), agents/comparator.md (blind A/B comparison), agents/analyzer.md (post-hoc analysis + benchmark)

Post-hoc Analyzer Agent

Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.

Role

After the blind comparator determines a winner, the Post-hoc Analyzer "unblinds" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?

Inputs

You receive these parameters in your prompt:

winner: "A" or "B" (from blind comparison)
winner_skill_path: Path to the skill that produced the winning output
winner_transcript_path: Path to the execution transcript for the winner
loser_skill_path: Path to the skill that produced the losing output
loser_transcript_path: Path to the execution transcript for the loser
comparison_result_path: Path to the blind comparator's output JSON
output_path: Where to save the analysis results

Process

Step 1: Read Comparison Result

1. Read the blind comparator's output at comparison_result_path 2. Note the winning side (A or B), the reasoning, and any scores 3. Understand what the comparator valued in the winning output

Step 2: Read Both Skills

1. Read the winner skill's SKILL.md and key referenced files 2. Read the loser skill's SKILL.md and key referenced files 3. Identify structural differences:

Instructions clarity and specificity
Script/tool usage patterns
Example coverage
Edge case handling

Step 3: Read Both Transcripts

1. Read the winner's transcript 2. Read the loser's transcript 3. Compare execution patterns:

How closely did each follow their skill's instructions?
What tools were used differently?
Where did the loser diverge from optimal behavior?
Did either encounter errors or make recovery attempts?

Step 4: Analyze Instruction Following

For each transcript, evaluate:

Did the agent follow the skill's explicit instructions?
Did the agent use the skill's provided tools/scripts?
Were there missed opportunities to leverage skill content?
Did the agent add unnecessary steps not in the skill?

Score instruction following 1-10 and note specific issues.

Step 5: Identify Winner Strengths

Determine what made the winner better:

Clearer instructions that led to better behavior?
Better scripts/tools that produced better output?
More comprehensive examples that guided edge cases?
Better error handling guidance?

Be specific. Quote from skills/transcripts where relevant.

Step 6: Identify Loser Weaknesses

Determine what held the loser back:

Ambiguous instructions that led to suboptimal choices?
Missing tools/scripts that forced workarounds?
Gaps in edge case coverage?
Poor error handling that caused failures?

Step 7: Generate Improvement Suggestions

Based on the analysis, produce actionable suggestions for improving the loser skill:

Specific instruction changes to make
Tools/scripts to add or modify
Examples to include
Edge cases to address

Prioritize by impact. Focus on changes that would have changed the outcome.

Step 8: Write Analysis Results

Save structured analysis to {output_path}.

Output Format

Write a JSON file with this structure:

{
  "comparison_summary": {
    "winner": "A",
    "winner_skill": "path/to/winner/skill",
    "loser_skill": "path/to/loser/skill",
    "comparator_reasoning": "Brief summary of why comparator chose winner"
  },
  "winner_strengths": ["Clear step-by-step instructions for handling multi-page documents", "Included validation script that caught formatting errors", "Explicit guidance on fallback behavior when OCR fails"],
  "loser_weaknesses": ["Vague instruction 'process the document appropriately' led to inconsistent behavior", "No script for validation, agent had to improvise and made errors", "No guidance on OCR failure, agent gave up instead of trying alternatives"],
  "instruction_following": {
    "winner": {
      "score": 9,
      "issues": ["Minor: skipped optional logging step"]
    },
    "loser": {
      "score": 6,
      "issues": ["Did not use the skill's formatting template", "Invented own approach instead of following step 3", "Missed the 'always validate output' instruction"]
    }
  },
  "improvement_suggestions": [
    {
      "priority": "high",
      "category": "instructions",
      "suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
      "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
    },
    {
      "priority": "high",
      "category": "tools",
      "suggestion": "Add validate_output.py script similar to winner skill's validation approach",
      "expected_impact": "Would catch formatting errors before final output"
    },
    {
      "priority": "medium",
      "category": "error_handling",
      "suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
      "expected_impact": "Would prevent early failure on difficult documents"
    }
  ],
  "transcript_insights": {
    "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
    "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
  }
}

Guidelines

Be specific: Quote from skills and transcripts, don't just say "instructions were unclear"
Be actionable: Suggestions should be concrete changes, not vague advice
Focus on skill improvements: The goal is to improve the losing skill, not critique the agent
Prioritize by impact: Which changes would most likely have changed the outcome?
Consider causation: Did the skill weakness actually cause the worse output, or is it incidental?
Stay objective: Analyze what happened, don't editorialize
Think about generalization: Would this improvement help on other evals too?

Categories for Suggestions

Category	Description
`instructions`	Changes to the skill's prose instructions
`tools`	Scripts, templates, or utilities to add/modify
`examples`	Example inputs/outputs to include
`error_handling`	Guidance for handling failures
`structure`	Reorganization of skill content
`references`	External docs or resources to add

Priority Levels

high: Would likely change the outcome of this comparison
medium: Would improve quality but may not change win/loss
low: Nice to have, marginal improvement

---

Analyzing Benchmark Results

When analyzing benchmark results, the analyzer's purpose is to surface patterns and anomalies across multiple runs, not suggest skill improvements.

Role

Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.

Inputs

You receive these parameters in your prompt:

benchmark_data_path: Path to the in-progress benchmark.json with all run results
skill_path: Path to the skill being benchmarked
output_path: Where to save the notes (as JSON array of strings)

Process

Step 1: Read Benchmark Data

1. Read the benchmark.json containing all run results 2. Note the configurations tested (with_skill, without_skill) 3. Understand the run_summary aggregates already calculated

Step 2: Analyze Per-Assertion Patterns

For each expectation across all runs:

Does it always pass in both configurations? (may not differentiate skill value)
Does it always fail in both configurations? (may be broken or beyond capability)
Does it always pass with skill but fail without? (skill clearly adds value here)
Does it always fail with skill but pass without? (skill may be hurting)
Is it highly variable? (flaky expectation or non-deterministic behavior)

Step 3: Analyze Cross-Eval Patterns

Look for patterns across evals:

Are certain eval types consistently harder/easier?
Do some evals show high variance while others are stable?
Are there surprising results that contradict expectations?

Step 4: Analyze Metrics Patterns

Look at time_seconds, tokens, tool_calls:

Does the skill significantly increase execution time?
Is there high variance in resource usage?
Are there outlier runs that skew the aggregates?

Step 5: Generate Notes

Write freeform observations as a list of strings. Each note should:

State a specific observation
Be grounded in the data (not speculation)
Help the user understand something the aggregate metrics don't show

Examples:

"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"
"Eval 3 shows high variance (50% +/- 40%) - run 2 had an unusual failure that may be flaky"
"Without-skill runs consistently fail on table extraction expectations (0% pass rate)"
"Skill adds 13s average execution time but improves pass rate by 50%"
"Token usage is 80% higher with skill, primarily due to script output parsing"
"All 3 without-skill runs for eval 1 produced empty output"

Step 6: Write Notes

Save notes to {output_path} as a JSON array of strings:

["Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value", "Eval 3 shows high variance (50% +/- 40%) - run 2 had an unusual failure", "Without-skill runs consistently fail on table extraction expectations", "Skill adds 13s average execution time but improves pass rate by 50%"]

Guidelines

DO:

Report what you observe in the data
Be specific about which evals, expectations, or runs you're referring to
Note patterns that aggregate metrics would hide
Provide context that helps interpret the numbers

DO NOT:

Suggest improvements to the skill (that's for the improvement step, not benchmarking)
Make subjective quality judgments ("the output was good/bad")
Speculate about causes without evidence
Repeat information already in the run_summary aggregates

Blind Comparator Agent

Compare two outputs WITHOUT knowing which skill produced them.

Role

The Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.

Your judgment is based purely on output quality and task completion.

Inputs

You receive these parameters in your prompt:

output_a_path: Path to the first output file or directory
output_b_path: Path to the second output file or directory
eval_prompt: The original task/prompt that was executed
expectations: List of expectations to check (optional - may be empty)

Process

Step 1: Read Both Outputs

1. Examine output A (file or directory) 2. Examine output B (file or directory) 3. Note the type, structure, and content of each 4. If outputs are directories, examine all relevant files inside

Step 2: Understand the Task

1. Read the eval_prompt carefully 2. Identify what the task requires:

What should be produced?
What qualities matter (accuracy, completeness, format)?
What would distinguish a good output from a poor one?

Step 3: Generate Evaluation Rubric

Based on the task, generate a rubric with two dimensions:

Content Rubric (what the output contains):

Criterion	1 (Poor)	3 (Acceptable)	5 (Excellent)
Correctness	Major errors	Minor errors	Fully correct
Completeness	Missing key elements	Mostly complete	All elements present
Accuracy	Significant inaccuracies	Minor inaccuracies	Accurate throughout

Structure Rubric (how the output is organized):

Criterion	1 (Poor)	3 (Acceptable)	5 (Excellent)
Organization	Disorganized	Reasonably organized	Clear, logical structure
Formatting	Inconsistent/broken	Mostly consistent	Professional, polished
Usability	Difficult to use	Usable with effort	Easy to use

Adapt criteria to the specific task. For example:

PDF form: "Field alignment", "Text readability", "Data placement"
Document: "Section structure", "Heading hierarchy", "Paragraph flow"
Data output: "Schema correctness", "Data types", "Completeness"

Step 4: Evaluate Each Output Against the Rubric

For each output (A and B):

1. Score each criterion on the rubric (1-5 scale) 2. Calculate dimension totals: Content score, Structure score 3. Calculate overall score: Average of dimension scores, scaled to 1-10

Step 5: Check Assertions (if provided)

If expectations are provided:

1. Check each expectation against output A 2. Check each expectation against output B 3. Count pass rates for each output 4. Use expectation scores as secondary evidence (not the primary decision factor)

Step 6: Determine the Winner

Compare A and B based on (in priority order):

1. Primary: Overall rubric score (content + structure) 2. Secondary: Assertion pass rates (if applicable) 3. Tiebreaker: If truly equal, declare a TIE

Be decisive - ties should be rare. One output is usually better, even if marginally.

Step 7: Write Comparison Results

Save results to a JSON file at the path specified (or comparison.json if not specified).

Output Format

Write a JSON file with this structure:

{
  "winner": "A",
  "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
  "rubric": {
    "A": {
      "content": {
        "correctness": 5,
        "completeness": 5,
        "accuracy": 4
      },
      "structure": {
        "organization": 4,
        "formatting": 5,
        "usability": 4
      },
      "content_score": 4.7,
      "structure_score": 4.3,
      "overall_score": 9.0
    },
    "B": {
      "content": {
        "correctness": 3,
        "completeness": 2,
        "accuracy": 3
      },
      "structure": {
        "organization": 3,
        "formatting": 2,
        "usability": 3
      },
      "content_score": 2.7,
      "structure_score": 2.7,
      "overall_score": 5.4
    }
  },
  "output_quality": {
    "A": {
      "score": 9,
      "strengths": ["Complete solution", "Well-formatted", "All fields present"],
      "weaknesses": ["Minor style inconsistency in header"]
    },
    "B": {
      "score": 5,
      "strengths": ["Readable output", "Correct basic structure"],
      "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
    }
  },
  "expectation_results": {
    "A": {
      "passed": 4,
      "total": 5,
      "pass_rate": 0.8,
      "details": [
        { "text": "Output includes name", "passed": true },
        { "text": "Output includes date", "passed": true },
        { "text": "Format is PDF", "passed": true },
        { "text": "Contains signature", "passed": false },
        { "text": "Readable text", "passed": true }
      ]
    },
    "B": {
      "passed": 3,
      "total": 5,
      "pass_rate": 0.6,
      "details": [
        { "text": "Output includes name", "passed": true },
        { "text": "Output includes date", "passed": false },
        { "text": "Format is PDF", "passed": true },
        { "text": "Contains signature", "passed": false },
        { "text": "Readable text", "passed": true }
      ]
    }
  }
}

If no expectations were provided, omit the expectation_results field entirely.

Field Descriptions

winner: "A", "B", or "TIE"
reasoning: Clear explanation of why the winner was chosen (or why it's a tie)
rubric: Structured rubric evaluation for each output
content: Scores for content criteria (correctness, completeness, accuracy)
structure: Scores for structure criteria (organization, formatting, usability)
content_score: Average of content criteria (1-5)
structure_score: Average of structure criteria (1-5)
overall_score: Combined score scaled to 1-10
output_quality: Summary quality assessment
score: 1-10 rating (should match rubric overall_score)
strengths: List of positive aspects
weaknesses: List of issues or shortcomings
expectation_results: (Only if expectations provided)
passed: Number of expectations that passed
total: Total number of expectations
pass_rate: Fraction passed (0.0 to 1.0)
details: Individual expectation results

Guidelines

Stay blind: DO NOT try to infer which skill produced which output. Judge purely on output quality.
Be specific: Cite specific examples when explaining strengths and weaknesses.
Be decisive: Choose a winner unless outputs are genuinely equivalent.
Output quality first: Assertion scores are secondary to overall task completion.
Be objective: Don't favor outputs based on style preferences; focus on correctness and completeness.

Grader Agent

Evaluate expectations against an execution transcript and outputs.

Role

The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.

You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.

Inputs

You receive these parameters in your prompt:

expectations: List of expectations to evaluate (strings)
transcript_path: Path to the execution transcript (markdown file)
outputs_dir: Directory containing output files from execution

Process

Step 1: Read the Transcript

1. Read the transcript file completely 2. Note the eval prompt, execution steps, and final result 3. Identify any issues or errors documented

Step 2: Examine Output Files

1. List files in outputs_dir 2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced. 3. Note contents, structure, and quality

Step 3: Evaluate Each Assertion

For each expectation:

1. Search for evidence in the transcript and outputs 2. Determine verdict:

PASS: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance
FAIL: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)

3. Cite the evidence: Quote the specific text or describe what you found

Step 4: Extract and Verify Claims

Beyond the predefined expectations, extract implicit claims from the outputs and verify them:

1. Extract claims from the transcript and outputs:

Factual statements ("The form has 12 fields")
Process claims ("Used pypdf to fill the form")
Quality claims ("All fields were filled correctly")

2. Verify each claim:

Factual claims: Can be checked against the outputs or external sources
Process claims: Can be verified from the transcript
Quality claims: Evaluate whether the claim is justified

3. Flag unverifiable claims: Note claims that cannot be verified with available information

This catches issues that predefined expectations might miss.

Step 5: Read User Notes

If {outputs_dir}/user_notes.md exists:

1. Read it and note any uncertainties or issues flagged by the executor 2. Include relevant concerns in the grading output 3. These may reveal problems even when expectations pass

Step 6: Critique the Evals

After grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.

Good suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion _discriminating_: it passes when the skill genuinely succeeds and fails when it doesn't.

Suggestions worth raising:

An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)
An important outcome you observed — good or bad — that no assertion covers at all
An assertion that can't actually be verified from the available outputs

Keep the bar high. The goal is to flag things the eval author would say "good catch" about, not to nitpick every assertion.

Step 7: Write Grading Results

Save results to {outputs_dir}/../grading.json (sibling to outputs_dir).

Grading Criteria

PASS when:

The transcript or outputs clearly demonstrate the expectation is true
Specific evidence can be cited
The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)

FAIL when:

No evidence found for the expectation
Evidence contradicts the expectation
The expectation cannot be verified from available information
The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete
The output appears to meet the assertion by coincidence rather than by actually doing the work

When uncertain: The burden of proof to pass is on the expectation.

Step 8: Read Executor Metrics and Timing

1. If {outputs_dir}/metrics.json exists, read it and include in grading output 2. If {outputs_dir}/../timing.json exists, read it and include timing data

Output Format

Write a JSON file with this structure:

{
  "expectations": [
    {
      "text": "The output includes the name 'John Smith'",
      "passed": true,
      "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
    },
    {
      "text": "The spreadsheet has a SUM formula in cell B10",
      "passed": false,
      "evidence": "No spreadsheet was created. The output was a text file."
    },
    {
      "text": "The assistant used the skill's OCR script",
      "passed": true,
      "evidence": "Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'"
    }
  ],
  "summary": {
    "passed": 2,
    "failed": 1,
    "total": 3,
    "pass_rate": 0.67
  },
  "execution_metrics": {
    "tool_calls": {
      "Read": 5,
      "Write": 2,
      "Bash": 8
    },
    "total_tool_calls": 15,
    "total_steps": 6,
    "errors_encountered": 0,
    "output_chars": 12450,
    "transcript_chars": 3200
  },
  "timing": {
    "executor_duration_seconds": 165.0,
    "grader_duration_seconds": 26.0,
    "total_duration_seconds": 191.0
  },
  "claims": [
    {
      "claim": "The form has 12 fillable fields",
      "type": "factual",
      "verified": true,
      "evidence": "Counted 12 fields in field_info.json"
    }
  ],
  "user_notes_summary": {
    "uncertainties": ["Used 2023 data, may be stale"],
    "needs_review": [],
    "workarounds": ["Fell back to text overlay for non-fillable fields"]
  },
  "eval_feedback": {
    "suggestions": [
      {
        "assertion": "The output includes the name 'John Smith'",
        "reason": "A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input"
      },
      {
        "reason": "No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught"
      }
    ],
    "overall": "Assertions check presence but not correctness. Consider adding content verification."
  }
}

Field Descriptions

expectations: Array of graded expectations
text: The original expectation text
passed: Boolean - true if expectation passes
evidence: Specific quote or description supporting the verdict
summary: Aggregate statistics
passed: Count of passed expectations
failed: Count of failed expectations
total: Total expectations evaluated
pass_rate: Fraction passed (0.0 to 1.0)
execution_metrics: Copied from executor's metrics.json (if available)
output_chars: Total character count of output files (proxy for tokens)
transcript_chars: Character count of transcript
timing: Wall clock timing from timing.json (if available)

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Eval Set Review - __SKILL_NAME_PLACEHOLDER__</title>
    <link rel="preconnect" href="https://fonts.googleapis.com" />
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
    <link
      href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap"
      rel="stylesheet"
    />
    <style>
      * {
        box-sizing: border-box;
        margin: 0;
        padding: 0;
      }
      body {
        font-family: "Lora", Georgia, serif;
        background: #faf9f5;
        padding: 2rem;
        color: #141413;
      }
      h1 {
        font-family: "Poppins", sans-serif;
        margin-bottom: 0.5rem;
        font-size: 1.5rem;
      }
      .description {
        color: #b0aea5;
        margin-bottom: 1.5rem;
        font-style: italic;
        max-width: 900px;
      }
      .controls {
        margin-bottom: 1rem;
        display: flex;
        gap: 0.5rem;
      }
      .btn {
        font-family: "Poppins", sans-serif;
        padding: 0.5rem 1rem;
        border: none;
        border-radius: 6px;
        cursor: pointer;
        font-size: 0.875rem;
        font-weight: 500;
      }
      .btn-add {
        background: #6a9bcc;
        color: white;
      }
      .btn-add:hover {
        background: #5889b8;
      }
      .btn-export {
        background: #d97757;
        color: white;
      }
      .btn-export:hover {
        background: #c4613f;
      }
      table {
        width: 100%;
        max-width: 1100px;
        border-collapse: collapse;
        background: white;
        border-radius: 6px;
        overflow: hidden;
        box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
      }
      th {
        font-family: "Poppins", sans-serif;
        background: #141413;
        color: #faf9f5;
        padding: 0.75rem 1rem;
        text-align: left;
        font-size: 0.875rem;
      }
      td {
        padding: 0.75rem 1rem;
        border-bottom: 1px solid #e8e6dc;
        vertical-align: top;
      }
      tr:nth-child(even) td {
        background: #faf9f5;
      }
      tr:hover td {
        background: #f3f1ea;
      }
      .section-header td {
        background: #e8e6dc;
        font-family: "Poppins", sans-serif;
        font-weight: 500;
        font-size: 0.8rem;
        color: #141413;
        text-transform: uppercase;
        letter-spacing: 0.05em;
      }
      .query-input {
        width: 100%;
        padding: 0.4rem;
        border: 1px solid #e8e6dc;
        border-radius: 4px;
        font-size: 0.875rem;
        font-family: "Lora", Georgia, serif;
        resize: vertical;
        min-height: 60px;
      }
      .query-input:focus {
        outline: none;
        border-color: #d97757;
        box-shadow: 0 0 0 2px rgba(217, 119, 87, 0.15);
      }
      .toggle {
        position: relative;
        display: inline-block;
        width: 44px;
        height: 24px;
      }
      .toggle input {
        opacity: 0;
        width: 0;
        height: 0;
      }
      .toggle .slider {
        position: absolute;
        inset: 0;
        background: #b0aea5;
        border-radius: 24px;
        cursor: pointer;
        transition: 0.2s;
      }
      .toggle .slider::before {
        content: "";
        position: absolute;
        width: 18px;
        height: 18px;
        left: 3px;
        bottom: 3px;
        background: white;
        border-radius: 50%;
        transition: 0.2s;
      }
      .toggle input:checked + .slider {
        background: #d97757;
      }
      .toggle input:checked + .slider::before {
        transform: translateX(20px);
      }
      .btn-delete {
        background: #c44;
        color: white;
        padding: 0.3rem 0.6rem;
        border: none;
        border-radius: 4px;
        cursor: pointer;
        font-size: 0.75rem;
        font-family: "Poppins", sans-serif;
      }
      .btn-delete:hover {
        background: #a33;
      }
      .summary {
        margin-top: 1rem;
        color: #b0aea5;
        font-size: 0.875rem;
      }
    </style>
  </head>
  <body>
    <h1>
      Eval Set Review: <span id="skill-name">__SKILL_NAME_PLACEHOLDER__</span>
    </h1>
    <p class="description">
      Current description:
      <span id="skill-desc">__SKILL_DESCRIPTION_PLACEHOLDER__</span>
    </p>

    <div class="controls">
      <button class="btn btn-add" onclick="addRow()">+ Add Query</button>
      <button class="btn btn-export" onclick="exportEvalSet()">
        Export Eval Set
      </button>
    </div>

    <table>
      <thead>
        <tr>
          <th style="width: 65%">Query</th>
          <th style="width: 18%">Should Trigger</th>
          <th style="width: 10%">Actions</th>
        </tr>
      </thead>
      <tbody id="eval-body"></tbody>
    </table>

    <p class="summary" id="summary"></p>

    <script>
      const EVAL_DATA = __EVAL_DATA_PLACEHOLDER__;

      let evalItems = [...EVAL_DATA];

      function render() {
        const tbody = document.getElementById("eval-body");
        tbody.innerHTML = "";

        // Sort: should-trigger first, then should-not-trigger
        const sorted = evalItems
          .map((item, origIdx) => ({ ...item, origIdx }))
          .sort(
            (a, b) => (b.should_trigger ? 1 : 0) - (a.should_trigger ? 1 : 0),
          );

        let lastGroup = null;
        sorted.forEach((item) => {
          const group = item.should_trigger ? "trigger" : "no-trigger";
          if (group !== lastGroup) {
            const headerRow = document.createElement("tr");
            headerRow.className = "section-header";
            headerRow.innerHTML = `<td colspan="3">${item.should_trigger ? "Should Trigger" : "Should NOT Trigger"}</td>`;
            tbody.appendChild(headerRow);
            lastGroup = group;
          }

          const idx = item.origIdx;
          const tr = document.createElement("tr");
          tr.innerHTML = `
          <td><textarea class="query-input" onchange="updateQuery(${idx}, this.value)">${escapeHtml(item.query)}</textarea></td>
          <td>
            <label class="toggle">
              <input type="checkbox" ${item.should_trigger ? "checked" : ""} onchange="updateTrigger(${idx}, this.checked)">
              <span class="slider"></span>
            </label>
            <span style="margin-left:8px;font-size:0.8rem;color:#b0aea5">${item.should_trigger ? "Yes" : "No"}</span>
          </td>
          <td><button class="btn-delete" onclick="deleteRow(${idx})">Delete</button></td>
        `;
          tbody.appendChild(tr);
        });
        updateSummary();
      }

      function escapeHtml(text) {
        const div = document.createElement("div");
        div.textContent = text;
        return div.innerHTML;
      }

      function updateQuery(idx, value) {
        evalItems[idx].query = value;
        updateSummary();
      }
      function updateTrigger(idx, value) {
        evalItems[idx].should_trigger = value;
        render();
      }
      function deleteRow(idx) {
        evalItems.splice(idx, 1);
        render();
      }

      function addRow() {
        evalItems.push({ query: "", should_trigger: true });
        render();
        const inputs = document.querySelectorAll(".query-input");
        inputs[inputs.length - 1].focus();
      }

      function updateSummary() {
        const trigger = evalItems.filter((i) => i.should_trigger).length;
        const noTrigger = evalItems.filter((i) => !i.should_trigger).length;
        document.getElementById("summary").textContent =
          `${evalItems.length} queries total: ${trigger} should trigger, ${noTrigger} should not trigger`;
      }

      function exportEvalSet() {
        const valid = evalItems.filter((i) => i.query.trim() !== "");
        const data = valid.map((i) => ({
          query: i.query.trim(),
          should_trigger: i.should_trigger,
        }));
        const blob = new Blob([JSON.stringify(data, null, 2)], {
          type: "application/json",
        });
        const url = URL.createObjectURL(blob);
        const a = document.createElement("a");
        a.href = url;
        a.download = "eval_set.json";
        document.body.appendChild(a);
        a.click();
        document.body.removeChild(a);
        URL.revokeObjectURL(url);
      }

      render();
    </script>
  </body>
</html>

Skill Templates

Ready-to-use templates for creating Agent Skills.

SKILL.md Template

---
name: <skill-name>
description: "[TODO] Describe what this skill does and when to use it. Include trigger keywords."
---

# <Skill Title>

## When to Use

- [TODO] Situations and triggers

## Quick Navigation

- Topic A: `references/topic-a.md`
- Topic B: `references/topic-b.md`

## Goal

[TODO] 1-3 sentences describing what this skill enables.

## Steps / Recipes

1. [TODO]

## Critical Prohibitions

- [TODO]

## Definition of Done

- [TODO]

## Links

- [TODO] External references

SKILL.md Template (Router Style)

For skills with extensive reference material:

---
name: <skill-name>
description: "[Purpose] + [Keywords for discovery]"
---

# <Skill Title> (Skill Router)

This file is intentionally short. Based on your situation, open the right note under `references/`.

## Start Here

- New to <topic>? Read: `references/concepts.md`
- Quick setup? Read: `references/quickstart.md`
- Integration? Read: `references/api.md`

## Choose by Situation

### Data Modeling

- What goes where? Read: `references/modeling.md`

### Operations

- Deployment: `references/deployment.md`
- Troubleshooting: `references/troubleshooting.md`

## Critical Prohibitions

- [Key prohibitions that apply broadly]

## Links

- Official docs: <URL>

Reference Note Template

# <Topic Title>

Source: <URL or "Internal">

## What This Covers

- 1-3 bullets summarizing scope

## Actionable Takeaways

- 5-15 practical bullets
- Focus on what helps build/operate/debug
- Include gotchas inline

## Examples

[Code examples if essential]

## Related

- `references/related-topic.md`

README.md Template

Human-readable description (not for LLM):

# <Skill Name>

Brief description of what this skill provides.

## What This Skill Covers

- Bullet list of capabilities
- Keep it scannable

## Quick Navigation

- [SKILL.md](SKILL.md) — Entry point for agents
- [references/](references/) — Detailed documentation

## When to Use

Activate this skill when [brief trigger description].

Frontmatter Examples

Minimal (Required Only)

---
name: pdf-processing
description: Extract text and tables from PDF files, fill forms, merge documents.
---

With Optional Fields

---
name: pdf-processing
description: Extract text and tables from PDF files, fill forms, merge documents. Use when working with PDFs, forms, or document extraction.
license: Apache-2.0
compatibility: Requires pdfplumber, PyPDF2
metadata:
  author: example-org
  version: "1.0.0"
  release_date: "2026-01-01"
---

With Allowed Tools

---
name: git-workflow
description: Git branching, merging, and release workflows.
allowed-tools: Bash(git:*) Read Write
---

With Context Fork (Subagent)

---
name: deep-research
description: Research a topic thoroughly in isolated context.
context: fork
agent: Explore
allowed-tools: Read Grep Glob
---

Manual-Only Skill

---
name: deploy
description: Deploy application to production
disable-model-invocation: true
---

Background Knowledge (Not User-Invocable)

---
name: api-conventions
description: API design patterns for this codebase
user-invocable: false
---

With Argument Hint

---
name: fix-issue
description: Fix a GitHub issue
disable-model-invocation: true
argument-hint: <issue-number>
metadata:
  author: team
  version: "1.0.0"
  release_date: "2026-01-01"
---

Full Example (All Fields)

---
name: pr-review
description: Review pull request for issues. Use when asked to review PR or code changes.
license: MIT
compatibility: Requires gh CLI
context: fork
agent: Explore
disable-model-invocation: true
argument-hint: [pr-number]
allowed-tools: Read Grep Bash(gh:*)
metadata:
  author: example-org
  version: "1.0.0"
  release_date: "2026-01-01"
---

Skill with Dynamic Context Template

---
name: pr-summary
description: Summarize changes in a pull request
context: fork
agent: Explore
allowed-tools: Bash(gh:*)
---

## Pull request context

- PR diff: !`gh pr diff`
- PR comments: !`gh pr view --comments`
- Changed files: !`gh pr diff --name-only`

## Your task

Summarize this pull request focusing on:

1. Main changes
2. Potential issues
3. Testing recommendations

Skill with Arguments Template

---
name: fix-issue
description: Fix a GitHub issue
disable-model-invocation: true
argument-hint: <issue-number>
---

Fix GitHub issue $ARGUMENTS following our coding standards:

1. Read the issue description
2. Understand the requirements
3. Implement the fix
4. Write tests
5. Create a commit with message referencing the issue

metadata.json Template

{
  "version": "1.0.0",
  "organization": "Your Organization",
  "date": "January 2026",
  "abstract": "Comprehensive description of what this skill provides, its scope, and how it helps automated code generation and review.",
  "references": ["https://docs.example.com", "https://github.com/org/repo", "https://example.com/api-reference"]
}

Description Examples

Good Descriptions

# Includes purpose + triggers + keywords
description: Extracts text and tables from PDF files, fills PDF forms, and merges multiple PDFs. Use when working with PDF documents or when the user mentions PDFs, forms, or document extraction.

description: Qdrant vector database playbook: core concepts (collections/points/payload), filtering, indexing, snapshots, deployment. Keywords: Qdrant, vector database, embeddings, ANN.

description: PostgreSQL RLS policies for multi-tenant applications. Use when implementing row-level security, tenant isolation, or app.current_user patterns.

Poor Descriptions

# Too vague
description: Helps with PDFs.

# Missing triggers
description: A skill for database operations.

# No keywords
description: Handles some common tasks.

Naming Examples

Valid Names

pdf-processing
data-analysis
code-review
qdrant
postgresql-rls
mcp-server

Invalid Names

PDF-Processing    # uppercase
-pdf              # leading hyphen
pdf-              # trailing hyphen
pdf--processing   # consecutive hyphens
my skill          # spaces
my_skill          # underscores

#!/usr/bin/env python3
"""Generate and serve a review page for eval results.

Reads the workspace directory, discovers runs (directories with outputs/),
embeds all output data into a self-contained HTML page, and serves it via
a tiny HTTP server. Feedback auto-saves to feedback.json in the workspace.

Usage:
    python generate_review.py <workspace-path> [--port PORT] [--skill-name NAME]
    python generate_review.py <workspace-path> --previous-workspace /path/to/old/workspace

No dependencies beyond the Python stdlib are required.
"""

import argparse
import base64
import json
import mimetypes
import os
import re
import signal
import subprocess
import sys
import time
import webbrowser
from functools import partial
from http.server import HTTPServer, BaseHTTPRequestHandler
from pathlib import Path

# Files to exclude from output listings
METADATA_FILES = {"transcript.md", "user_notes.md", "metrics.json"}

# Extensions we render as inline text
TEXT_EXTENSIONS = {
    ".txt", ".md", ".json", ".csv", ".py", ".js", ".ts", ".tsx", ".jsx",
    ".yaml", ".yml", ".xml", ".html", ".css", ".sh", ".rb", ".go", ".rs",
    ".java", ".c", ".cpp", ".h", ".hpp", ".sql", ".r", ".toml",
}

# Extensions we render as inline images
IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"}

# MIME type overrides for common types
MIME_OVERRIDES = {
    ".svg": "image/svg+xml",
    ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
    ".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
}


def get_mime_type(path: Path) -> str:
    ext = path.suffix.lower()
    if ext in MIME_OVERRIDES:
        return MIME_OVERRIDES[ext]
    mime, _ = mimetypes.guess_type(str(path))
    return mime or "application/octet-stream"


def find_runs(workspace: Path) -> list[dict]:
    """Recursively find directories that contain an outputs/ subdirectory."""
    runs: list[dict] = []
    _find_runs_recursive(workspace, workspace, runs)
    runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"]))
    return runs


def _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None:
    if not current.is_dir():
        return

    outputs_dir = current / "outputs"
    if outputs_dir.is_dir():
        run = build_run(root, current)
        if run:
            runs.append(run)
        return

    skip = {"node_modules", ".git", "__pycache__", "skill", "inputs"}
    for child in sorted(current.iterdir()):
        if child.is_dir() and child.name not in skip:
            _find_runs_recursive(root, child, runs)


def build_run(root: Path, run_dir: Path) -> dict | None:
    """Build a run dict with prompt, outputs, and grading data."""
    prompt = ""
    eval_id = None

    # Try eval_metadata.json
    for candidate in [run_dir / "eval_metadata.json", run_dir.parent / "eval_metadata.json"]:
        if candidate.exists():
            try:
                metadata = json.loads(candidate.read_text())
                prompt = metadata.get("prompt", "")
                eval_id = metadata.get("eval_id")
            except (json.JSONDecodeError, OSError):
                pass
            if prompt:
                break

    # Fall back to transcript.md
    if not prompt:
        for candidate in [run_dir / "transcript.md", run_dir / "outputs" / "transcript.md"]:
            if candidate.exists():
                try:
                    text = candidate.read_text()
                    match = re.search(r"## Eval Prompt\n\n([\s\S]*?)(?=\n##|$)", text)
                    if match:
                        prompt = match.group(1).strip()
                except OSError:
                    pass
                if prompt:
                    break

    if not prompt:
        prompt = "(No prompt found)"

    run_id = str(run_dir.relative_to(root)).replace("/", "-").replace("\\", "-")

    # Collect output files
    outputs_dir = run_dir / "outputs"
    output_files: list[dict] = []
    if outputs_dir.is_dir():
        for f in sorted(outputs_dir.iterdir()):
            if f.is_file() and f.name not in METADATA_FILES:
                output_files.append(embed_file(f))

    # Load grading if present
    grading = None
    for candidate in [run_dir / "grading.json", run_dir.parent / "grading.json"]:
        if candidate.exists():
            try:
                grading = json.loads(candidate.read_text())
            except (json.JSONDecodeError, OSError):
                pass
            if grading:
                break

    return {
        "id": run_id,
        "prompt": prompt,
        "eval_id": eval_id,
        "outputs": output_files,
        "grading": grading,
    }


def embed_file(path: Path) -> dict:
    """Read a file and return an embedded representation."""
    ext = path.suffix.lower()
    mime = get_mime_type(path)

    if ext in TEXT_EXTENSIONS:
        try:
            content = path.read_text(errors="replace")
        except OSError:
            content = "(Error reading file)"
        return {
            "name": path.name,
            "type": "text",
            "content": content,
        }
    elif ext in IMAGE_EXTENSIONS:
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "image",
            "mime": mime,
            "data_uri": f"data:{mime};base64,{b64}",
        }
    elif ext == ".pdf":
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "pdf",
            "data_uri": f"data:{mime};base64,{b64}",
        }
    elif ext == ".xlsx":
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "xlsx",
            "data_b64": b64,
        }
    else:
        # Binary / unknown — base64 download link
        try:
            raw = path.read_bytes()
            b64 = base64.b64encode(raw).decode("ascii")
        except OSError:
            return {"name": path.name, "type": "error", "content": "(Error reading file)"}
        return {
            "name": path.name,
            "type": "binary",
            "mime": mime,
            "data_uri": f"data:{mime};base64,{b64}",
        }


def load_previous_iteration(workspace: Path) -> dict[str, dict]:
    """Load previous iteration's feedback and outputs.

    Returns a map of run_id -> {"feedback": str, "outputs": list[dict]}.
    """
    result: dict[str, dict] = {}

    # Load feedback
    feedback_map: dict[str, str] = {}
    feedback_path = workspace / "feedback.json"
    if feedback_path.exists():
        try:
            data = json.loads(feedback_path.read_text())
            feedback_map = {
                r["run_id"]: r["feedback"]
                for r in data.get("reviews", [])
                if r.get("feedback", "").strip()
            }
        except (json.JSONDecodeError, OSError, KeyError):
            pass

    # Load runs (to get outputs)
    prev_runs = find_runs(workspace)
    for run in prev_runs:
        result[run["id"]] = {
            "feedback": feedback_map.get(run["id"], ""),
            "outputs": run.get("outputs", []),
        }

    # Also add feedback for run_ids that had feedback but no matching run
    for run_id, fb in feedback_map.items():
        if run_id not in result:
            result[run_id] = {"feedback": fb, "outputs": []}

    return result


def generate_html(
    runs: list[dict],
    skill_name: str,
    previous: dict[str, dict] | None = None,
    benchmark: dict | None = None,
) -> str:
    """Generate the complete standalone HTML page with embedded data."""
    template_path = Path(__file__).parent / "viewer.html"
    template = template_path.read_text()

    # Build previous_feedback and previous_outputs maps for the template
    previous_feedback: dict[str, str] = {}
    previous_outputs: dict[str, list[dict]] = {}
    if previous:
        for run_id, data in previous.items():
            if data.get("feedback"):
                previous_feedback[run_id] = data["feedback"]
            if data.get("outputs"):
                previous_outputs[run_id] = data["outputs"]

    embedded = {
        "skill_name": skill_name,
        "runs": runs,
        "previous_feedback": previous_feedback,
        "previous_outputs": previous_outputs,
    }
    if benchmark:
        embedded["benchmark"] = benchmark

    data_json = json.dumps(embedded)

    return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};")


# ---------------------------------------------------------------------------
# HTTP server (stdlib only, zero dependencies)
# ---------------------------------------------------------------------------

def _kill_port(port: int) -> None:
    """Kill any process listening on the given port."""
    try:
        result = subprocess.run(
            ["lsof", "-ti", f":{port}"],
            capture_output=True, text=True, timeout=5,
        )
        for pid_str in result.stdout.strip().split("\n"):
            if pid_str.strip():
                try:
                    os.kill(int(pid_str.strip()), signal.SIGTERM)
                except (ProcessLookupError, ValueError):
                    pass
        if result.stdout.strip():
            time.sleep(0.5)
    except subprocess.TimeoutExpired:
        pass
    except FileNotFoundError:
        print("Note: lsof not found, cannot check if port is in use", file=sys.stderr)

class ReviewHandler(BaseHTTPRequestHandler):
    """Serves the review HTML and handles feedback saves.

    Regenerates the HTML on each page load so that refreshing the browser
    picks up new eval outputs without restarting the server.
    """

    def __init__(
        self,
        workspace: Path,
        skill_name: str,
        feedback_path: Path,
        previous: dict[str, dict],
        benchmark_path: Path | None,
        *args,
        **kwargs,
    ):
        self.workspace = workspace
        self.skill_name = skill_name
        self.feedback_path = feedback_path
        self.previous = previous
        self.benchmark_path = benchmark_path
        super().__init__(*args, **kwargs)

    def do_GET(self) -> None:
        if self.path == "/" or self.path == "/index.html":
            # Regenerate HTML on each request (re-scans workspace for new outputs)
            runs = find_runs(self.workspace)
            benchmark = None
            if self.benchmark_path and self.benchmark_path.exists():
                try:
                    benchmark = json.loads(self.benchmark_path.read_text())
                except (json.JSONDecodeError, OSError):
                    pass
            html = generate_html(runs, self.skill_name, self.previous, benchmark)
            content = html.encode("utf-8")
            self.send_response(200)
            self.send_header("Content-Type", "text/html; charset=utf-8")
            self.send_header("Content-Length", str(len(content)))
            self.end_headers()
            self.wfile.write(content)
        elif self.path == "/api/feedback":
            data = b"{}"
            if self.feedback_path.exists():
                data = self.feedback_path.read_bytes()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(data)))
            self.end_headers()
            self.wfile.write(data)
        else:
            self.send_error(404)

    def do_POST(self) -> None:
        if self.path == "/api/feedback":
            length = int(self.headers.get("Content-Length", 0))
            body = self.rfile.read(length)
            try:
                data = json.loads(body)
                if not isinstance(data, dict) or "reviews" not in data:
                    raise ValueError("Expected JSON object with 'reviews' key")
                self.feedback_path.write_text(json.dumps(data, indent=2) + "\n")
                resp = b'{"ok":true}'
                self.send_response(200)
            except (json.JSONDecodeError, OSError, ValueError) as e:
                resp = json.dumps({"error": str(e)}).encode()
                self.send_response(500)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(resp)))
            self.end_headers()
            self.wfile.write(resp)
        else:
            self.send_error(404)

    def log_message(self, format: str, *args: object) -> None:
        # Suppress request logging to keep terminal clean
        pass


def main() -> None:
    parser = argparse.ArgumentParser(description="Generate and serve eval review")
    parser.add_argument("workspace", type=Path, help="Path to workspace directory")
    parser.add_argument("--port", "-p", type=int, default=3117, help="Server port (default: 3117)")
    parser.add_argument("--skill-name", "-n", type=str, default=None, help="Skill name for header")
    parser.add_argument(
        "--previous-workspace", type=Path, default=None,
        help="Path to previous iteration's workspace (shows old outputs and feedback as context)",
    )
    parser.add_argument(
        "--benchmark", type=Path, default=None,
        help="Path to benchmark.json to show in the Benchmark tab",
    )
    parser.add_argument(
        "--static", "-s", type=Path, default=None,
        help="Write standalone HTML to this path instead of starting a server",
    )
    args = parser.parse_args()

    workspace = args.workspace.resolve()
    if not workspace.is_dir():
        print(f"Error: {workspace} is not a directory", file=sys.stderr)
        sys.exit(1)

    runs = find_runs(workspace)
    if not runs:
        print(f"No runs found in {workspace}", file=sys.stderr)
        sys.exit(1)

    skill_name = args.skill_name or workspace.name.replace("-workspace", "")
    feedback_path = workspace / "feedback.json"

    previous: dict[str, dict] = {}
    if args.previous_workspace:
        previous = load_previous_iteration(args.previous_workspace.resolve())

    benchmark_path = args.benchmark.resolve() if args.benchmark else None
    benchmark = None
    if benchmark_path and benchmark_path.exists():
        try:
            benchmark = json.loads(benchmark_path.read_text())
        except (json.JSONDecodeError, OSError):
            pass

    if args.static:
        html = generate_html(runs, skill_name, previous, benchmark)
        args.static.parent.mkdir(parents=True, exist_ok=True)
        args.static.write_text(html)
        print(f"\n  Static viewer written to: {args.static}\n")
        sys.exit(0)

    # Kill any existing process on the target port
    port = args.port
    _kill_port(port)
    handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path)
    try:
        server = HTTPServer(("127.0.0.1", port), handler)
    except OSError:
        # Port still in use after kill attempt — find a free one
        server = HTTPServer(("127.0.0.1", 0), handler)
        port = server.server_address[1]

    url = f"http://localhost:{port}"
    print(f"\n  Eval Viewer")
    print(f"  ─────────────────────────────────")
    print(f"  URL:       {url}")
    print(f"  Workspace: {workspace}")
    print(f"  Feedback:  {feedback_path}")
    if previous:
        print(f"  Previous:  {args.previous_workspace} ({len(previous)} runs)")
    if benchmark_path:
        print(f"  Benchmark: {benchmark_path}")
    print(f"\n  Press Ctrl+C to stop.\n")

    webbrowser.open(url)

    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nStopped.")
        server.server_close()


if __name__ == "__main__":
    main()

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Eval Review</title>
    <link rel="preconnect" href="https://fonts.googleapis.com" />
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
    <link
      href="https://fonts.googleapis.com/css2?family=Poppins:wght@500;600&family=Lora:wght@400;500&display=swap"
      rel="stylesheet"
    />
    <script
      src="https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js"
      integrity="sha384-EnyY0/GSHQGSxSgMwaIPzSESbqoOLSexfnSMN2AP+39Ckmn92stwABZynq1JyzdT"
      crossorigin="anonymous"
    ></script>
    <style>
      :root {
        --bg: #faf9f5;
        --surface: #ffffff;
        --border: #e8e6dc;
        --text: #141413;
        --text-muted: #b0aea5;
        --accent: #d97757;
        --accent-hover: #c4613f;
        --green: #788c5d;
        --green-bg: #eef2e8;
        --red: #c44;
        --red-bg: #fceaea;
        --header-bg: #141413;
        --header-text: #faf9f5;
        --radius: 6px;
      }

      * {
        box-sizing: border-box;
        margin: 0;
        padding: 0;
      }

      body {
        font-family: "Lora", Georgia, serif;
        background: var(--bg);
        color: var(--text);
        height: 100vh;
        display: flex;
        flex-direction: column;
      }

      /* ---- Header ---- */
      .header {
        background: var(--header-bg);
        color: var(--header-text);
        padding: 1rem 2rem;
        display: flex;
        justify-content: space-between;
        align-items: center;
        flex-shrink: 0;
      }
      .header h1 {
        font-family: "Poppins", sans-serif;
        font-size: 1.25rem;
        font-weight: 600;
      }
      .header .instructions {
        font-size: 0.8rem;
        opacity: 0.7;
        margin-top: 0.25rem;
      }
      .header .progress {
        font-size: 0.875rem;
        opacity: 0.8;
        text-align: right;
      }

      /* ---- Main content ---- */
      .main {
        flex: 1;
        overflow-y: auto;
        padding: 1.5rem 2rem;
        display: flex;
        flex-direction: column;
        gap: 1.25rem;
      }

      /* ---- Sections ---- */
      .section {
        background: var(--surface);
        border: 1px solid var(--border);
        border-radius: var(--radius);
        flex-shrink: 0;
      }
      .section-header {
        font-family: "Poppins", sans-serif;
        padding: 0.75rem 1rem;
        font-size: 0.75rem;
        font-weight: 500;
        text-transform: uppercase;
        letter-spacing: 0.05em;
        color: var(--text-muted);
        border-bottom: 1px solid var(--border);
        background: var(--bg);
      }
      .section-body {
        padding: 1rem;
      }

      /* ---- Config badge ---- */
      .config-badge {
        display: inline-block;
        padding: 0.2rem 0.625rem;
        border-radius: 9999px;
        font-family: "Poppins", sans-serif;
        font-size: 0.6875rem;
        font-weight: 600;
        text-transform: uppercase;
        letter-spacing: 0.03em;
        margin-left: 0.75rem;
        vertical-align: middle;
      }
      .config-badge.config-primary {
        background: rgba(33, 150, 243, 0.12);
        color: #1976d2;
      }
      .config-badge.config-baseline {
        background: rgba(255, 193, 7, 0.15);
        color: #f57f17;
      }

      /* ---- Prompt ---- */
      .prompt-text {
        white-space: pre-wrap;
        font-size: 0.9375rem;
        line-height: 1.6;
      }

      /* ---- Outputs ---- */
      .output-file {
        border: 1px solid var(--border);
        border-radius: var(--radius);
        overflow: hidden;
      }
      .output-file + .output-file {
        margin-top: 1rem;
      }
      .output-file-header {
        padding: 0.5rem 0.75rem;
        font-size: 0.8rem;
        font-weight: 600;
        color: var(--text-muted);
        background: var(--bg);
        border-bottom: 1px solid var(--border);
        font-family:
          "SF Mono", SFMono-Regular, Consolas, "Liberation Mono", Menlo,
          monospace;
        display: flex;
        justify-content: space-between;
        align-items: center;
      }
      .output-file-header .dl-btn {
        font-size: 0.7rem;
        color: var(--accent);
        text-decoration: none;
        cursor: pointer;
        font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
        font-weight: 500;
        opacity: 0.8;
      }
      .output-file-header .dl-btn:hover {
        opacity: 1;
        text-decoration: underline;
      }
      .output-file-content {
        padding: 0.75rem;
        overflow-x: auto;
      }
      .output-file-content pre {
        font-size: 0.8125rem;
        line-height: 1.5;
        white-space: pre-wrap;
        word-break: break-word;
        font-family:
          "SF Mono", SFMono-Regular, Consolas, "Liberation Mono", Menlo,
          monospace;
      }
      .output-file-content img {
        max-width: 100%;
        height: auto;
        border-radius: 4px;
      }
      .output-file-content iframe {
        width: 100%;
        height: 600px;
        border: none;
      }
      .output-file-content table {
        border-collapse: collapse;
        font-size: 0.8125rem;
        width: 100%;
      }
      .output-file-content table td,
      .output-file-content table th {
        border: 1px solid var(--border);
        padding: 0.375rem 0.5rem;
        text-align: left;
      }
      .output-file-content table th {
        background: var(--bg);
        font-weight: 600;
      }
      .output-file-content .download-link {
        display: inline-flex;
        align-items: center;
        gap: 0.5rem;
        padding: 0.5rem 1rem;
        background: var(--bg);
        border: 1px solid var(--border);
        border-radius: 4px;
        color: var(--accent);
        text-decoration: none;
        font-size: 0.875rem;
        cursor: pointer;
      }
      .output-file-content .download-link:hover {
        background: var(--border);
      }
      .empty-state {
        color: var(--text-muted);
        font-style: italic;
        padding: 2rem;
        text-align: center;
      }

      /* ---- Feedback ---- */
      .prev-feedback {
        background: var(--bg);
        border: 1px solid var(--border);
        border-radius: 4px;
        padding: 0.625rem 0.75rem;
        margin-top: 0.75rem;
        font-size: 0.8125rem;
        color: var(--text-muted);
        line-height: 1.5;
      }
      .prev-feedback-label {
        font-size: 0.7rem;
        font-weight: 600;
        text-transform: uppercase;
        letter-spacing: 0.04em;
        margin-bottom: 0.25rem;
        color: var(--text-muted);
      }
      .feedback-textarea {
        width: 100%;
        min-height: 100px;
        padding: 0.75rem;
        border: 1px solid var(--border);
        border-radius: 4px;
        font-family: inherit;
        font-size: 0.9375rem;
        line-height: 1.5;
        resize: vertical;
        color: var(--text);
      }
      .feedback-textarea:focus {
        outline: none;
        border-color: var(--accent);
        box-shadow: 0 0 0 3px rgba(37, 99, 235, 0.1);
      }
      .feedback-status {
        font-size: 0.75rem;
        color: var(--text-muted);
        margin-top: 0.5rem;
        min-height: 1.1em;
      }

      /* ---- Grades (collapsible) ---- */
      .grades-toggle {
        display: flex;
        align-items: center;
        cursor: pointer;
        user-select: none;
      }
      .grades-toggle:hover {
        color: var(--accent);
      }
      .grades-toggle .arrow {
        margin-right: 0.5rem;
        transition: transform 0.15s;
        font-size: 0.75rem;
      }
      .grades-toggle .arrow.open {
        transform: rotate(90deg);
      }
      .grades-content {
        display: none;
        margin-top: 0.75rem;
      }
      .grades-content.open {
        display: block;
      }
      .grades-summary {
        font-size: 0.875rem;
        margin-bottom: 0.75rem;
        display: flex;
        align-items: center;
        gap: 0.5rem;
      }
      .grade-badge {
        display: inline-block;
        padding: 0.125rem 0.5rem;
        border-radius: 9999px;
        font-size: 0.75rem;
        font-weight: 600;
      }
      .grade-pass {
        background: var(--green-bg);
        color: var(--green);
      }
      .grade-fail {
        background: var(--red-bg);
        color: var(--red);
      }
      .assertion-list {
        list-style: none;
      }
      .assertion-item {
        padding: 0.625rem 0;
        border-bottom: 1px solid var(--border);
        font-size: 0.8125rem;
      }
      .assertion-item:last-child {
        border-bottom: none;
      }
      .assertion-status {
        font-weight: 600;
        margin-right: 0.5rem;
      }
      .assertion-status.pass {
        color: var(--green);
      }
      .assertion-status.fail {
        color: var(--red);
      }
      .assertion-evidence {
        color: var(--text-muted);
        font-size: 0.75rem;
        margin-top: 0.25rem;
        padding-left: 1.5rem;
      }

      /* ---- View tabs ---- */
      .view-tabs {
        display: flex;
        gap: 0;
        padding: 0 2rem;
        background: var(--bg);
        border-bottom: 1px solid var(--border);
        flex-shrink: 0;
      }
      .view-tab {
        font-family: "Poppins", sans-serif;
        padding: 0.625rem 1.25rem;
        font-size: 0.8125rem;
        font-weight: 500;
        cursor: pointer;
        border: none;
        background: none;
        color: var(--text-muted);
        border-bottom: 2px solid transparent;
        transition: all 0.15s;
      }
      .view-tab:hover {
        color: var(--text);
      }
      .view-tab.active {
        color: var(--accent);
        border-bottom-color: var(--accent);
      }
      .view-panel {
        display: none;
      }
      .view-panel.active {
        display: flex;
        flex-direction: column;
        flex: 1;
        overflow: hidden;
      }

      /* ---- Benchmark view ---- */
      .benchmark-view {
        padding: 1.5rem 2rem;
        overflow-y: auto;
        flex: 1;
      }
      .benchmark-table {
        border-collapse: collapse;
        background: var(--surface);
        border: 1px solid var(--border);
        border-radius: var(--radius);
        font-size: 0.8125rem;
        width: 100%;
        margin-bottom: 1.5rem;
      }
      .benchmark-table th,
      .benchmark-table td {
        padding: 0.625rem 0.75rem;
        text-align: left;
        border: 1px solid var(--border);
      }
      .benchmark-table th {
        font-family: "Poppins", sans-serif;
        background: var(--header-bg);
        color: var(--header-text);
        font-weight: 500;
        font-size: 0.75rem;
        text-transform: uppercase;
        letter-spacing: 0.04em;
      }
      .benchmark-table tr:hover {
        background: var(--bg);
      }
      .benchmark-table tr.benchmark-row-with {
        background: rgba(33, 150, 243, 0.06);
      }
      .benchmark-table tr.benchmark-row-without {
        background: rgba(255, 193, 7, 0.06);
      }
      .benchmark-table tr.benchmark-row-with:hover {
        background: rgba(33, 150, 243, 0.12);
      }
      .benchmark-table tr.benchmark-row-without:hover {
        background: rgba(255, 193, 7, 0.12);
      }
      .benchmark-table tr.benchmark-row-avg {
        font-weight: 600;
        border-top: 2px solid var(--border);
      }
      .benchmark-table tr.benchmark-row-avg.benchmark-row-with {
        background: rgba(33, 150, 243, 0.12);
      }
      .benchmark-table tr.benchmark-row-avg.benchmark-row-without {
        background: rgba(255, 193, 7, 0.12);
      }
      .benchmark-delta-positive {
        color: var(--green);
        font-weight: 600;
      }
      .benchmark-delta-negative {
        color: var(--red);
        font-weight: 600;
      }
      .benchmark-notes {
        background: var(--surface);
        border: 1px solid var(--border);
        border-radius: var(--radius);
        padding: 1rem;
      }
      .benchmark-notes h3 {
        font-family: "Poppins", sans-serif;
        font-size: 0.875rem;
        margin-bottom: 0.75rem;
      }
      .benchmark-notes ul {
        list-style: disc;
        padding-left: 1.25rem;
      }
      .benchmark-notes li {
        font-size: 0.8125rem;
        line-height: 1.6;
        margin-bottom: 0.375rem;
      }
      .benchmark-empty {
        color: var(--text-muted);
        font-style: italic;
        text-align: center;
        padding: 3rem;
      }

      /* ---- Navigation ---- */
      .nav {
        display: flex;
        justify-content: space-between;
        align-items: center;
        padding: 1rem 2rem;
        border-top: 1px solid var(--border);
        background: var(--surface);
        flex-shrink: 0;
      }
      .nav-btn {
        font-family: "Poppins", sans-serif;
        padding: 0.5rem 1.25rem;
        border: 1px solid var(--border);
        border-radius: var(--radius);
        background: var(--surface);
        cursor: pointer;
        font-size: 0.875rem;
        font-weight: 500;
        color: var(--text);
        transition: all 0.15s;
      }
      .nav-btn:hover:not(:disabled) {
        background: var(--bg);
        border-color: var(--text-muted);
      }
      .nav-btn:disabled {
        opacity: 0.4;
        cursor: not-allowed;
      }
      .done-btn {
        font-family: "Poppins", sans-serif;
        padding: 0.5rem 1.5rem;
        border: 1px solid var(--border);
        border-radius: var(--radius);
        background: var(--surface);
        color: var(--text);
        cursor: pointer;
        font-size: 0.875rem;
        font-weight: 500;
        transition: all 0.15s;
      }
      .done-btn:hover {
        background: var(--bg);
        border-color: var(--text-muted);
      }
      .done-btn.ready {
        border: none;
        background: var(--accent);
        color: white;
        font-weight: 600;
      }
      .done-btn.ready:hover {
        background: var(--accent-hover);
      }
      /* ---- Done overlay ---- */
      .done-overlay {
        display: none;
        position: fixed;
        inset: 0;
        background: rgba(0, 0, 0, 0.5);
        z-index: 100;
        justify-content: center;
        align-items: center;
      }
      .done-overlay.visible {
        display: flex;
      }
      .done-card {
        background: var(--surface);
        border-radius: 12px;
        padding: 2rem 3rem;
        text-align: center;
        box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
        max-width: 500px;
      }
      .done-card h2 {
        font-size: 1.5rem;
        margin-bottom: 0.5rem;
      }
      .done-card p {
        color: var(--text-muted);
        margin-bottom: 1.5rem;
        line-height: 1.5;
      }
      .done-card .btn-row {
        display: flex;
        gap: 0.5rem;
        justify-content: center;
      }
      .done-card button {
        padding: 0.5rem 1.25rem;
        border: 1px solid var(--border);
        border-radius: var(--radius);
        background: var(--surface);
        cursor: pointer;
        font-size: 0.875rem;
      }
      .done-card button:hover {
        background: var(--bg);
      }
      /* ---- Toast ---- */
      .toast {
        position: fixed;
        bottom: 5rem;
        left: 50%;
        transform: translateX(-50%);
        background: var(--header-bg);
        color: var(--header-text);
        padding: 0.625rem 1.25rem;
        border-radius: var(--radius);
        font-size: 0.875rem;
        opacity: 0;
        transition: opacity 0.3s;
        pointer-events: none;
        z-index: 200;
      }
      .toast.visible {
        opacity: 1;
      }
    </style>
  </head>
  <body>
    <div id="app" style="height: 100vh; display: flex; flex-direction: column">
      <div class="header">
        <div>
          <h1>Eval Review: <span id="skill-name"></span></h1>
          <div class="instructions">
            Review each output and leave feedback below. Navigate with arrow
            keys or buttons. When done, copy feedback and paste into Claude
            Code.
          </div>
        </div>
        <div class="progress" id="progress"></div>
      </div>

      <!-- View tabs (only shown when benchmark data exists) -->
      <div class="view-tabs" id="view-tabs" style="display: none">
        <button class="view-tab active" onclick="switchView('outputs')">
          Outputs
        </button>
        <button class="view-tab" onclick="switchView('benchmark')">
          Benchmark
        </button>
      </div>

      <!-- Outputs panel (qualitative review) -->
      <div class="view-panel active" id="panel-outputs">
        <div class="main">
          <!-- Prompt -->
          <div class="section">
            <div class="section-header">
              Prompt
              <span
                class="config-badge"
                id="config-badge"
                style="display: none"
              ></span>
            </div>
            <div class="section-body">
              <div class="prompt-text" id="prompt-text"></div>
            </div>
          </div>

          <!-- Outputs -->
          <div class="section">
            <div class="section-header">Output</div>
            <div class="section-body" id="outputs-body">
              <div class="empty-state">No output files found</div>
            </div>
          </div>

          <!-- Previous Output (collapsible) -->
          <div class="section" id="prev-outputs-section" style="display: none">
            <div class="section-header">
              <div class="grades-toggle" onclick="togglePrevOutputs()">
                <span class="arrow" id="prev-outputs-arrow">&#9654;</span>
                Previous Output
              </div>
            </div>
            <div class="grades-content" id="prev-outputs-content"></div>
          </div>

          <!-- Grades (collapsible) -->
          <div class="section" id="grades-section" style="display: none">
            <div class="section-header">
              <div class="grades-toggle" onclick="toggleGrades()">
                <span class="arrow" id="grades-arrow">&#9654;</span>
                Formal Grades
              </div>
            </div>
            <div class="grades-content" id="grades-content"></div>
          </div>

          <!-- Feedback -->
          <div class="section">
            <div class="section-header">Your Feedback</div>
            <div class="section-body">
              <textarea
                class="feedback-textarea"
                id="feedback"
                placeholder="What do you think of this output? Any issues, suggestions, or things that look great?"
              ></textarea>
              <div class="feedback-status" id="feedback-status"></div>
              <div
                class="prev-feedback"
                id="prev-feedback"
                style="display: none"
              >
                <div class="prev-feedback-label">Previous feedback</div>
                <div id="prev-feedback-text"></div>
              </div>
            </div>
          </div>
        </div>

        <div class="nav" id="outputs-nav">
          <button class="nav-btn" id="prev-btn" onclick="navigate(-1)">
            &#8592; Previous
          </button>
          <button class="done-btn" id="done-btn" onclick="showDoneDialog()">
            Submit All Reviews
          </button>
          <button class="nav-btn" id="next-btn" onclick="navigate(1)">
            Next &#8594;
          </button>
        </div>
      </div>
      <!-- end panel-outputs -->

      <!-- Benchmark panel (quantitative stats) -->
      <div class="view-panel" id="panel-benchmark">
        <div class="benchmark-view" id="benchmark-content">
          <div class="benchmark-empty">
            No benchmark data available. Run a benchmark to see quantitative
            results here.
          </div>
        </div>
      </div>
    </div>

    <!-- Done overlay -->
    <div class="done-overlay" id="done-overlay">
      <div class="done-card">
        <h2>Review Complete</h2>
        <p>
          Your feedback has been saved. Go back to your Claude Code session and
          tell Claude you're done reviewing.
        </p>
        <div class="btn-row">
          <button onclick="closeDoneDialog()">OK</button>
        </div>
      </div>
    </div>

    <!-- Toast -->
    <div class="toast" id="toast"></div>

    <script>
      // ---- Embedded data (injected by generate_review.py) ----
      /*__EMBEDDED_DATA__*/

      // ---- State ----
      let feedbackMap = {}; // run_id -> feedback text
      let currentIndex = 0;
      let visitedRuns = new Set();

      // ---- Init ----
      async function init() {
        // Load saved feedback from server — but only if this isn't a fresh
        // iteration (indicated by previous_feedback being present). When
        // previous feedback exists, the feedback.json on disk is stale from
        // the prior iteration and should not pre-fill the textareas.
        const hasPrevious =
          Object.keys(EMBEDDED_DATA.previous_feedback || {}).length > 0 ||
          Object.keys(EMBEDDED_DATA.previous_outputs || {}).length > 0;
        if (!hasPrevious) {
          try {
            const resp = await fetch("/api/feedback");
            const data = await resp.json();
            if (data.reviews) {
              for (const r of data.reviews) feedbackMap[r.run_id] = r.feedback;
            }
          } catch {
            /* first run, no feedback yet */
          }
        }

        document.getElementById("skill-name").textContent =
          EMBEDDED_DATA.skill_name;
        showRun(0);

        // Wire up feedback auto-save
        const textarea = document.getElementById("feedback");
        let saveTimeout = null;
        textarea.addEventListener("input", () => {
          clearTimeout(saveTimeout);
          document.getElementById("feedback-status").textContent = "";
          saveTimeout = setTimeout(() => saveCurrentFeedback(), 800);
        });
      }

      // ---- Navigation ----
      function navigate(delta) {
        const newIndex = currentIndex + delta;
        if (newIndex >= 0 && newIndex < EMBEDDED_DATA.runs.length) {
          saveCurrentFeedback();
          showRun(newIndex);
        }
      }

      function updateNavButtons() {
        document.getElementById("prev-btn").disabled = currentIndex === 0;
        document.getElementById("next-btn").disabled =
          currentIndex === EMBEDDED_DATA.runs.length - 1;
      }

      // ---- Show a run ----
      function showRun(index) {
        currentIndex = index;
        const run = EMBEDDED_DATA.runs[index];

        // Progress
        document.getElementById("progress").textContent =
          `${index + 1} of ${EMBEDDED_DATA.runs.length}`;

        // Prompt
        document.getElementById("prompt-text").textContent = run.prompt;

        // Config badge
        const badge = document.getElementById("config-badge");
        const configMatch = run.id.match(
          /(with_skill|without_skill|new_skill|old_skill)/,
        );
        if (configMatch) {
          const config = configMatch[1];
          const isBaseline =
            config === "without_skill" || config === "old_skill";
          badge.textContent = config.replace(/_/g, " ");
          badge.className =
            "config-badge " +
            (isBaseline ? "config-baseline" : "config-primary");
          badge.style.display = "inline-block";
        } else {
          badge.style.display = "none";
        }

        // Outputs
        renderOutputs(run);

        // Previous outputs
        renderPrevOutputs(run);

        // Grades
        renderGrades(run);

        // Previous feedback
        const prevFb = (EMBEDDED_DATA.previous_feedback || {})[run.id];
        const prevEl = document.getElementById("prev-feedback");
        if (prevFb) {
          document.getElementById("prev-feedback-text").textContent = prevFb;
          prevEl.style.display = "block";
        } else {
          prevEl.style.display = "none";
        }

        // Feedback
        document.getElementById("feedback").value = feedbackMap[run.id] || "";
        document.getElementById("feedback-status").textContent = "";

        updateNavButtons();

        // Track visited runs and promote done button when all visited
        visitedRuns.add(index);
        const doneBtn = document.getElementById("done-btn");
        if (visitedRuns.size >= EMBEDDED_DATA.runs.length) {
          doneBtn.classList.add("ready");
        }

        // Scroll main content to top
        document.querySelector(".main").scrollTop = 0;
      }

      // ---- Render outputs ----
      function renderOutputs(run) {
        const container = document.getElementById("outputs-body");
        container.innerHTML = "";

        const outputs = run.outputs || [];
        if (outputs.length === 0) {
          container.innerHTML =
            '<div class="empty-state">No output files</div>';
          return;
        }

        for (const file of outputs) {
          const fileDiv = document.createElement("div");
          fileDiv.className = "output-file";

          // Always show file header with download link
          const header = document.createElement("div");
          header.className = "output-file-header";
          const nameSpan = document.createElement("span");
          nameSpan.textContent = file.name;
          header.appendChild(nameSpan);
          const dlBtn = document.createElement("a");
          dlBtn.className = "dl-btn";
          dlBtn.textContent = "Download";
          dlBtn.download = file.name;
          dlBtn.href = getDownloadUri(file);
          header.appendChild(dlBtn);
          fileDiv.appendChild(header);

          const content = document.createElement("div");
          content.className = "output-file-content";

          if (file.type === "text") {
            const pre = document.createElement("pre");
            pre.textContent = file.content;
            content.appendChild(pre);
          } else if (file.type === "image") {
            const img = document.createElement("img");
            img.src = file.data_uri;
            img.alt = file.name;
            content.appendChild(img);
          } else if (file.type === "pdf") {
            const iframe = document.createElement("iframe");
            iframe.src = file.data_uri;
            content.appendChild(iframe);
          } else if (file.type === "xlsx") {
            renderXlsx(content, file.data_b64);
          } else if (file.type === "binary") {
            const a = document.createElement("a");
            a.className = "download-link";
            a.href = file.data_uri;
            a.download = file.name;
            a.textContent = "Download " + file.name;
            content.appendChild(a);
          } else if (file.type === "error") {
            const pre = document.createElement("pre");
            pre.textContent = file.content;
            pre.style.color = "var(--red)";
            content.appendChild(pre);
          }

          fileDiv.appendChild(content);
          container.appendChild(fileDiv);
        }
      }

      // ---- XLSX rendering via SheetJS ----
      function renderXlsx(container, b64Data) {
        try {
          const raw = Uint8Array.from(atob(b64Data), (c) => c.charCodeAt(0));
          const wb = XLSX.read(raw, { type: "array" });

          for (let i = 0; i < wb.SheetNames.length; i++) {
            const sheetName = wb.SheetNames[i];
            const ws = wb.Sheets[sheetName];

            if (wb.SheetNames.length > 1) {
              const sheetLabel = document.createElement("div");
              sheetLabel.style.cssText =
                "font-weight:600; font-size:0.8rem; color:#b0aea5; margin-top:0.5rem; margin-bottom:0.25rem;";
              sheetLabel.textContent = "Sheet: " + sheetName;
              container.appendChild(sheetLabel);
            }

            const htmlStr = XLSX.utils.sheet_to_html(ws, { editable: false });
            const wrapper = document.createElement("div");
            wrapper.innerHTML = htmlStr;
            container.appendChild(wrapper);
          }
        } catch (err) {
          container.textContent = "Error rendering spreadsheet: " + err.message;
        }
      }

      // ---- Grades ----
      function renderGrades(run) {
        const section = document.getElementById("grades-section");
        const content = document.getElementById("grades-content");

        if (!run.grading) {
          section.style.display = "none";
          return;
        }

        const grading = run.grading;
        section.style.display = "block";
        // Reset to collapsed
        content.classList.remove("open");
        document.getElementById("grades-arrow").classList.remove("open");

        const summary = grading.summary || {};
        const expectations = grading.expectations || [];

        let html = '<div style="padding: 1rem;">';

        // Summary line
        const passRate =
          summary.pass_rate != null
            ? Math.round(summary.pass_rate * 100) + "%"
            : "?";
        const badgeClass =
          summary.pass_rate >= 0.8
            ? "grade-pass"
            : summary.pass_rate >= 0.5
              ? ""
              : "grade-fail";
        html += '<div class="grades-summary">';
        html +=
          '<span class="grade-badge ' +
          badgeClass +
          '">' +
          passRate +
          "</span>";
        html +=
          "<span>" +
          (summary.passed || 0) +
          " passed, " +
          (summary.failed || 0) +
          " failed of " +
          (summary.total || 0) +
          "</span>";
        html += "</div>";

        // Assertions list
        html += '<ul class="assertion-list">';
        for (const exp of expectations) {
          const statusClass = exp.passed ? "pass" : "fail";
          const statusIcon = exp.passed ? "\u2713" : "\u2717";
          html += '<li class="assertion-item">';
          html +=
            '<span class="assertion-status ' +
            statusClass +
            '">' +
            statusIcon +
            "</span>";
          html += "<span>" + escapeHtml(exp.text) + "</span>";
          if (exp.evidence) {
            html +=
              '<div class="assertion-evidence">' +
              escapeHtml(exp.evidence) +
              "</div>";
          }
          html += "</li>";
        }
        html += "</ul>";

        html += "</div>";
        content.innerHTML = html;
      }

      function toggleGrades() {
        const content = document.getElementById("grades-content");
        const arrow = document.getElementById("grades-arrow");
        content.classList.toggle("open");
        arrow.classList.toggle("open");
      }

      // ---- Previous outputs (collapsible) ----
      function renderPrevOutputs(run) {
        const section = document.getElementById("prev-outputs-section");
        const content = document.getElementById("prev-outputs-content");
        const prevOutputs = (EMBEDDED_DATA.previous_outputs || {})[run.id];

        if (!prevOutputs || prevOutputs.length === 0) {
          section.style.display = "none";
          return;
        }

        section.style.display = "block";
        // Reset to collapsed
        content.classList.remove("open");
        document.getElementById("prev-outputs-arrow").classList.remove("open");

        // Render the files into the content area
        content.innerHTML = "";
        const wrapper = document.createElement("div");
        wrapper.style.padding = "1rem";

        for (const file of prevOutputs) {
          const fileDiv = document.createElement("div");
          fileDiv.className = "output-file";

          const header = document.createElement("div");
          header.className = "output-file-header";
          const nameSpan = document.createElement("span");
          nameSpan.textContent = file.name;
          header.appendChild(nameSpan);
          const dlBtn = document.createElement("a");
          dlBtn.className = "dl-btn";
          dlBtn.textContent = "Download";
          dlBtn.download = file.name;
          dlBtn.href = getDownloadUri(file);
          header.appendChild(dlBtn);
          fileDiv.appendChild(header);

          const fc = document.createElement("div");
          fc.className = "output-file-content";

          if (file.type === "text") {
            const pre = document.createElement("pre");
            pre.textContent = file.content;
            fc.appendChild(pre);
          } else if (file.type === "image") {
            const img = document.createElement("img");
            img.src = file.data_uri;
            img.alt = file.name;
            fc.appendChild(img);
          } else if (file.type === "pdf") {
            const iframe = document.createElement("iframe");
            iframe.src = file.data_uri;
            fc.appendChild(iframe);
          } else if (file.type === "xlsx") {
            renderXlsx(fc, file.data_b64);
          } else if (file.type === "binary") {
            const a = document.createElement("a");
            a.className = "download-link";
            a.href = file.data_uri;
            a.download = file.name;
            a.textContent = "Download " + file.name;
            fc.appendChild(a);
          }

          fileDiv.appendChild(fc);
          wrapper.appendChild(fileDiv);
        }

        content.appendChild(wrapper);
      }

      function togglePrevOutputs() {
        const content = document.getElementById("prev-outputs-content");
        const arrow = document.getElementById("prev-outputs-arrow");
        content.classList.toggle("open");
        arrow.classList.toggle("open");
      }

      // ---- Feedback (saved to server -> feedback.json) ----
      function saveCurrentFeedback() {
        const run = EMBEDDED_DATA.runs[currentIndex];
        const text = document.getElementById("feedback").value;

        if (text.trim() === "") {
          delete feedbackMap[run.id];
        } else {
          feedbackMap[run.id] = text;
        }

        // Build reviews array from map
        const reviews = [];
        for (const [run_id, feedback] of Object.entries(feedbackMap)) {
          if (feedback.trim()) {
            reviews.push({
              run_id,
              feedback,
              timestamp: new Date().toISOString(),
            });
          }
        }

        fetch("/api/feedback", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ reviews, status: "in_progress" }),
        })
          .then(() => {
            document.getElementById("feedback-status").textContent = "Saved";
          })
          .catch(() => {
            // Static mode or server unavailable — no-op on auto-save,
            // feedback will be downloaded on final submit
            document.getElementById("feedback-status").textContent =
              "Will download on submit";
          });
      }

      // ---- Done ----
      function showDoneDialog() {
        // Save current textarea to feedbackMap (but don't POST yet)
        const run = EMBEDDED_DATA.runs[currentIndex];
        const text = document.getElementById("feedback").value;
        if (text.trim() === "") {
          delete feedbackMap[run.id];
        } else {
          feedbackMap[run.id] = text;
        }

        // POST once with status: complete — include ALL runs so the model
        // can distinguish "no feedback" (looks good) from "not reviewed"
        const reviews = [];
        const ts = new Date().toISOString();
        for (const r of EMBEDDED_DATA.runs) {
          reviews.push({
            run_id: r.id,
            feedback: feedbackMap[r.id] || "",
            timestamp: ts,
          });
        }
        const payload = JSON.stringify(
          { reviews, status: "complete" },
          null,
          2,
        );
        fetch("/api/feedback", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: payload,
        })
          .then(() => {
            document.getElementById("done-overlay").classList.add("visible");
          })
          .catch(() => {
            // Server not available (static mode) — download as file
            const blob = new Blob([payload], { type: "application/json" });
            const url = URL.createObjectURL(blob);
            const a = document.createElement("a");
            a.href = url;
            a.download = "feedback.json";
            a.click();
            URL.revokeObjectURL(url);
            document.getElementById("done-overlay").classList.add("visible");
          });
      }

      function closeDoneDialog() {
        // Reset status back to in_progress
        saveCurrentFeedback();
        document.getElementById("done-overlay").classList.remove("visible");
      }

      // ---- Toast ----
      function showToast(message) {
        const toast = document.getElementById("toast");
        toast.textContent = message;
        toast.classList.add("visible");
        setTimeout(() => toast.classList.remove("visible"), 2000);
      }

      // ---- Keyboard nav ----
      document.addEventListener("keydown", (e) => {
        // Don't capture when typing in textarea
        if (e.target.tagName === "TEXTAREA") return;

        if (e.key === "ArrowLeft" || e.key === "ArrowUp") {
          e.preventDefault();
          navigate(-1);
        } else if (e.key === "ArrowRight" || e.key === "ArrowDown") {
          e.preventDefault();
          navigate(1);
        }
      });

      // ---- Util ----
      function getDownloadUri(file) {
        if (file.data_uri) return file.data_uri;
        if (file.data_b64)
          return "data:application/octet-stream;base64," + file.data_b64;
        if (file.type === "text")
          return (
            "data:text/plain;charset=utf-8," + encodeURIComponent(file.content)
          );
        return "#";
      }

      function escapeHtml(text) {
        const div = document.createElement("div");
        div.textContent = text;
        return div.innerHTML;
      }

      // ---- View switching ----
      function switchView(view) {
        document
          .querySelectorAll(".view-tab")
          .forEach((t) => t.classList.remove("active"));
        document
          .querySelectorAll(".view-panel")
          .forEach((p) => p.classList.remove("active"));
        document
          .querySelector(`[onclick="switchView('${view}')"]`)
          .classList.add("active");
        document.getElementById("panel-" + view).classList.add("active");
      }

      // ---- Benchmark rendering ----
      function renderBenchmark() {
        const data = EMBEDDED_DATA.benchmark;
        if (!data) return;

        // Show the tabs
        document.getElementById("view-tabs").style.display = "flex";

        const container = document.getElementById("benchmark-content");
        const summary = data.run_summary || {};
        const metadata = data.metadata || {};
        const notes = data.notes || [];

        let html = "";

        // Header
        html +=
          "<h2 style='font-family: Poppins, sans-serif; margin-bottom: 0.5rem;'>Benchmark Results</h2>";
        html +=
          "<p style='color: var(--text-muted); font-size: 0.875rem; margin-bottom: 1.25rem;'>";
        if (metadata.skill_name)
          html +=
            "<strong>" + escapeHtml(metadata.skill_name) + "</strong> &mdash; ";
        if (metadata.timestamp) html += metadata.timestamp + " &mdash; ";
        if (metadata.evals_run)
          html += "Evals: " + metadata.evals_run.join(", ") + " &mdash; ";
        html +=
          (metadata.runs_per_configuration || "?") + " runs per configuration";
        html += "</p>";

        // Summary table
        html += '<table class="benchmark-table">';

        function fmtStat(stat, pct) {
          if (!stat) return "\u2014";
          const suffix = pct ? "%" : "";
          const m = pct ? (stat.mean * 100).toFixed(0) : stat.mean.toFixed(1);
          const s = pct
            ? (stat.stddev * 100).toFixed(0)
            : stat.stddev.toFixed(1);
          return m + suffix + " \u00b1 " + s + suffix;
        }

        function deltaClass(val) {
          if (!val) return "";
          const n = parseFloat(val);
          if (n > 0) return "benchmark-delta-positive";
          if (n < 0) return "benchmark-delta-negative";
          return "";
        }

        // Discover config names dynamically (everything except "delta")
        const configs = Object.keys(summary).filter((k) => k !== "delta");
        const configA = configs[0] || "config_a";
        const configB = configs[1] || "config_b";
        const labelA = configA
          .replace(/_/g, " ")
          .replace(/\b\w/g, (c) => c.toUpperCase());
        const labelB = configB
          .replace(/_/g, " ")
          .replace(/\b\w/g, (c) => c.toUpperCase());
        const a = summary[configA] || {};
        const b = summary[configB] || {};
        const delta = summary.delta || {};

        html +=
          "<thead><tr><th>Metric</th><th>" +
          escapeHtml(labelA) +
          "</th><th>" +
          escapeHtml(labelB) +
          "</th><th>Delta</th></tr></thead>";
        html += "<tbody>";

        html += "<tr><td><strong>Pass Rate</strong></td>";
        html += "<td>" + fmtStat(a.pass_rate, true) + "</td>";
        html += "<td>" + fmtStat(b.pass_rate, true) + "</td>";
        html +=
          '<td class="' +
          deltaClass(delta.pass_rate) +
          '">' +
          (delta.pass_rate || "\u2014") +
          "</td></tr>";

        // Time (only show row if data exists)
        if (a.time_seconds || b.time_seconds) {
          html += "<tr><td><strong>Time (s)</strong></td>";
          html += "<td>" + fmtStat(a.time_seconds, false) + "</td>";
          html += "<td>" + fmtStat(b.time_seconds, false) + "</td>";
          html +=
            '<td class="' +
            deltaClass(delta.time_seconds) +
            '">' +
            (delta.time_seconds ? delta.time_seconds + "s" : "\u2014") +
            "</td></tr>";
        }

        // Tokens (only show row if data exists)
        if (a.tokens || b.tokens) {
          html += "<tr><td><strong>Tokens</strong></td>";
          html += "<td>" + fmtStat(a.tokens, false) + "</td>";
          html += "<td>" + fmtStat(b.tokens, false) + "</td>";
          html +=
            '<td class="' +
            deltaClass(delta.tokens) +
            '">' +
            (delta.tokens || "\u2014") +
            "</td></tr>";
        }

        html += "</tbody></table>";

        // Per-eval breakdown (if runs data available)
        const runs = data.runs || [];
        if (runs.length > 0) {
          const evalIds = [...new Set(runs.map((r) => r.eval_id))].sort(
            (a, b) => a - b,
          );

          html +=
            "<h3 style='font-family: Poppins, sans-serif; margin-bottom: 0.75rem;'>Per-Eval Breakdown</h3>";

          const hasTime = runs.some(
            (r) => r.result && r.result.time_seconds != null,
          );
          const hasErrors = runs.some((r) => r.result && r.result.errors > 0);

          for (const evalId of evalIds) {
            const evalRuns = runs.filter((r) => r.eval_id === evalId);
            const evalName =
              evalRuns[0] && evalRuns[0].eval_name
                ? evalRuns[0].eval_name
                : "Eval " + evalId;

            html +=
              "<h4 style='font-family: Poppins, sans-serif; margin: 1rem 0 0.5rem; color: var(--text);'>" +
              escapeHtml(evalName) +
              "</h4>";
            html += '<table class="benchmark-table">';
            html += "<thead><tr><th>Config</th><th>Run</th><th>Pass Rate</th>";
            if (hasTime) html += "<th>Time (s)</th>";
            if (hasErrors) html += "<th>Crashes During Execution</th>";
            html += "</tr></thead>";
            html += "<tbody>";

            // Group by config and render with average rows
            const configGroups = [
              ...new Set(evalRuns.map((r) => r.configuration)),
            ];
            for (let ci = 0; ci < configGroups.length; ci++) {
              const config = configGroups[ci];
              const configRuns = evalRuns.filter(
                (r) => r.configuration === config,
              );
              if (configRuns.length === 0) continue;

              const rowClass =
                ci === 0 ? "benchmark-row-with" : "benchmark-row-without";
              const configLabel = config
                .replace(/_/g, " ")
                .replace(/\b\w/g, (c) => c.toUpperCase());

              for (const run of configRuns) {
                const r = run.result || {};
                const prClass =
                  r.pass_rate >= 0.8
                    ? "benchmark-delta-positive"
                    : r.pass_rate < 0.5
                      ? "benchmark-delta-negative"
                      : "";
                html += '<tr class="' + rowClass + '">';
                html += "<td>" + configLabel + "</td>";
                html += "<td>" + run.run_number + "</td>";
                html +=
                  '<td class="' +
                  prClass +
                  '">' +
                  ((r.pass_rate || 0) * 100).toFixed(0) +
                  "% (" +
                  (r.passed || 0) +
                  "/" +
                  (r.total || 0) +
                  ")</td>";
                if (hasTime)
                  html +=
                    "<td>" +
                    (r.time_seconds != null
                      ? r.time_seconds.toFixed(1)
                      : "\u2014") +
                    "</td>";
                if (hasErrors) html += "<td>" + (r.errors || 0) + "</td>";
                html += "</tr>";
              }

              // Average row
              const rates = configRuns.map(
                (r) => (r.result || {}).pass_rate || 0,
              );
              const avgRate = rates.reduce((a, b) => a + b, 0) / rates.length;
              const avgPrClass =
                avgRate >= 0.8
                  ? "benchmark-delta-positive"
                  : avgRate < 0.5
                    ? "benchmark-delta-negative"
                    : "";
              html += '<tr class="benchmark-row-avg ' + rowClass + '">';
              html += "<td>" + configLabel + "</td>";
              html += "<td>Avg</td>";
              html +=
                '<td class="' +
                avgPrClass +
                '">' +
                (avgRate * 100).toFixed(0) +
                "%</td>";
              if (hasTime) {
                const times = configRuns
                  .map((r) => (r.result || {}).time_seconds)
                  .filter((t) => t != null);
                html +=
                  "<td>" +
                  (times.length
                    ? (times.reduce((a, b) => a + b, 0) / times.length).toFixed(
                        1,
                      )
                    : "\u2014") +
                  "</td>";
              }
              if (hasErrors) html += "<td></td>";
              html += "</tr>";
            }
            html += "</tbody></table>";

            // Per-assertion detail for this eval
            const runsWithExpectations = {};
            for (const config of configGroups) {
              runsWithExpectations[config] = evalRuns.filter(
                (r) =>
                  r.configuration === config &&
                  r.expectations &&
                  r.expectations.length > 0,
              );
            }
            const hasAnyExpectations = Object.values(runsWithExpectations).some(
              (runs) => runs.length > 0,
            );
            if (hasAnyExpectations) {
              // Collect all unique assertion texts across all configs
              const allAssertions = [];
              const seen = new Set();
              for (const config of configGroups) {
                for (const run of runsWithExpectations[config]) {
                  for (const exp of run.expectations || []) {
                    if (!seen.has(exp.text)) {
                      seen.add(exp.text);
                      allAssertions.push(exp.text);
                    }
                  }
                }
              }

              html +=
                '<table class="benchmark-table" style="margin-top: 0.5rem;">';
              html += "<thead><tr><th>Assertion</th>";
              for (const config of configGroups) {
                const label = config
                  .replace(/_/g, " ")
                  .replace(/\b\w/g, (c) => c.toUpperCase());
                html += "<th>" + escapeHtml(label) + "</th>";
              }
              html += "</tr></thead><tbody>";

              for (const assertionText of allAssertions) {
                html += "<tr><td>" + escapeHtml(assertionText) + "</td>";

                for (const config of configGroups) {
                  html += "<td>";
                  for (const run of runsWithExpectations[config]) {
                    const exp = (run.expectations || []).find(
                      (e) => e.text === assertionText,
                    );
                    if (exp) {
                      const cls = exp.passed
                        ? "benchmark-delta-positive"
                        : "benchmark-delta-negative";
                      const icon = exp.passed ? "\u2713" : "\u2717";
                      html +=
                        '<span class="' +
                        cls +
                        '" title="Run ' +
                        run.run_number +
                        ": " +
                        escapeHtml(exp.evidence || "") +
                        '">' +
                        icon +
                        "</span> ";
                    } else {
                      html += "\u2014 ";
                    }
                  }
                  html += "</td>";
                }
                html += "</tr>";
              }
              html += "</tbody></table>";
            }
          }
        }

        // Notes
        if (notes.length > 0) {
          html += '<div class="benchmark-notes">';
          html += "<h3>Analysis Notes</h3>";
          html += "<ul>";
          for (const note of notes) {
            html += "<li>" + escapeHtml(note) + "</li>";
          }
          html += "</ul></div>";
        }

        container.innerHTML = html;
      }

      // ---- Start ----
      init();
      renderBenchmark();
    </script>
  </body>
</html>

Workflow Patterns

Source: The Complete Guide to Building Skills for Claude (Anthropic)

When building skills, especially those that automate multi-step processes or enhance MCP tools, structuring the workflow correctly is critical.

Problem-First vs. Tool-First Framing

Before choosing a pattern, decide on the framing:

Problem-first: User describes an outcome → skill orchestrates the right tool calls. Example: "I need to set up a project workspace"
Tool-first: User has a tool connected → skill teaches the optimal workflows. Example: "I have Notion MCP connected"

Most skills lean one direction. Choose the framing that fits your use case before writing instructions.

Core Workflow Patterns

Choose the pattern that best fits your use case.

Pattern 1: Sequential Workflow Orchestration

Use when: users need multi-step processes in a specific order. Key techniques: explicit step ordering, dependencies between steps, validation at each stage, rollback instructions.

_Tip: It is often helpful to give Claude an overview of the process towards the beginning of SKILL.md:_

Filling a PDF form involves these steps:

1. Analyze the form (run analyze_form.py)
2. Create field mapping (edit fields.json)
3. Validate mapping (run validate_fields.py)
4. Fill the form (run fill_form.py)
5. Verify output (run verify_output.py)

Pattern 2: Conditional Workflows (Context-Aware Tool Selection)

Use when: same outcome, different tools depending on context, or tasks with branching logic. Key techniques: decision tree with clear criteria, fallback options, explain the choice to user.

_Tip: Guide Claude through decision points explicitly:_

1. Determine the modification type:
   **Creating new content?** → Follow "Creation workflow" below
   **Editing existing content?** → Follow "Editing workflow" below

2. Creation workflow: [steps]
3. Editing workflow: [steps]

Pattern 3: Multi-Service Coordination

Use when: workflow spans multiple services (e.g., Figma → Drive → Linear → Slack). Key techniques: clear phase separation, data passing between services, validation before moving to next phase.

Pattern 4: Iterative Refinement

Use when: output quality improves with iteration (e.g., report generation). Key techniques: generate draft → run validation script → fix issues → re-validate → repeat until threshold met.

Pattern 5: Domain-Specific Intelligence

Use when: skill adds specialized knowledge beyond tool access (compliance, security, finance). Key techniques: domain logic before action, comprehensive audit trail, clear governance rules.

When to use `references/workflows.md`

If your skill contains complex, multi-step processes, conditional logic, or orchestrates multiple tools, you should extract these patterns into a dedicated references/workflows.md file.

This keeps SKILL.md lean (focused on triggers and high-level navigation) while providing Claude with detailed procedural knowledge when it actually needs to execute the workflow.

Related skills

Find SkillsSo their agent can intelligently recommend and install the right modular skills from the open ecosystem when they ask how to do something.3M27.9k

Lark Skill MakerRapidly create custom skills for Lark (Feishu) that can be invoked by their AI agents and automation workflows.513k

Skill CreatorGenerate, test, and iterate on new reusable agent skills directly inside their Claude or Cursor workflow.337k166k

BrainstormingTurn vague product ideas into clear, approved designs and specifications through guided collaborative dialogue before writing any code.307k265k

Writing Great SkillsCreate high-quality, reusable agent skills that other developers can discover and install from the Skillselion catalog.290k200k

Find SkillsInstantly surface the best relevant agent skills, MCP servers, and workflows across multiple registries before starting any new project.226k12

Skill Developmentagentsautomation

About

Skill Master by the numbers

Add your badge

What it does

Files

Skill Master

Links

Quick Navigation

When to Use

Skill Structure (Required)

Folder Purposes (CRITICAL)

When to Use Each

Frontmatter Schema

Required Fields

Optional Fields (Top Level)

metadata Object (Common Fields)

Optional Fields (Claude Code / Advanced)

Invocation Control Matrix

Variable Substitutions

Dynamic Context Injection

metadata.json (Optional)

Name Validation Examples

Description Rules

How Skills Work (Progressive Disclosure)

Creating a New Skill

Step 1: Scaffold

Step 2: Write Frontmatter

Step 3: Write Body

Step 4: Add References (documentation)

Step 5: Add Scripts (if applicable)

Step 6: Add Assets (if needed)

Step 7: Validate

Creating a Skill from Documentation

Phase 1: Scaffold

Phase 2: Build Queue

Phase 3: Ingest Loop

Phase 4: Finalize

Critical Prohibitions

Version Tracking

What version Means

When to Update

Validation Checklist

Scripts

Links

Post-hoc Analyzer Agent

Role

Inputs

Process

Step 1: Read Comparison Result

Step 2: Read Both Skills

Step 3: Read Both Transcripts

Step 4: Analyze Instruction Following

Step 5: Identify Winner Strengths

Step 6: Identify Loser Weaknesses

Step 7: Generate Improvement Suggestions

Step 8: Write Analysis Results

Output Format

Guidelines

Categories for Suggestions

Priority Levels

Analyzing Benchmark Results

Role

Inputs

Process

Step 1: Read Benchmark Data

Step 2: Analyze Per-Assertion Patterns

Step 3: Analyze Cross-Eval Patterns

Step 4: Analyze Metrics Patterns

Step 5: Generate Notes

Step 6: Write Notes

Guidelines

Blind Comparator Agent

Role

Inputs

Process

Step 1: Read Both Outputs

Step 2: Understand the Task

Step 3: Generate Evaluation Rubric

Step 4: Evaluate Each Output Against the Rubric

Step 5: Check Assertions (if provided)

What `version` Means

When to use `references/workflows.md`