Skill Forge
- 1 installs
- Updated August 2, 2026
- arendon1/agent-skills
skill-forge is a skill that creates, optimizes, and audits AI agent skills using Test-Driven Development loops with XML prompt-based evaluation.
About
This skill is a framework for creating, optimizing, and auditing AI agent skills. It scaffolds new skill directories, runs a test-driven-development loop to improve a skill's trigger rate, and audits a skill's structure and security using XML prompts that agents parse to spawn subagents internally. It exposes /skill-create, /skill-improve, and /skill-audit commands and enforces a required frontmatter schema and a sub-500-line SKILL.md.
- Framework to create, optimize, and audit AI agent skills
- Uses Test-Driven Development loops with XML prompt-based evaluation and no external API calls
- Ships /skill-create, /skill-improve, and /skill-audit commands with structure and security checks
Skill Forge by the numbers
- 1 all-time installs (skills.sh)
- Ranked #642 of 782 Skill Development skills by installs in the Skillselion catalog
- Data as of Aug 3, 2026 (Skillselion catalog sync)
skill-forge capabilities & compatibility
- Capabilities
- skill creation · skill audit · skill optimization
What skill-forge says it does
The definitive framework for creating, optimizing, and auditing AI agent skills. Uses Test-Driven Development loops with XML prompt-based evaluation.
Scripts output structured XML prompts that agents parse to understand evaluation tasks.
npx skills add https://github.com/arendon1/agent-skills --skill skill-forgeAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| Last updated | August 2, 2026 |
| Repository | arendon1/agent-skills ↗ |
What it does
Create, optimize, and audit AI agent skills with TDD loops and structure/security checks.
Who is it for?
Scaffolding, TDD-optimizing, and security-auditing agent skills
Skip if: Skills mixing languages (skills must be written entirely in English or Spanish es-CO)
When should I use this skill?
when creating a new skill, improving an existing skill's trigger rate, or auditing a skill for structural/security issues
What you get
A structurally valid skill directory with an optimized description that passes generated test cases
- scaffolded skill directory
- audited SKILL.md
- TDD-optimized skill description
By the numbers
- SKILL.md must stay under 500 lines
- reference folder depth max 1 level
- 3 commands: /skill-create, /skill-improve, /skill-audit
Files
skill-forge
`skill-forge` generates XML prompts that agents consume to spawn subagents internally for skill evaluation. No external API calls required.
Core Principles
1. XML Prompt Architecture: Scripts output structured XML prompts that agents parse to understand evaluation tasks. 2. Agent-Native Execution: Agents spawn subagents with different models internally to run evaluations. 3. TDD Workflow: Test cases are generated first, then skill is iteratively improved until tests pass. 4. Language Consistency: Skills must be written ENTIRELY in English (en) or Spanish (es-CO).
🚀 Self-Deployment
If skill-forge's workflows (like /skill-improve, /skill-audit) are not appearing in your agent's slash-commands, run: python scripts/deploy.py --workspace .
Command Glossary
/skill-create
Scaffold a brand new, structurally flawless skill directory.
- Behind the scenes: Runs
scripts/init.py. - Next steps: Focus on polishing the
descriptionstring. KeepSKILL.mdunder 500 lines.
/skill-improve (TDD Optimization)
Optimize a skill using Test-Driven Development.
- Behind the scenes: Runs
scripts/audit.pywhich outputs an XML prompt. - Process: Agent receives XML prompt with test cases and improvement criteria. Agent runs entire TDD loop internally, spawning subagents as needed.
- Output: Improved
SKILL.mdwith description that passes all test cases.
/skill-audit (Validation and Security)
Validate a skill's structure and security.
- Behind the scenes: Runs
scripts/audit.pywhich outputs an XML prompt. - Process: Agent spawns subagents internally to run different audit checks.
- Checks:
SKILL.mdlength (<500 lines)- Description format ("Use when..." pattern)
- Reference folder depth (max 1 level)
- Language consistency (en or es-CO)
- Security scan of
scripts/directory
Required Skill Frontmatter
Every skill created by the forge MUST use this schema at the top of SKILL.md:
---
name: [lowercase-with-hyphens]
description: >-
[What it does briefly].
Use when [Specific trigger phrases, contexts, error symptoms, file types].
---Supported Agents
skill-forge works with any agent capable of spawning subagents:
- Antigravity CLI - Google Antigravity agent harness
- OpenCode - OpenCode agent framework
- Copilot CLI - GitHub Copilot CLI agent
- Kiro CLI - Kiro custom agent framework
Agents are auto-detected by deploy.py based on available CLI tools.
# Python
__pycache__/
*.pyc
*.pyo
# Test/scratch
tmp/
Analyzer Agent Instructions
You are an analyzer agent reviewing benchmark results to surface insights that aggregate statistics might hide.
Input
You will receive:
benchmark.json- Aggregated results from multiple eval runsbenchmark.md- Human-readable summary of the benchmark- Iteration history if this is not the first iteration
Your Task
1. Surface Non-Discriminating Assertions
Find assertions that pass 100% in BOTH configurations (with_skill and without_skill). These don't differentiate skill value - they're too easy. Flag them.
2. Identify High-Variance Evals
Find evals where pass rates vary wildly across runs (e.g., 50% ± 40%). These may be:
- Flaky tests that depend on non-deterministic factors
- Model-dependent behavior
- Edge cases that need special handling
3. Analyze Time/Token Tradeoffs
Compare the delta between with_skill and without_skill configurations. Note:
- Is the skill faster/slower?
- Does it use more/fewer tokens?
- Is there a quality/speed tradeoff?
4. Identify Skill-Specific Patterns
Look for things the skill consistently helps with vs. things that don't improve:
- Does the skill reduce errors on complex tasks?
- Does the skill improve output format consistency?
- Are there specific failure modes the skill prevents?
5. Check for Regression Risks
If comparing against a previous iteration:
- Did any assertion pass rate decrease?
- Did execution time increase significantly?
- Are there new failure modes?
Output Format
Write your analysis to analysis.json in the same directory:
{
"non_discriminating_assertions": [
{
"assertion": "The output is a PDF file",
"reason": "Passes 100% in both configurations",
"recommendation": "Remove or make more specific"
}
],
"high_variance_evals": [
{
"eval_name": "Multi-page document processing",
"pass_rate": "50% ± 40%",
"possible_cause": "Different document structures trigger different code paths",
"recommendation": "Add preprocessing step to normalize document structure"
}
],
"time_token_tradeoff": {
"with_skill_is_slower": true,
"delta_seconds": "+13s",
"delta_tokens": "+1700",
"quality_improvement": "+50% pass rate",
"verdict": "Worth the trade-off for complex tasks"
},
"skill_patterns": {
"helps_with": ["Multi-step workflows", "Format consistency", "Error recovery"],
"does_not_help_with": ["Simple single-step tasks"]
},
"improvement_suggestions": [
{
"priority": "high",
"category": "instructions",
"suggestion": "Add explicit handling for edge case X",
"expected_impact": "Would reduce variance in eval Y"
}
],
"iteration_comparison": {
"is_improvement": true,
"pass_rate_delta": "+15%",
"time_delta": "-5s",
"regressions": []
}
}Guidance
- Be specific - cite eval names, assertion texts, and actual numbers
- Focus on actionable insights, not just observations
- If everything looks good, say so and explain why
Comparator Agent Instructions
You are a comparator agent performing blind A/B comparison between two skill versions.
Input
You will receive two output directories, labeled A and B (but you won't know which is the new vs old):
output_a/- Contains outputs from configuration Aoutput_b/- Contains outputs from configuration Beval_metadata.json- The original eval prompt for context
Your Task
1. Gather Information
Read the eval metadata to understand what the skill was supposed to do. Then examine all files in both output directories, including:
- Final output files
- Any intermediate files
- Metrics or logs if present
2. Perform Blind Comparison
Compare the outputs WITHOUT knowing which is newer/better. Evaluate each on:
- Correctness: Does it solve the task accurately?
- Completeness: Does it include all required elements?
- Coherence: Is the output well-structured and consistent?
- Quality: Is the output production-ready?
3. Score Each Output
For each output, rate these categories (1-5 scale, 5 being best):
Content:
- Correctness
- Completeness
- Accuracy
Structure:
- Organization
- Formatting
- Usability
4. Declare a Winner
Based on your analysis:
- Which output better solves the task?
- What specific differences drove your decision?
- Are there areas where the loser actually excelled?
5. Provide Improvement Suggestions
For the losing version's skill:
- What specific instructions would have helped?
- What patterns or examples were missing?
- What would have made the output match the winner's quality?
Output Format
Write to comparison.json:
{
"winner": "A",
"reasoning": "Output A provides a complete solution with proper formatting. Output B is missing the date field and has inconsistencies.",
"rubric": {
"A": {
"content": { "correctness": 5, "completeness": 5, "accuracy": 4 },
"structure": { "organization": 4, "formatting": 5, "usability": 4 },
"content_score": 4.7,
"structure_score": 4.3,
"overall_score": 9.0
},
"B": {
"content": { "correctness": 3, "completeness": 2, "accuracy": 3 },
"structure": { "organization": 3, "formatting": 2, "usability": 3 },
"content_score": 2.7,
"structure_score": 2.7,
"overall_score": 5.4
}
},
"output_quality": {
"A": {
"score": 9,
"strengths": ["Complete solution", "Well-formatted"],
"weaknesses": ["Minor style inconsistency"]
},
"B": {
"score": 5,
"strengths": ["Readable output"],
"weaknesses": ["Missing date field", "Inconsistencies"]
}
},
"expectation_results": {
"A": { "passed": 4, "total": 5, "pass_rate": 0.8 },
"B": { "passed": 3, "total": 5, "pass_rate": 0.6 }
}
}Important
- You MUST NOT reveal which output is A vs B in your reasoning
- Be specific about what drove your decision
- If it's genuinely close, say so and explain why you picked a winner anyway
- Focus on what the skill INSTRUCTIONS could have done differently, not just the outputs
Grader Agent Instructions
You are a grader agent evaluating the output of a skill evaluation run.
Input
You will receive:
eval_metadata.json- Contains the eval prompt and expected output descriptionoutputs/directory - Contains the files produced by the skillgrading.json(template to fill) - Where you save your grades
Your Task
1. Read eval_metadata.json to understand what was being tested 2. Navigate the outputs/ directory to examine what the skill produced 3. For each expectation in eval_metadata.json["expectations"]:
- Evaluate whether the output satisfies the expectation
- Mark as passed=true or passed=false
- Provide evidence explaining your reasoning
Grading Criteria
- Factual assertions: Verify against the actual output files
- Structural assertions: Check if expected format/structure is present
- Behavioral assertions: Confirm the skill followed the documented workflow
- Quality assertions: Judge completeness and coherence
Output Format
Write your grades to grading.json in the same directory as this readme:
{
"expectations": [
{
"text": "The exact expectation text",
"passed": true,
"evidence": "Why it passed - quote from output or observation"
},
{
"text": "Another expectation",
"passed": false,
"evidence": "Why it failed - what was missing or incorrect"
}
],
"summary": {
"passed": 5,
"failed": 2,
"total": 7,
"pass_rate": 0.71
},
"execution_metrics": {
"tool_calls": { "Read": 5, "Write": 2, "Bash": 8 },
"total_tool_calls": 15,
"total_steps": 6,
"errors_encountered": 0,
"output_chars": 12450,
"transcript_chars": 3200
},
"timing": {
"executor_duration_seconds": 165.0,
"grader_duration_seconds": 26.0,
"total_duration_seconds": 191.0
},
"claims": [
{
"claim": "The form has 12 fillable fields",
"type": "factual",
"verified": true,
"evidence": "Counted 12 fields in field_info.json"
}
],
"user_notes_summary": {
"uncertainties": ["Used 2023 data, may be stale"],
"needs_review": [],
"workarounds": ["Fell back to text overlay for non-fillable fields"]
},
"eval_feedback": {
"suggestions": [
{
"assertion": "The output includes the name 'John Smith'",
"reason": "A hallucinated document that mentions the name would also pass"
}
],
"overall": "Assertions check presence but not correctness."
}
}Important
- Use the EXACT field names:
text,passed,evidence(notname/met/details) - Be objective and provide clear evidence for each verdict
- For file-based outputs, actually read the files to verify claims
- If an expectation is subjective, note that in the evidence but still provide a pass/fail
#!/usr/bin/env python3
"""
Generate an interactive HTML review page for skill evaluation results.
Usage:
python generate_review.py <workspace> --skill-name <name> [--benchmark <benchmark.json>]
python generate_review.py <workspace> --skill-name <name> --static <output.html>
For Cowork/headless environments, use --static to write a standalone HTML file.
"""
import argparse
import json
import sys
from pathlib import Path
def generate_html(workspace: Path, skill_name: str, benchmark_path: Path | None = None, static_output: Path | None = None):
"""Generate the interactive review HTML."""
# Load benchmark data if provided
benchmark = {}
if benchmark_path and benchmark_path.exists():
benchmark = json.loads(benchmark_path.read_text(encoding="utf-8"))
# Discover eval directories
eval_dirs = sorted(workspace.glob("eval-*"))
if not eval_dirs:
eval_dirs = sorted((workspace / "runs").glob("eval-*")) if (workspace / "runs").exists() else []
# Build the HTML
html_parts = []
# Header
html_parts.append("""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Skill Eval Review: """ + skill_name + """</title>
<style>
* { box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; margin: 0; padding: 20px; background: #1a1a2e; color: #eee; }
.container { max-width: 1400px; margin: 0 auto; }
h1 { color: #00d4ff; border-bottom: 2px solid #00d4ff; padding-bottom: 10px; }
h2 { color: #e94560; margin-top: 30px; }
.tabs { display: flex; gap: 10px; margin: 20px 0; }
.tab { padding: 12px 24px; background: #16213e; border: none; color: #eee; cursor: pointer; border-radius: 8px 8px 0 0; font-size: 16px; }
.tab.active { background: #0f3460; color: #00d4ff; }
.tab-content { display: none; background: #0f3460; padding: 20px; border-radius: 0 8px 8px 8px; }
.tab-content.active { display: block; }
.eval-nav { display: flex; justify-content: space-between; align-items: center; margin: 20px 0; padding: 15px; background: #16213e; border-radius: 8px; }
.eval-nav button { padding: 10px 20px; background: #e94560; border: none; color: white; cursor: pointer; border-radius: 6px; }
.eval-nav button:disabled { opacity: 0.5; cursor: not-allowed; }
.eval-title { font-size: 24px; color: #00d4ff; }
.prompt { background: #16213e; padding: 15px; border-radius: 8px; margin: 10px 0; border-left: 4px solid #e94560; }
.prompt-label { color: #e94560; font-weight: bold; margin-bottom: 5px; }
.output-section { background: #16213e; padding: 15px; border-radius: 8px; margin: 10px 0; }
.config-label { font-size: 18px; padding: 8px 16px; border-radius: 6px; display: inline-block; margin: 5px 0; }
.with-skill { background: #00d4ff; color: #1a1a2e; }
.without-skill { background: #e94560; color: white; }
.files-list { list-style: none; padding: 0; }
.files-list li { padding: 8px; background: #1a1a2e; margin: 5px 0; border-radius: 4px; }
.file-preview { background: #1a1a2e; padding: 15px; border-radius: 8px; margin: 10px 0; max-height: 400px; overflow-y: auto; white-space: pre-wrap; font-family: monospace; font-size: 13px; }
.grading { background: #1a1a2e; padding: 15px; border-radius: 8px; margin: 10px 0; }
.assertion { padding: 10px; margin: 5px 0; border-radius: 4px; }
.assertion.pass { background: rgba(0, 255, 100, 0.2); border-left: 4px solid #00ff64; }
.assertion.fail { background: rgba(255, 100, 100, 0.2); border-left: 4px solid #ff6464; }
.assertion-text { font-weight: bold; }
.assertion-evidence { font-size: 13px; color: #aaa; margin-top: 5px; }
.feedback-section { margin-top: 20px; }
.feedback-section textarea { width: 100%; height: 120px; background: #16213e; color: #eee; border: 1px solid #333; border-radius: 8px; padding: 15px; font-size: 14px; resize: vertical; }
.benchmark-table { width: 100%; border-collapse: collapse; margin: 20px 0; }
.benchmark-table th, .benchmark-table td { padding: 12px; text-align: left; border-bottom: 1px solid #333; }
.benchmark-table th { background: #16213e; color: #00d4ff; }
.benchmark-table tr:hover { background: rgba(0, 212, 255, 0.1); }
.delta { color: #00ff64; font-weight: bold; }
.notes { background: #16213e; padding: 15px; border-radius: 8px; margin: 10px 0; }
.note-item { padding: 8px; margin: 5px 0; border-left: 3px solid #e94560; }
.submit-btn { background: #00d4ff; color: #1a1a2e; padding: 15px 30px; border: none; border-radius: 8px; font-size: 18px; cursor: pointer; margin: 20px 0; }
.submit-btn:hover { background: #00b8e6; }
.previous-output { border-top: 2px dashed #666; margin-top: 20px; padding-top: 20px; }
.previous-label { color: #888; font-size: 14px; }
</style>
</head>
<body>
<div class="container">
<h1>Skill Eval Review: """ + skill_name + """</h1>
<div class="tabs">
<button class="tab active" onclick="showTab('outputs')">Outputs</button>
<button class="tab" onclick="showTab('benchmark')">Benchmark</button>
</div>
""")
# Outputs tab
html_parts.append(' <div id="outputs" class="tab-content active">')
html_parts.append(' <div class="eval-nav">')
html_parts.append(' <button onclick="prevEval()">← Previous</button>')
html_parts.append(' <span class="eval-title" id="evalTitle">Eval 0</span>')
html_parts.append(' <button onclick="nextEval()">Next →</button>')
html_parts.append(' </div>')
for idx, eval_dir in enumerate(eval_dirs):
eval_id = eval_dir.name
metadata_path = eval_dir / "eval_metadata.json"
metadata = {}
if metadata_path.exists():
try:
metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
except:
pass
html_parts.append(f' <div class="eval-container" id="eval-{idx}" style="display: {"block" if idx == 0 else "none"};">')
# Prompt
prompt_text = metadata.get("prompt", "No prompt recorded")
html_parts.append(f'''
<div class="prompt">
<div class="prompt-label">PROMPT:</div>
{prompt_text}
</div>
''')
# Find configurations
config_dirs = sorted([d for d in eval_dir.iterdir() if d.is_dir() and list(d.glob("run-*"))])
for config_dir in config_dirs:
config_name = config_dir.name
run_dirs = sorted(config_dir.glob("run-*"))
label_class = "with-skill" if "with_skill" in config_name else "without_skill"
label_text = "With Skill" if "with_skill" in config_name else "Without Skill"
html_parts.append(f'''
<div class="output-section">
<span class="config-label {label_class}">{label_text}</span>
''')
for run_dir in run_dirs:
# Find output files
outputs_dir = run_dir / "outputs"
grading_path = run_dir / "grading.json"
if outputs_dir.exists():
files = list(outputs_dir.iterdir())
if files:
html_parts.append(f'''
<div style="margin: 15px 0;">
<strong>Run {run_dir.name}:</strong>
<ul class="files-list">
''')
for f in files:
html_parts.append(f'<li>{f.name}</li>')
html_parts.append(' </ul>')
# Try to preview text files
for f in files:
if f.suffix in ['.txt', '.md', '.json', '.csv', '.py', '.js']:
try:
content = f.read_text(encoding="utf-8", errors="replace")[:2000]
html_parts.append(f'''
<div class="file-preview"><strong>{f.name}:</strong>\n{content}</div>
''')
except:
pass
html_parts.append(' </div>')
# Grading if exists
if grading_path.exists():
try:
grading = json.loads(grading_path.read_text(encoding="utf-8"))
expectations = grading.get("expectations", [])
summary = grading.get("summary", {})
html_parts.append(f'''
<div class="grading">
<strong>Grading ({summary.get("passed", 0)}/{summary.get("total", 0)} passed, {summary.get("pass_rate", 0)*100:.0f}%)</strong>
''')
for exp in expectations:
status = "pass" if exp.get("passed") else "fail"
html_parts.append(f'''
<div class="assertion {status}">
<div class="assertion-text">{"✅" if exp.get("passed") else "❌"} {exp.get("text", "")}</div>
<div class="assertion-evidence">{exp.get("evidence", "")}</div>
</div>
''')
html_parts.append(' </div>')
except:
pass
html_parts.append(' </div>')
# Feedback
html_parts.append(f'''
<div class="feedback-section">
<label for="feedback-{idx}">Your Feedback:</label>
<textarea id="feedback-{idx}" placeholder="Leave your feedback here..."></textarea>
</div>
''')
html_parts.append(' </div>')
html_parts.append(' </div>')
# Benchmark tab
html_parts.append(' <div id="benchmark" class="tab-content">')
if benchmark:
run_summary = benchmark.get("run_summary", {})
html_parts.append('''
<table class="benchmark-table">
<tr>
<th>Configuration</th>
<th>Pass Rate</th>
<th>Time (s)</th>
<th>Tokens</th>
</tr>
''')
configs = [k for k in run_summary if k != "delta"]
for config in configs:
stats = run_summary[config]
pr = stats.get("pass_rate", {})
time = stats.get("time_seconds", {})
tokens = stats.get("tokens", {})
html_parts.append(f'''
<tr>
<td>{config.replace("_", " ").title()}</td>
<td>{pr.get("mean", 0)*100:.1f}% ± {pr.get("stddev", 0)*100:.1f}%</td>
<td>{time.get("mean", 0):.1f} ± {time.get("stddev", 0):.1f}</td>
<td>{tokens.get("mean", 0):.0f} ± {tokens.get("stddev", 0):.0f}</td>
</tr>
''')
delta = run_summary.get("delta", {})
if delta:
html_parts.append(f'''
<tr style="background: rgba(0, 255, 100, 0.1);">
<td><strong>Delta</strong></td>
<td class="delta">{delta.get("pass_rate", "—")}</td>
<td class="delta">{delta.get("time_seconds", "—")}</td>
<td class="delta">{delta.get("tokens", "—")}</td>
</tr>
''')
html_parts.append(' </table>')
notes = benchmark.get("notes", [])
if notes:
html_parts.append('''
<h3>Analysis Notes</h3>
<div class="notes">
''')
for note in notes:
html_parts.append(f'<div class="note-item">{note}</div>')
html_parts.append(' </div>')
else:
html_parts.append('<p>No benchmark data available. Run evaluations first.</p>')
html_parts.append(' </div>')
# JavaScript
html_parts.append("""
<button class="submit-btn" onclick="submitFeedback()">Submit All Reviews</button>
</div>
<script>
let currentEval = 0;
const totalEvals = document.querySelectorAll('.eval-container').length;
function showTab(tabId) {
document.querySelectorAll('.tab-content').forEach(tc => tc.classList.remove('active'));
document.querySelectorAll('.tab').forEach(t => t.classList.remove('active'));
document.getElementById(tabId).classList.add('active');
event.target.classList.add('active');
}
function showEval(idx) {
document.querySelectorAll('.eval-container').forEach(ec => ec.style.display = 'none');
document.getElementById('eval-' + idx).style.display = 'block';
document.getElementById('evalTitle').textContent = 'Eval ' + (idx + 1) + ' of ' + totalEvals;
currentEval = idx;
}
function prevEval() {
if (currentEval > 0) showEval(currentEval - 1);
}
function nextEval() {
if (currentEval < totalEvals - 1) showEval(currentEval + 1);
}
document.addEventListener('keydown', function(e) {
if (e.key === 'ArrowLeft') prevEval();
if (e.key === 'ArrowRight') nextEval();
});
function submitFeedback() {
const feedback = [];
for (let i = 0; i < totalEvals; i++) {
const textarea = document.getElementById('feedback-' + i);
if (textarea && textarea.value.trim()) {
feedback.push({
eval_index: i,
feedback: textarea.value.trim(),
timestamp: new Date().toISOString()
});
}
}
const blob = new Blob([JSON.stringify({reviews: feedback, status: 'complete'}, null, 2)],
{type: 'application/json'});
const a = document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = 'feedback.json';
a.click();
alert('Feedback submitted! Downloaded feedback.json');
}
</script>
</body>
</html>
""")
return "\n".join(html_parts)
def main():
parser = argparse.ArgumentParser(description="Generate interactive HTML review page for skill evaluations")
parser.add_argument("workspace", type=Path, help="Path to the workspace containing eval-* directories")
parser.add_argument("--skill-name", required=True, help="Name of the skill being evaluated")
parser.add_argument("--benchmark", type=Path, help="Path to benchmark.json")
parser.add_argument("--static", type=Path, help="Write standalone HTML to this path (for headless environments)")
parser.add_argument("--previous-workspace", type=Path, help="Path to previous iteration's workspace for comparison")
args = parser.parse_args()
if not args.workspace.exists():
print(f"Error: Workspace not found: {args.workspace}", file=sys.stderr)
sys.exit(1)
html = generate_html(args.workspace, args.skill_name, args.benchmark, args.static)
if args.static:
args.static.write_text(html, encoding="utf-8")
print(f"Static HTML written to: {args.static}")
else:
try:
import webbrowser
import tempfile
import os
# Write to temp file and open
temp_path = Path(tempfile.gettempdir()) / f"skill_eval_review_{args.skill_name}.html"
temp_path.write_text(html, encoding="utf-8")
webbrowser.open(str(temp_path))
print(f"Opened review in browser: {temp_path}")
except Exception as e:
# Fallback: write to current directory
output_path = Path.cwd() / f"skill_eval_review_{args.skill_name}.html"
output_path.write_text(html, encoding="utf-8")
print(f"Browser not available. Written to: {output_path}")
if __name__ == "__main__":
main()Skill Archetypes
Four proven patterns for organising skills based on their purpose.
1. CLI Reference Skill
Best for: Tool documentation (git, gcloud, vercel, npm)
Characteristics:
- Pure reference, minimal explanatory prose
- Claude already knows CLI semantics
- Group commands by function
- Include common workflows as recipes
Structure:
# Tool Name Skill
## Authentication
[auth commands with examples]
## Core Operations
### Create
command create [options]
-n, --name Name for resource
-t, --type Type (default: basic)
### Read
command list [filters]
command get <id>
### Update
command update <id> [options]
### Delete
command delete <id> [--force]
## Common Workflows
### Deploy to Production
1. command build --prod
2. command deploy --env production
3. command verify --waitToken efficiency: Highest. No explanations Claude doesn't need.
---
2. Methodology Skill
Best for: Workflows, processes, thinking frameworks
Characteristics:
- Philosophy statement upfront
- THE EXACT PROMPT pattern for reproducibility
- Before/after examples
- Why it works explanations
Structure:
# Methodology Name
> **Core Philosophy:** [One-liner insight that drives everything]
## Why This Matters
[Brief motivation - 2-3 sentences max]
## THE EXACT PROMPT
[Copy-paste ready prompt in code block]
## Why This Prompt Works
[Technical breakdown of each component]
## Before/After Examples
### Before (Without Methodology)
[Concrete example of suboptimal approach]
### After (With Methodology)
[Same scenario, better outcome]
## When NOT to Use
[Boundaries and limitations]Example - Code Review Methodology:
# Deep Code Review
> **Core Philosophy:** Reviews should find bugs the tests missed, not style issues the linter catches.
## THE EXACT PROMPT
Review this code focusing on:
1. Logic errors that tests wouldn't catch
2. Edge cases in error handling
3. Security implications of data flow
4. Performance under scale
Skip: formatting, naming conventions, documentation.
## Why This Prompt Works
- "Logic errors tests wouldn't catch" - focuses on value-add
- "Edge cases in error handling" - common bug source
- "Security implications" - expertise most devs lack
- "Skip formatting" - prevents noise---
3. Safety Tool Skill
Best for: Validation, security, guardrails
Characteristics:
- Threat model explicit
- Risk tiering tables
- What it blocks AND allows
- Modular/extensible design
- Security considerations section
Structure:
# Tool Name
## Why This Exists
[Threat model - what could go wrong without this]
## Critical Design Principles
1. **Principle One** - [Explanation]
2. **Principle Two** - [Explanation]
## Risk Tiers
| Tier | Approval | Auto-approve | Examples |
|------|----------|--------------|----------|
| CRITICAL | 2+ humans | Never | rm -rf /, DROP DATABASE |
| DANGEROUS | 1 human | Never | git reset --hard |
| CAUTION | 0 | After 30s | rm single_file.txt |
| SAFE | 0 | Immediately | rm *.log |
## What It Blocks
| Pattern | Reason | Override |
|---------|--------|----------|
| `rm -rf /` | Catastrophic | None |
| `DROP TABLE` | Data loss | --i-really-mean-it |
## What It Allows
| Pattern | Why Safe |
|---------|----------|
| `rm *.tmp` | Temporary files |
| `git stash` | Recoverable |
## Modular Extensions
[How to add new rules]
## Security Considerations
- [Assumption 1]
- [Limitation 1]
- [Known bypass scenarios]---
4. Orchestration Tool Skill
Best for: Multi-agent coordination, automation, pipelines
Characteristics:
- Quick start for immediate use
- Robot mode (JSON APIs) for automation
- Integration matrix with other tools
- State diagrams for complex flows
Structure:
# Tool Name
## Why This Exists
[Pain points solved - bullet list]
## Quick Start
[Minimal viable usage - 3 commands max]
## Core Commands
### Session Management
[commands grouped logically]
### Agent Control
[commands grouped logically]
## Robot Mode (Automation APIs)
### Status Query
tool --robot-status
Output:
{"sessions": [...], "agents": [...], "status": "ok"}
### Snapshot
tool --robot-snapshot
Output:
{"type": "snapshot", "timestamp": "...", "data": {...}}
## Integration Matrix
| Tool | Integration Type | Setup |
|------|-----------------|-------|
| Agent Mail | Message routing | Automatic |
| Validator | Pre-flight checks | Enable in config |
## State Diagram
┌─────────────┐
│ IDLE │
└──────┬──────┘
│ start
▼
┌─────────────┐
│ RUNNING │◄──────┐
└──────┬──────┘ │
│ complete │ retry
▼ │
┌─────────────┐ │
│ VALIDATE │───────┘
└──────┬──────┘
│ pass
▼
┌─────────────┐
│ COMPLETE │
└─────────────┘---
Choosing Your Archetype
| If your skill... | Use |
|---|---|
| Documents a CLI tool | CLI Reference |
| Teaches a process/workflow | Methodology |
| Validates/blocks/guards | Safety Tool |
| Coordinates multiple agents | Orchestration |
| Does multiple of above | Hybrid (pick primary, add sections) |
Hybrid Example
A "deployment skill" might combine:
- CLI Reference sections for deploy commands
- Safety Tool tiers for production vs staging
- Orchestration for multi-step deploy pipeline
# Deployment Skill
## Commands (CLI Reference style)
[deploy commands]
## Safety Tiers (Safety Tool style)
| Environment | Approval Required |
|-------------|-------------------|
| Production | 2 humans |
| Staging | 1 human |
| Dev | None |
## Deployment Pipeline (Orchestration style)
[state diagram and robot mode]JSON Schemas
This document defines the JSON schemas used by skill-creator.
---
evals.json
Defines the evals for a skill. Located at evals/evals.json within the skill directory.
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"prompt": "User's example prompt",
"expected_output": "Description of expected result",
"files": ["evals/files/sample1.pdf"],
"expectations": [
"The output includes X",
"The skill used script Y"
]
}
]
}Fields:
skill_name: Name matching the skill's frontmatterevals[].id: Unique integer identifierevals[].prompt: The task to executeevals[].expected_output: Human-readable description of successevals[].files: Optional list of input file paths (relative to skill root)evals[].expectations: List of verifiable statements
---
history.json
Tracks version progression in Improve mode. Located at workspace root.
{
"started_at": "2026-01-15T10:30:00Z",
"skill_name": "pdf",
"current_best": "v2",
"iterations": [
{
"version": "v0",
"parent": null,
"expectation_pass_rate": 0.65,
"grading_result": "baseline",
"is_current_best": false
},
{
"version": "v1",
"parent": "v0",
"expectation_pass_rate": 0.75,
"grading_result": "won",
"is_current_best": false
},
{
"version": "v2",
"parent": "v1",
"expectation_pass_rate": 0.85,
"grading_result": "won",
"is_current_best": true
}
]
}Fields:
started_at: ISO timestamp of when improvement startedskill_name: Name of the skill being improvedcurrent_best: Version identifier of the best performeriterations[].version: Version identifier (v0, v1, ...)iterations[].parent: Parent version this was derived fromiterations[].expectation_pass_rate: Pass rate from gradingiterations[].grading_result: "baseline", "won", "lost", or "tie"iterations[].is_current_best: Whether this is the current best version
---
grading.json
Output from the grader agent. Located at <run-dir>/grading.json.
{
"expectations": [
{
"text": "The output includes the name 'John Smith'",
"passed": true,
"evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
},
{
"text": "The spreadsheet has a SUM formula in cell B10",
"passed": false,
"evidence": "No spreadsheet was created. The output was a text file."
}
],
"summary": {
"passed": 2,
"failed": 1,
"total": 3,
"pass_rate": 0.67
},
"execution_metrics": {
"tool_calls": {
"Read": 5,
"Write": 2,
"Bash": 8
},
"total_tool_calls": 15,
"total_steps": 6,
"errors_encountered": 0,
"output_chars": 12450,
"transcript_chars": 3200
},
"timing": {
"executor_duration_seconds": 165.0,
"grader_duration_seconds": 26.0,
"total_duration_seconds": 191.0
},
"claims": [
{
"claim": "The form has 12 fillable fields",
"type": "factual",
"verified": true,
"evidence": "Counted 12 fields in field_info.json"
}
],
"user_notes_summary": {
"uncertainties": ["Used 2023 data, may be stale"],
"needs_review": [],
"workarounds": ["Fell back to text overlay for non-fillable fields"]
},
"eval_feedback": {
"suggestions": [
{
"assertion": "The output includes the name 'John Smith'",
"reason": "A hallucinated document that mentions the name would also pass"
}
],
"overall": "Assertions check presence but not correctness."
}
}Fields:
expectations[]: Graded expectations with evidencesummary: Aggregate pass/fail countsexecution_metrics: Tool usage and output size (from executor's metrics.json)timing: Wall clock timing (from timing.json)claims: Extracted and verified claims from the outputuser_notes_summary: Issues flagged by the executoreval_feedback: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising
---
metrics.json
Output from the executor agent. Located at <run-dir>/outputs/metrics.json.
{
"tool_calls": {
"Read": 5,
"Write": 2,
"Bash": 8,
"Edit": 1,
"Glob": 2,
"Grep": 0
},
"total_tool_calls": 18,
"total_steps": 6,
"files_created": ["filled_form.pdf", "field_values.json"],
"errors_encountered": 0,
"output_chars": 12450,
"transcript_chars": 3200
}Fields:
tool_calls: Count per tool typetotal_tool_calls: Sum of all tool callstotal_steps: Number of major execution stepsfiles_created: List of output files createderrors_encountered: Number of errors during executionoutput_chars: Total character count of output filestranscript_chars: Character count of transcript
---
timing.json
Wall clock timing for a run. Located at <run-dir>/timing.json.
How to capture: When a subagent task completes, the task notification includes total_tokens and duration_ms. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.
{
"total_tokens": 84852,
"duration_ms": 23332,
"total_duration_seconds": 23.3,
"executor_start": "2026-01-15T10:30:00Z",
"executor_end": "2026-01-15T10:32:45Z",
"executor_duration_seconds": 165.0,
"grader_start": "2026-01-15T10:32:46Z",
"grader_end": "2026-01-15T10:33:12Z",
"grader_duration_seconds": 26.0
}---
benchmark.json
Output from Benchmark mode. Located at benchmarks/<timestamp>/benchmark.json.
{
"metadata": {
"skill_name": "pdf",
"skill_path": "/path/to/pdf",
"executor_model": "claude-sonnet-4-20250514",
"analyzer_model": "most-capable-model",
"timestamp": "2026-01-15T10:30:00Z",
"evals_run": [1, 2, 3],
"runs_per_configuration": 3
},
"runs": [
{
"eval_id": 1,
"eval_name": "Ocean",
"configuration": "with_skill",
"run_number": 1,
"result": {
"pass_rate": 0.85,
"passed": 6,
"failed": 1,
"total": 7,
"time_seconds": 42.5,
"tokens": 3800,
"tool_calls": 18,
"errors": 0
},
"expectations": [
{"text": "...", "passed": true, "evidence": "..."}
],
"notes": [
"Used 2023 data, may be stale",
"Fell back to text overlay for non-fillable fields"
]
}
],
"run_summary": {
"with_skill": {
"pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90},
"time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0},
"tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100}
},
"without_skill": {
"pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45},
"time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0},
"tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500}
},
"delta": {
"pass_rate": "+0.50",
"time_seconds": "+13.0",
"tokens": "+1700"
}
},
"notes": [
"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent",
"Without-skill runs consistently fail on table extraction expectations",
"Skill adds 13s average execution time but improves pass rate by 50%"
]
}Fields:
metadata: Information about the benchmark runskill_name: Name of the skilltimestamp: When the benchmark was runevals_run: List of eval names or IDsruns_per_configuration: Number of runs per config (e.g. 3)runs[]: Individual run resultseval_id: Numeric eval identifiereval_name: Human-readable eval name (used as section header in the viewer)configuration: Must be"with_skill"or"without_skill"(the viewer uses this exact string for grouping and color coding)run_number: Integer run number (1, 2, 3...)result: Nested object withpass_rate,passed,total,time_seconds,tokens,errorsrun_summary: Statistical aggregates per configurationwith_skill/without_skill: Each containspass_rate,time_seconds,tokensobjects withmeanandstddevfieldsdelta: Difference strings like"+0.50","+13.0","+1700"notes: Freeform observations from the analyzer
Important: The viewer reads these field names exactly. Using config instead of configuration, or putting pass_rate at the top level of a run instead of nested under result, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.
---
comparison.json
Output from blind comparator. Located at <grading-dir>/comparison-N.json.
{
"winner": "A",
"reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
"rubric": {
"A": {
"content": {
"correctness": 5,
"completeness": 5,
"accuracy": 4
},
"structure": {
"organization": 4,
"formatting": 5,
"usability": 4
},
"content_score": 4.7,
"structure_score": 4.3,
"overall_score": 9.0
},
"B": {
"content": {
"correctness": 3,
"completeness": 2,
"accuracy": 3
},
"structure": {
"organization": 3,
"formatting": 2,
"usability": 3
},
"content_score": 2.7,
"structure_score": 2.7,
"overall_score": 5.4
}
},
"output_quality": {
"A": {
"score": 9,
"strengths": ["Complete solution", "Well-formatted", "All fields present"],
"weaknesses": ["Minor style inconsistency in header"]
},
"B": {
"score": 5,
"strengths": ["Readable output", "Correct basic structure"],
"weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
}
},
"expectation_results": {
"A": {
"passed": 4,
"total": 5,
"pass_rate": 0.80,
"details": [
{"text": "Output includes name", "passed": true}
]
},
"B": {
"passed": 3,
"total": 5,
"pass_rate": 0.60,
"details": [
{"text": "Output includes name", "passed": true}
]
}
}
}---
analysis.json
Output from post-hoc analyzer. Located at <grading-dir>/analysis.json.
{
"comparison_summary": {
"winner": "A",
"winner_skill": "path/to/winner/skill",
"loser_skill": "path/to/loser/skill",
"comparator_reasoning": "Brief summary of why comparator chose winner"
},
"winner_strengths": [
"Clear step-by-step instructions for handling multi-page documents",
"Included validation script that caught formatting errors"
],
"loser_weaknesses": [
"Vague instruction 'process the document appropriately' led to inconsistent behavior",
"No script for validation, agent had to improvise"
],
"instruction_following": {
"winner": {
"score": 9,
"issues": ["Minor: skipped optional logging step"]
},
"loser": {
"score": 6,
"issues": [
"Did not use the skill's formatting template",
"Invented own approach instead of following step 3"
]
}
},
"improvement_suggestions": [
{
"priority": "high",
"category": "instructions",
"suggestion": "Replace 'process the document appropriately' with explicit steps",
"expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
}
],
"transcript_insights": {
"winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script",
"loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods"
}
}Token Hierarchy Deep Dive
Understanding how Claude loads and processes skills is critical for efficient design.
The Three-Level Loading Model
Level 1: Metadata (Always Loaded)
Cost: ~100 tokens per skill When: Every conversation startup What: Only name and description from YAML frontmatter
---
name: pdf-processing
description: Extract text from PDFs. Use when working with PDF files.
---Implication: You want many skills? Keep descriptions concise. 50 skills × 100 tokens = 5,000 tokens at startup.
Level 2: SKILL.md Body (On Trigger)
Cost: ~1,500-5,000 tokens (aim for <2,000) When: After Claude selects the skill based on description match What: Full markdown content of SKILL.md
Implication: This is where core guidance lives. Make it count but keep it lean.
Level 3: Bundled Resources (On-Demand)
Cost: Unlimited potential, but zero until accessed When: Claude reads a referenced file What: scripts/, references/, assets/
Implication: Put detailed docs, schemas, examples here. They cost nothing until needed.
Selection Mechanism
Critical insight: Claude uses pure LLM reasoning to select skills.
There is:
- No embedding similarity search
- No keyword matching algorithm
- No classifier model
Claude literally reads all descriptions and uses natural language understanding to decide relevance.
What this means:
- Vague descriptions = skill never triggers
- Specific trigger phrases = reliable activation
- Description quality is THE critical factor
Token Budget Strategy
For a Project with 20 Skills
Startup overhead: 20 × 100 = 2,000 tokens (unavoidable)
Active skill: ~2,000 tokens (SKILL.md body)
Reference files: ~5,000 tokens (only if accessed)
Conversation: Remaining context window
Total for one skill usage: ~9,000 tokensOptimisation Strategies
1. Front-load essentials in SKILL.md
- Quick start that covers 80% of use cases
- Only reference files for advanced features
2. Keep references shallow
- SKILL.md -> reference.md (one level)
- Never: SKILL.md -> overview.md -> details.md
3. Use grep hints for large references
## Schema reference
See [schema.md](references/schema.md)
Quick lookup: `grep -i "column_name" references/schema.md`4. Scripts execute, not load
# This loads script content (~500 tokens)
See scripts/validate.py for implementation details
# This executes without loading (~50 tokens output)
Run: python scripts/validate.py input.pdfPractical Examples
Token-Efficient Skill (~1,500 tokens active)
skill/
├── SKILL.md (800 words core)
└── references/
├── api.md (full API, loaded only when needed)
└── examples.md (working code, loaded only when needed)Token-Heavy Skill (~5,000 tokens active) - Avoid
skill/
└── SKILL.md (3,000 words, everything inline)Optimal Structure
# In SKILL.md
## Quick start (always loaded)
[80% of use cases covered in 500 words]
## Advanced features (pointers only)
- **Forms**: See [FORMS.md](references/FORMS.md)
- **API**: See [API.md](references/API.md)
## Quick search (for large references)
grep -i "topic" references/Measuring Token Impact
1. Check skill loading: claude --debug shows what loads 2. Count SKILL.md: ~4 characters = 1 token (rough estimate) 3. Test with fresh context: Does skill work without reference files?
Summary
| Level | Tokens | Optimisation |
|---|---|---|
| Metadata | ~100 fixed | Concise descriptions |
| SKILL.md | ~1,500-2,000 target | 80% coverage, pointers to rest |
| References | 0 until accessed | Large docs, schemas, examples |
| Scripts | Output only | Execute don't load |
#!/usr/bin/env python3
"""Output audit prompt for skill evaluation.
Usage:
python audit.py <skill-path>
Outputs a structured text prompt that agents consume to run audit checks.
"""
import argparse
import re
import sys
from pathlib import Path
from typing import Dict
PROJECT_SKILL_DIRS = [
".agents/skills",
".agent/skills",
".github/skills",
".gemini/skills",
".kiro/skills",
]
REPO_SKILL_DIRS = ["skills"]
GLOBAL_SKILL_DIRS = [
("~/.agents/skills", "~/.agents/skills"),
("~/.gemini/antigravity/skills", "~/.gemini/antigravity/skills"),
("~/.copilot/skills", "~/.copilot/skills"),
("~/.kiro/skills", "~/.kiro/skills"),
]
def find_skill(skill_identifier: str) -> Path | None:
if Path(skill_identifier).is_absolute():
p = Path(skill_identifier)
if (p / "SKILL.md").exists():
return p
if p.is_dir() and (p.parent / "SKILL.md").exists():
return p.parent
cwd = Path.cwd()
for parent in [cwd, *cwd.parents]:
if parent != cwd and (parent / ".git").exists():
break
for skill_dir in PROJECT_SKILL_DIRS:
skill_path = parent / skill_dir / skill_identifier
if (skill_path / "SKILL.md").exists():
return skill_path
for skill_dir in REPO_SKILL_DIRS:
skill_path = parent / skill_dir / skill_identifier
if (skill_path / "SKILL.md").exists():
return skill_path
direct_path = parent / skill_identifier
if (direct_path / "SKILL.md").exists():
return direct_path
home = Path.home()
for local_key, global_path in GLOBAL_SKILL_DIRS:
skill_path = Path(global_path.replace("~", str(home))) / skill_identifier
if (skill_path / "SKILL.md").exists():
return skill_path
return None
def parse_frontmatter(skill_md: Path) -> Dict[str, str]:
content = skill_md.read_text(encoding="utf-8")
lines = content.split('\n')
frontmatter = {}
in_fm = False
fm_lines = []
for line in lines:
if line.strip() == '---':
if not in_fm:
in_fm = True
continue
else:
break
if in_fm:
fm_lines.append(line)
current_key = None
block_content = []
in_block = False
block_markers = ['>-', '|-', '>|', '|>']
for l in fm_lines:
l_stripped = l.rstrip()
if any(l_stripped.endswith(marker) for marker in block_markers):
current_key = l_stripped.split(':')[0].strip()
in_block = True
block_content = []
continue
elif in_block:
if l_stripped == '' or l[0] != ' ':
in_block = False
frontmatter[current_key] = '\n'.join(block_content).strip()
block_content = []
else:
block_content.append(l_stripped)
continue
m = re.match(r'^(\w+):\s*(.*)', l)
if m:
key = m.group(1).strip()
val = m.group(2).strip().split('#')[0].strip()
frontmatter[key] = val
if in_block and current_key:
frontmatter[current_key] = '\n'.join(block_content).strip()
return frontmatter
def get_skill_info(skill_path: Path) -> Dict:
skill_md = skill_path / "SKILL.md"
frontmatter = parse_frontmatter(skill_md)
content = skill_md.read_text(encoding="utf-8")
lines = content.split('\n')
documented_scripts = set()
for m in re.finditer(r'(?:scripts?|script)[\s:>]+[`"]?([\w-]+\.(?:py|sh|js|ts|rb|mjs))', content, re.IGNORECASE):
documented_scripts.add(m.group(1).lower())
for m in re.finditer(r'\|\s*`?([\w-]+\.(?:py|sh|js|ts|rb|mjs))\s*`?\s*\|', content):
documented_scripts.add(m.group(1).lower())
scripts_dir = skill_path / "scripts"
actual_scripts = set()
if scripts_dir.exists():
actual_scripts = {f.name.lower() for f in scripts_dir.iterdir()
if f.is_file() and f.name != "__init__.py"}
return {
"name": frontmatter.get("name", skill_path.name),
"description": frontmatter.get("description", ""),
"language": frontmatter.get("language", ""),
"line_count": len(lines),
"documented_scripts": sorted(documented_scripts),
"actual_scripts": sorted(actual_scripts),
"has_scripts_dir": scripts_dir.exists(),
"has_references_dir": (skill_path / "references").exists(),
}
def generate_audit_prompt(skill_path: Path) -> str:
info = get_skill_info(skill_path)
lines = [
"SKILLFORGE AUDIT PROMPT",
"=======================",
"",
f"Skill: {info['name']}",
f"Path: {skill_path}",
"",
"CHECKS TO RUN:",
"",
"[1] FILE COVERAGE",
f" Documented scripts: {', '.join(info['documented_scripts']) or 'none'}",
f" Actual scripts: {', '.join(info['actual_scripts']) or 'none'}",
" Task: Verify documented files exist. Flag orphaned scripts.",
"",
"[2] STRUCTURE",
f" Name: {info['name']}",
f" Has description: {'yes' if info['description'] else 'NO'}",
f" Language: {info['language'] or 'MISSING'}",
f" Line count: {info['line_count']}",
" Task: Verify frontmatter has name, description, language. Check line count <= 500.",
"",
"[3] DESCRIPTION FORMAT",
" Required: Description must contain 'Use when...' or 'Usa cuando...'",
" Task: Verify trigger phrase is present.",
"",
"[4] LANGUAGE CONSISTENCY",
f" Declared: {info['language'] or 'MISSING'}",
" Allowed: en or es-CO",
" Task: Verify all content matches declared language.",
"",
"[5] SECURITY SCAN",
" Dangerous patterns to find: os.system, shell=True, eval(), exec(), __import__(), base64.b64decode",
" Task: Scan scripts/ directory. Report any matches.",
"",
"[6] LENGTH",
f" Max lines: 500",
f" Current: {info['line_count']}",
f" Status: {'PASS' if info['line_count'] <= 500 else 'FAIL'}",
" Task: Verify SKILL.md is under 500 lines.",
"",
"INSTRUCTIONS:",
" 1. Spawn subagents to run each check in parallel",
" 2. Each subagent reports pass/fail with details",
" 3. Aggregate results and report specific issues found",
"",
"OUTPUT FORMAT:",
" summary: passed=X failed=Y warnings=Z",
" issues:",
" - check=... severity=error|warning message=...",
"",
]
return '\n'.join(lines)
def main():
parser = argparse.ArgumentParser(description="Output audit prompt for skill evaluation")
parser.add_argument("skill", help="Skill name or absolute path")
args = parser.parse_args()
skill_path = find_skill(args.skill)
if not skill_path:
print(f"Error: Skill not found: {args.skill}", file=sys.stderr)
sys.exit(1)
prompt = generate_audit_prompt(skill_path)
print(prompt)
if __name__ == "__main__":
main()#!/usr/bin/env python3
"""Deploy skill-forge slash-commands to agent directories.
Detects available agents (Antigravity, OpenCode, Copilot, Kiro) and deploys
slash-commands to appropriate directories.
"""
import os
import shutil
from pathlib import Path
AGENT_DIRS = {
"universal": ".agents/skills",
"kiro": ".kiro/skills",
}
AGENT_TOOLS = ["antigravity", "opencode", "copilot", "kiro"]
def detect_agents() -> dict[str, bool]:
"""Detect which agents are available in the system."""
return {agent: shutil.which(agent) is not None for agent in AGENT_TOOLS}
WORKFLOW_TEMPLATES = {
"skill-create": """---
description: Scaffold a new skill directory using the skill-forge standard
---
## Creating a New Skill
1. **Invoke Initialization**: Run `python {forge_script_dir}/init.py <skill-name> --path <destination>`
2. **Document Triggers**: Edit the newly created `SKILL.md` and focus on the `description` block. Ensure you answer "Use when...", not "What this does".
3. **Draft Instructions**: Write the instructional prose inside `SKILL.md`, keeping it under 500 lines.
""",
"skill-audit": """---
description: Validate a skill's structure, description format, and security
---
## Auditing a Skill
1. **Run Audit**: Execute `python {forge_script_dir}/audit.py <path-to-skill>`
2. **Review Output**: The script outputs an XML prompt. Present this to a subagent capable of running the audit checks.
3. **Apply Fixes**: Based on the audit results, update `SKILL.md` accordingly.
""",
"skill-improve": """---
description: Optimize a skill using Test-Driven Development
---
## Improving a Skill (TDD)
1. **Run TDD Loop**: Present the XML prompt from `audit.py` to a subagent. The agent spawns additional subagents internally to run different models against test cases.
2. **Understand Output**: The script outputs a single XML prompt containing test cases and improvement criteria.
3. **Execute Loop**: Present the XML to a subagent. The agent spawns additional subagents internally to run different models against test cases.
4. **Iterate**: The agent runs the TDD loop until all tests pass or max iterations reached.
5. **Result**: Updated `SKILL.md` with optimized description.
""",
}
def setup_slash_commands(workspace_root: Path, forge_script_dir: Path):
"""Deploy slash-commands to all available agent directories."""
detected = detect_agents()
available = [a for a, present in detected.items() if present]
if available:
print(f"Detected agents: {', '.join(available)}")
else:
print("Warning: No agent CLIs detected (antigravity, opencode, copilot, kiro)")
deployed_files = []
for scope, rel_path in AGENT_DIRS.items():
target_dir = workspace_root / rel_path
target_dir.mkdir(parents=True, exist_ok=True)
for workflow_name, template in WORKFLOW_TEMPLATES.items():
filepath = target_dir / f"{workflow_name}.md"
content = template.format(
forge_script_dir=str(forge_script_dir.absolute()).replace("\\", "/")
)
filepath.write_text(content, encoding="utf-8")
deployed_files.append(str(filepath))
print(f"Deployment complete. Generated {len(deployed_files)} workflow files.")
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Deploy skill-forge slash-commands to agent directories")
parser.add_argument("--workspace", default=os.getcwd(), help="Path to the workspace root")
parser.add_argument("--forge-scripts", required=True, help="Path to the skill-forge scripts directory")
args = parser.parse_args()
setup_slash_commands(Path(args.workspace), Path(args.forge_scripts))import sys
from pathlib import Path
def generate_frontmatter(name: str) -> str:
"""Generates agent-skills spec compliant frontmatter"""
return f"""---
name: {name}
description: >-
[What it does - actions, capabilities].
Use when [trigger phrases, contexts, file types].
---
# {name}
## When to Use
- [Symptom 1]
- [Symptom 2]
- [Symptom 3]
## Core Pattern
```bash
# Add your core instructional patterns here
```
## Quick Reference
| Action | Command |
| ------ | ------- |
| Do X | `run X` |
"""
def init_skill(skill_name: str, target_dir: Path):
"""
Scaffolds the directory structure for a new skill.
Enforces the Token Loading Hierarchy.
"""
skill_dir = target_dir / skill_name
if skill_dir.exists():
print(f"Error: Directory {skill_dir} already exists.")
sys.exit(1)
skill_dir.mkdir(parents=True)
# Create Token Loading Hierarchy Dirs
(skill_dir / "scripts").mkdir()
(skill_dir / "references").mkdir()
(skill_dir / "examples").mkdir()
(skill_dir / "evals").mkdir()
# Write main SKILL.md
with open(skill_dir / "SKILL.md", "w", encoding="utf-8") as f:
f.write(generate_frontmatter(skill_name))
print(f"✅ Created core structure for {skill_name}")
print(f" Docs: {skill_dir}/SKILL.md")
print(f" Context: {skill_dir}/references/")
print(f" Logic: {skill_dir}/scripts/")
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser("Scaffold a unified skill-forge compliant skill")
parser.add_argument("name", help="Name of the skill (lowercase, hyphens)")
parser.add_argument("--path", required=True, help="Destination directory path")
args = parser.parse_args()
init_skill(args.name, Path(args.path))
"""Shared utilities for skill-creator scripts."""
from pathlib import Path
def parse_skill_md(skill_path: Path) -> tuple[str, str, str]:
"""Parse a SKILL.md file, returning (name, description, full_content)."""
content = (skill_path / "SKILL.md").read_text()
lines = content.split("\n")
if lines[0].strip() != "---":
raise ValueError("SKILL.md missing frontmatter (no opening ---)")
end_idx = None
for i, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
end_idx = i
break
if end_idx is None:
raise ValueError("SKILL.md missing frontmatter (no closing ---)")
name = ""
description = ""
frontmatter_lines = lines[1:end_idx]
i = 0
while i < len(frontmatter_lines):
line = frontmatter_lines[i]
if line.startswith("name:"):
name = line[len("name:"):].strip().strip('"').strip("'")
elif line.startswith("description:"):
value = line[len("description:"):].strip()
# Handle YAML multiline indicators (>, |, >-, |-)
if value in (">", "|", ">-", "|-"):
continuation_lines: list[str] = []
i += 1
while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(" ") or frontmatter_lines[i].startswith("\t")):
continuation_lines.append(frontmatter_lines[i].strip())
i += 1
description = " ".join(continuation_lines)
continue
else:
description = value.strip('"').strip("'")
i += 1
return name, description, content
Related skills
Find SkillsSo their agent can intelligently recommend and install the right modular skills from the open ecosystem when they ask how to do something.3M27.9k
Lark Skill MakerRapidly create custom skills for Lark (Feishu) that can be invoked by their AI agents and automation workflows.513k
Skill CreatorGenerate, test, and iterate on new reusable agent skills directly inside their Claude or Cursor workflow.337k166k
BrainstormingTurn vague product ideas into clear, approved designs and specifications through guided collaborative dialogue before writing any code.307k265k
Writing Great SkillsCreate high-quality, reusable agent skills that other developers can discover and install from the Skillselion catalog.290k200k
Find SkillsInstantly surface the best relevant agent skills, MCP servers, and workflows across multiple registries before starting any new project.226k12FAQ
Does it call external APIs?
No, scripts output structured XML prompts that agents parse and then spawn subagents internally to run evaluations; no external API calls are required.
What does /skill-audit check?
SKILL.md length under 500 lines, the 'Use when...' description format, reference folder depth (max 1 level), language consistency (en or es-CO), and a security scan of the scripts directory.