Review Skill

Review Skill Workflow

You are helping an SME review an Agent Skill before publishing. This is a multi-step process: determine environment, verify prerequisites, run structural validation, review content, optionally run LLM scoring, and interpret results. Follow every step in order.

Step 0: Determine Environment

Check for saved configuration:

cat ~/.config/skill-validator/review-state.yaml 2>/dev/null

If the state file exists with prereqs_passed: true, offer:

Found saved settings — configured for [full/structural-only] reviews.

>

1. Continue with saved settings — skip to Step 2

2. Re-run prerequisite checks

3. Change environment — switch between full and structural-only

Option 1: read llm_scoring from the file and skip to Step 2. Options 2-3: continue below.

If no state file exists, or the user chose to re-check/change, ask:

LLM scoring evaluates content quality across multiple dimensions.

>

1. Yes, run LLM scoring — full review with LLM scoring

2. No, skip LLM scoring — structural validation only

Option 1: set LLM_SCORING=true. Option 2: set LLM_SCORING=false. Run Step 1a only, then jump to Step 2.

Step 1: Verify Prerequisites

1a. Check for skill-validator binary

skill-validator --version

If not found, search common locations (/usr/local/bin, /opt/homebrew/bin, ~/go/bin). If found but not on PATH, tell the user. If not found anywhere, follow references/install-skill-validator.md.

If --version is not at least v1.5.1, help the user upgrade with brew upgrade skill-validator or go install github.com/agent-ecosystem/skill-validator/cmd/skill-validator@latest.

Do NOT proceed until this succeeds.

1b. Check for claude CLI (LLM scoring only)

If LLM_SCORING=true, verify the Claude CLI is available:

claude --version

If not found, tell the user to install Claude Code:

• macOS: curl -fsSL https://claude.ai/install.sh | bash
• Other platforms: follow the Claude Code quickstart guide

The user must authenticate by running claude interactively before continuing.

Do NOT proceed with LLM scoring until this succeeds.

Save state after prerequisites pass

Persist state so future runs skip this step. Replace <true or false> with the actual LLM_SCORING value:

mkdir -p ~/.config/skill-validator
cat > ~/.config/skill-validator/review-state.yaml << 'EOF'
prereqs_passed: true
llm_scoring: <true or false>
EOF

Step 2: Locate the Skill

Ask the user for the path to the skill they want to review, unless they have already provided it. Verify the path contains a SKILL.md file:

ls <path>/SKILL.md

If SKILL.md does not exist at the given path, tell the user this is not a valid skill directory and ask them to provide the correct path.

Step 3: Run Structural Validation

Run the full check suite:

skill-validator check <path>

Capture the exit code:

Exit 0: proceed. Exit 2: note warnings, proceed. Exit 1: list errors — these are blocking. The user must fix them before the skill can be published. Do NOT proceed to LLM scoring if exit code is 1.

Step 4: Content Review

Read the SKILL.md and any reference files, then evaluate each check below. Report which checks pass and which do not, with specific details on what is missing.

Flag any failing checks as areas the SME should address. These are not blocking but should be resolved before publishing for best results.

Step 5: LLM Scoring and Interpretation

If LLM_SCORING=false, skip to Step 6.

If LLM_SCORING=true, follow the "Run LLM Scoring" and "Interpret LLM Scores" sections of references/llm-scoring.md.

Step 6: Present the Review Summary

If LLM_SCORING=true, follow the "Full Review Summary" section of references/llm-scoring.md. Include any failing content review checks from Step 4 in the action items.

If LLM_SCORING=false, present structural result, content review result, areas to address, and a self-assessment checklist using the scoring dimensions from assets/report.md. Note that LLM scoring was skipped; advise re-running with LLM scoring enabled or self-assessing against the report dimensions.

Example Review Summary Structure

Structure the final summary with these sections in order:

1. Structural validation — pass/fail with errors or warnings 2. SKILL.md scores — overall and per-dimension table 3. Reference scores — per-file table with overall and lowest dimension 4. Novelty assessment — mean novelty vs threshold of 3; list novel_info per file for SME verification 5. Action items — prioritized list of what to fix 6. Recommendation — ready to publish / minor revisions / significant rework

Review Report Interpretation Framework

Use this framework to interpret LLM scoring results and present them to the SME.

Reading the scores

Each dimension is scored 1-5. The overall score is the mean of all dimensions.

Score scale

Dimension-specific guidance

When a dimension scores low, use this guidance to advise the SME on what to fix.

Clarity (SKILL.md and references)

Low clarity means the instructions are ambiguous or confusing. Common causes:

• Vague language where precise instructions are needed
• Missing prerequisite declarations (tools, runtimes, permissions)
• Instructions that could be interpreted multiple ways
• Poor formatting or disorganized structure

Advice: Rewrite ambiguous sections to have exactly one interpretation. Declare all dependencies and prerequisites explicitly. Use consistent formatting and logical section ordering.

Actionability (SKILL.md only)

Low actionability means an agent cannot follow the instructions step-by-step. Common causes:

• Abstract advice instead of concrete steps
• Missing intermediate steps that an agent would need
• Assumptions about context the agent won't have
• No examples of expected inputs/outputs

Advice: Convert abstract guidance into numbered steps. Add examples. Fill in any gaps where an agent would need to guess what to do next.

Token Efficiency (SKILL.md and references)

Low token efficiency means the content is bloated relative to its instructional value. Common causes:

• Redundant explanations of the same concept
• Boilerplate text that doesn't help the agent
• Verbose phrasing where concise language would work
• Content that could be compressed without losing meaning

Advice: Cut redundant sections. Replace verbose explanations with concise directives. Remove boilerplate. Every sentence should teach the agent something it needs to know.

Scope Discipline (SKILL.md only)

Low scope discipline means the skill sprawls beyond its stated purpose. Common causes:

• Covering multiple languages or frameworks when the skill targets one
• Including tangential content that could confuse the agent
• Trying to do too many things in a single skill

Advice: Split broad skills into focused ones. Remove content that doesn't directly serve the skill's stated purpose. If the skill mentions other languages or frameworks, ensure those references are clearly delineated.

Directive Precision (SKILL.md only)

Low directive precision means the skill hedges when it should be direct. Common causes:

• Using "consider", "may", "could", "possibly" for required actions
• Missing conditional gates (when to proceed vs skip vs abort)
• Ambiguity about what is required vs optional

Advice: Replace hedged language with precise directives (must, always, never, ensure). Add explicit conditions for branching paths. Make it clear what is required vs optional.

Novelty (SKILL.md and references)

Low novelty means the content mostly restates what the model already knows from training data. This is the most important quality signal for deciding whether a skill justifies its context window cost.

Score below 3 — warning sign: The skill may not contribute enough value. The SME should critically evaluate whether the skill teaches the agent something genuinely new. Common low-novelty patterns:

• Restating official documentation that models have already ingested
• Describing standard patterns or best practices that are common knowledge
• Covering well-documented public APIs without adding proprietary context
• Tutorials or guides on widely-known topics

Advice: Focus the skill on what is genuinely proprietary or non-obvious: internal API conventions, organization-specific workflows, undocumented gotchas, non-standard configurations, or domain knowledge not available in public docs. Cut or heavily compress sections that just restate public knowledge.

For deeper context on why novelty matters and the research behind it, refer the SME to: https://agentskillreport.com/

Instructional Value (references only)

Low instructional value means the reference is abstract rather than practically useful. Common causes:

• Descriptions without working code examples
• Theoretical explanations without concrete patterns
• API docs without usage examples or signatures

Advice: Add concrete, copy-pasteable code examples. Include actual API signatures. Show patterns the agent can use directly, not just descriptions of concepts.

Skill Relevance (references only)

Low skill relevance means the reference includes content unrelated to the parent skill's purpose. Common causes:

• Generic reference docs bundled without curation
• Tangential content that doesn't support the skill's task
• Reference files that cover a broader scope than the skill needs

Advice: Curate reference files tightly to the skill's purpose. Remove sections that an agent would never need for the skill's specific task. If a reference covers too broad a scope, extract only the relevant portions.

Structuring the review summary

Present results to the SME in this order:

1. Structural validation: Did the skill pass? List any errors (blocking) or warnings (non-blocking).

2. SKILL.md scores: Show the per-dimension breakdown with the overall score. Highlight any dimension at 2 or below.

3. Reference file scores (if applicable): Show the per-file breakdown so the SME can see exactly which reference files need attention. Flag any file with an overall score below 3 or any individual dimension at 2 or below.

4. Novelty assessment: Explicitly call out whether the mean novelty score meets the threshold of 3. If it doesn't, this is the most important finding in the review. List the novel_info details for each file — these tell the SME what the LLM identified as genuinely new. The SME should verify these claims are accurate, since the LLM may hallucinate or miss truly novel content.

5. Prioritized action items: List specific things the SME should fix, ordered by impact. Structural errors first, then low-novelty concerns, then other low-scoring dimensions.

6. Publish recommendation: One of:

• Ready to publish — passes structural validation, no dimension below 3,

novelty >= 3

• Minor revisions needed — passes structural validation, 1-2 dimensions

need attention, novelty is borderline

• Significant rework needed — structural errors, multiple low dimensions,

or novelty below 3 with no clear path to improvement

Installing skill-validator

Installation methods

Homebrew (recommended for macOS)

brew tap agent-ecosystem/homebrew-tap
brew install skill-validator

From source (requires Go 1.25.5+)

go install github.com/agent-ecosystem/skill-validator/cmd/skill-validator@latest

Ensure $GOPATH/bin (usually ~/go/bin) is on your PATH:

export PATH="$PATH:$(go env GOPATH)/bin"

From a pre-built binary

cp /path/to/skill-validator /usr/local/bin/ && chmod +x /usr/local/bin/skill-validator

Verify installation

skill-validator --version

Prerequisites for LLM scoring

LLM scoring uses the claude-cli provider, which shells out to the locally installed claude binary. No API keys are needed — it uses the CLI's existing authentication.

1. Claude Code CLI (claude --version) — install with curl -fsSL https://claude.ai/install.sh | bash on macOS, or follow the Claude Code quickstart guide for other platforms 2. Authenticated session — run claude interactively to complete login if not yet authenticated

LLM Scoring Flow

LLM scoring steps using the claude-cli provider. This provider shells out to the locally installed claude binary — no API keys are needed. It uses the CLI's existing authentication (e.g., company or team subscription).

Only follow this if the user selected LLM scoring in Step 0.

Accuracy caveat

The Claude CLI loads local context (CLAUDE.md files, project memory, rules) into each scoring call. This extra context may influence scores, making them less reproducible across environments compared to API-based providers.

Run LLM Scoring (after structural validation passes)

Check for cached scores:

skill-validator score report <path> -o json 2>/dev/null

If scored output exists, use --rescore to generate fresh scores (content may have changed since the last run):

skill-validator score evaluate <path> --provider claude-cli --full-content --display files -o json --rescore

If no cached scores exist, run without --rescore:

skill-validator score evaluate <path> --provider claude-cli --full-content --display files -o json

After scoring completes, run the comparison report:

skill-validator score report <path> -o json

Capture both outputs for interpretation.

Error handling

Interpret LLM Scores

Read ../assets/report.md for the full interpretation framework, then present results to the user following that structure.

Quality thresholds

There are no hard pass/fail gates on most dimensions. Use these guidelines:

• Overall >= 3.5: The skill is in good shape across most dimensions.
• Overall 2.5-3.5: The skill needs work in specific areas. Identify which

dimensions are dragging the score down and advise accordingly.

• Overall < 2.5: The skill needs significant revision across multiple areas.
• Any dimension at 2 or below: Flag this specifically as an area needing

attention. Explain what a low score on that dimension means and suggest concrete improvements.

Novelty is the key differentiator

If mean novelty across all files is below 3, the skill may not justify its context window cost. A low-novelty skill restates what models already know. The critical question: does this skill teach the agent something it genuinely doesn't know?

Surface novel_info for SME review

Each scored file includes a novel_info field describing what the LLM judge identified as genuinely novel content. Present these details to the SME for each file, because:

• Verification: The LLM may misidentify something as novel or miss truly

novel content. The SME should confirm the claims are accurate.

• Focus: Items listed in novel_info represent the skill's highest-value

content. If these details are wrong or missing, the novelty score is unreliable.

• Trimming guidance: Content NOT mentioned in novel_info is likely

restating common knowledge and is a candidate for compression or removal.

If novelty is low, advise the SME to:

1. Identify which sections contain information that is NOT available in public documentation or model training data (proprietary APIs, internal conventions, non-obvious gotchas, organization-specific workflows). 2. Cut or compress sections that merely restate common knowledge. 3. Focus the skill on the genuinely novel content.

For more context on why novelty matters and how to think about skill quality, refer the SME to: https://agentskillreport.com/

Important scoring caveat

Scores are generated using Anthropic Claude, so novelty reflects what Claude knows from training data. Other model families may produce different novelty scores due to different training data coverage.

Full Review Summary

When LLM scoring was performed, present the review summary with:

1. Structural validation result: Pass/fail, with any errors or warnings. 2. LLM score summary: Overall score and per-dimension breakdown for SKILL.md and references (if any). 3. Areas to address: Specific dimensions that need improvement, with concrete suggestions. 4. Novelty assessment: Whether the skill provides sufficient novel value, with specific guidance if it doesn't. Include the novel_info details for each file so the SME can verify accuracy and identify what to keep or cut. 5. Recommendation: Whether the skill is ready to publish, needs minor revisions, or needs significant rework.