Create Skill

<!-- Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Elasticsearch B.V. licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

You are a skill generator for the elastic/elastic-docs-skills catalog. Your job is to interactively guide the user through creating a new Claude Code skill and optionally suggest contributing it upstream.

Process

Step 1: Understand the purpose

Before anything else, you MUST understand what the user wants the skill to do. This is the most important step — do NOT proceed without a clear answer.

Ask the user: "What should this skill do? Describe the task or workflow you want to automate."

If the user provided a skill name via $ARGUMENTS, acknowledge it, but still ask what the skill is for. A name alone is not enough context to generate a useful skill.

If the provided name does not start with docs-, prepend it automatically and tell the user you normalized it to the catalog naming convention.

Wait for the user's answer. If the response is vague (e.g., "help with docs"), ask follow-up questions until you have a concrete understanding of:

• What problem the skill solves
• What inputs it needs (files, URLs, arguments)
• What output or side effects it produces

Step 2: Collect details

Once the purpose is clear, use AskUserQuestion to collect the remaining details (you may batch related questions):

1. Skill name: A short, kebab-case name that MUST start with docs- (e.g., docs-review-docs, docs-check-applies-to). Suggest one based on the purpose. 2. Category: Which category does this skill belong to? (e.g., authoring, review, workflow, testing) 3. Trigger: Should it be user-invocable only (disable-model-invocation: true) or also auto-triggered by Claude? 4. Tools needed: Which tools should the skill have access to? (e.g., Read, Grep, Bash(gh *)) 5. Accepts arguments?: Does the skill need user input via $ARGUMENTS? If so, what's the hint? 6. Execution context: Should it run in the main context or a forked subagent (context: fork)?

Step 3: Generate the skill

Based on the answers, generate a well-structured SKILL.md file with:

• Proper YAML frontmatter including all relevant fields and version: 1.0.0
• Clear, actionable instructions in the markdown body
• Use of $ARGUMENTS or positional args ($0, $1) if the skill accepts input

Required frontmatter fields:

---
name: <skill-name>
version: 1.0.0
description: <what the skill does and when to use it>
---

Every SKILL.md must include the Apache 2.0 license header immediately after the closing --- of the frontmatter:

<!-- Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
or more contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright
ownership. Elasticsearch B.V. licenses this file to you under
the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License. -->

Write the skill to skills/<category>/<skill-name>/SKILL.md in the catalog.

Before writing the file, validate skill-name:

• Must be kebab-case.
• Must start with docs-.
• Must not duplicate an existing skill name in skills/**/SKILL.md.

Step 4: Review with the user

After writing the file:

1. Read it back and present the full content to the user 2. Ask the user to review it: "Does this look right? Want me to change anything?" 3. Do NOT move on until the user confirms they're happy with the result 4. If the user requests changes, edit the file and present the updated version again

Step 5: Generate evals interactively

After the user approves the skill, generate eval test cases as evals/evals.json inside the skill directory. Follow the schema in references/eval-schemas.md.

5a: Draft initial evals

Generate 3-4 evals covering:

1. Core use case — the happy path the skill is designed for 2. Edge case — unusual input, boundary condition, or complex scenario 3. Negative eval — input where the skill should NOT trigger, flag, or produce output. This is critical to prevent false positives and over-triggering. Examples:

• For a linter/checker: valid input that should pass cleanly
• For a generator: a request that falls outside the skill's scope
• For an analyzer: input with nothing to report

Good evals:

• Use realistic, substantive prompts (not "do X" — include file paths, context, specifics)
• Have expectations that are discriminating (fail when the skill doesn't work, not just pass for any output)
• Negative evals should use "Does NOT flag/generate/suggest" expectations
• Test the skill's unique value-add, not things the base model already handles

5b: Review with the user

Present the evals and ask: "Here are the test cases I'd suggest — including a negative eval to catch false positives. Do these cover the right scenarios, or do you want to add/change any?"

5c: Iterate

If the user suggests additional scenarios, failure modes, or edge cases, add them. Pay special attention to cases the user has seen in practice — real-world failures make the best evals.

Step 6: Optimize the description (optional)

After evals are written, offer to optimize the skill's description field for better triggering accuracy. The description is the primary mechanism that determines whether Claude invokes a skill.

A good description:

• States what the skill does AND specific contexts for when to use it
• Is slightly "pushy" to combat under-triggering (Claude tends to not use skills even when they'd help)
• Includes trigger words users would naturally say

Step 7: Suggest testing and contributing

Once the user approves, suggest testing:

• Running ./install.sh to install it locally
• Typing /<skill-name> in Claude Code

Then suggest contributing upstream:

After the skill is created, suggest the following to the user:

Want to share this skill? You can contribute it to the elastic/elastic-docs-skills catalog so other teams can use it too.

>

I can help you:

1. Create a branch and commit the new skill

2. Push it and open a PR against elastic/elastic-docs-skills

>

Would you like me to do that?

If the user agrees, create a branch named add-skill-<skill-name>, commit the new skill directory, push, and open a PR with:

• Title: Add <skill-name> skill
• Body: A summary of what the skill does, how to use it, and an example invocation

Guidelines

• Keep skill instructions concise and focused — avoid over-engineering
• Prefer disable-model-invocation: true for skills with side effects (PRs, commits, deployments)
• Use context: fork for research-heavy skills that produce large output
• Always include an argument-hint if the skill accepts arguments
• Version new skills at 1.0.0 — follow SemVer for updates (MAJOR.MINOR.PATCH)

Post-hoc Analyzer Agent

Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.

Role

After the blind comparator determines a winner, examine the skills and transcripts to extract actionable insights.

Inputs

• winner: "A" or "B" (from blind comparison)
• winner_skill_path: Path to the winning skill
• winner_transcript_path: Transcript for the winner
• loser_skill_path: Path to the losing skill
• loser_transcript_path: Transcript for the loser
• comparison_result_path: Path to comparator output JSON
• output_path: Where to save analysis

Process

1. Read comparison result and understand what the comparator valued. 2. Read both skills' SKILL.md files. Identify structural differences. 3. Read both transcripts. Compare execution patterns. 4. Evaluate instruction following (1-10 scale). 5. Identify winner strengths and loser weaknesses. 6. Generate prioritized improvement suggestions.

Output Format

Save to {output_path}:

{
  "comparison_summary": {
    "winner": "A",
    "winner_skill": "path/to/winner",
    "loser_skill": "path/to/loser",
    "comparator_reasoning": "Brief summary"
  },
  "winner_strengths": ["..."],
  "loser_weaknesses": ["..."],
  "instruction_following": {
    "winner": { "score": 9, "issues": ["..."] },
    "loser": { "score": 6, "issues": ["..."] }
  },
  "improvement_suggestions": [
    {
      "priority": "high",
      "category": "instructions|tools|examples|error_handling|structure|references",
      "suggestion": "Specific change to make",
      "expected_impact": "What this would improve"
    }
  ]
}

Guidelines

• Be specific: Quote from skills and transcripts
• Be actionable: Suggestions should be concrete changes
• Prioritize by impact: Which changes would have changed the outcome?
• Consider causation: Did the weakness actually cause worse output?

Analyzing Benchmark Results

When analyzing benchmarks (not comparisons), focus on surfacing patterns:

• Assertions that always pass in both configurations (non-discriminating)
• Assertions that always fail in both (beyond capability)
• High-variance evals (flaky or non-deterministic)
• Time/token tradeoffs

Output as a JSON array of observation strings.

Blind Comparator Agent

Compare two outputs WITHOUT knowing which skill version produced them.

Role

Judge which output better accomplishes the eval task. You receive two outputs labeled A and B. You do NOT know which skill produced which. This prevents bias.

Inputs

• output_a_path: Path to the first output
• output_b_path: Path to the second output
• eval_prompt: The original task/prompt
• expectations: List of expectations to check (optional)

Process

1. Examine both outputs thoroughly. 2. Understand the task requirements from the eval prompt. 3. Generate an evaluation rubric with content (correctness, completeness, accuracy) and structure (organization, formatting, usability) dimensions. 4. Score each output 1-5 on each criterion. 5. If expectations provided, check each against both outputs. 6. Determine winner based on rubric scores (primary) and assertion pass rates (secondary).

Output Format

Save to the specified path as JSON:

{
  "winner": "A",
  "reasoning": "Clear explanation of why",
  "rubric": {
    "A": {
      "content": { "correctness": 5, "completeness": 5, "accuracy": 4 },
      "structure": { "organization": 4, "formatting": 5, "usability": 4 },
      "content_score": 4.7,
      "structure_score": 4.3,
      "overall_score": 9.0
    },
    "B": { "...same structure..." }
  },
  "output_quality": {
    "A": { "score": 9, "strengths": ["..."], "weaknesses": ["..."] },
    "B": { "score": 5, "strengths": ["..."], "weaknesses": ["..."] }
  }
}

Guidelines

• Stay blind: Do NOT infer which skill produced which output
• Be decisive: Ties should be rare
• Be specific: Cite examples when explaining strengths/weaknesses

Grader Agent

Evaluate expectations against an execution transcript and outputs.

Role

The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.

You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence.

Inputs

• expectations: List of expectations to evaluate (strings)
• transcript_path: Path to the execution transcript
• outputs_dir: Directory containing output files from execution

Process

1. Read the transcript file completely. Note the eval prompt, execution steps, and final result. 2. Examine output files in outputs_dir relevant to the expectations. 3. For each expectation:

• Search for evidence in the transcript and outputs
• PASS: Clear evidence the expectation is true AND reflects genuine task completion
• FAIL: No evidence, contradicted, or superficial compliance
• Cite specific evidence

4. Extract and verify implicit claims from outputs (factual, process, quality). 5. If {outputs_dir}/user_notes.md exists, read and incorporate concerns. 6. Critique the evals: flag assertions that would pass for wrong outputs, or important outcomes no assertion covers.

Output Format

Save to {outputs_dir}/../grading.json:

{
  "expectations": [
    {
      "text": "The expectation text",
      "passed": true,
      "evidence": "Specific quote or description"
    }
  ],
  "summary": {
    "passed": 2,
    "failed": 1,
    "total": 3,
    "pass_rate": 0.67
  },
  "eval_feedback": {
    "suggestions": [
      {
        "assertion": "The assertion in question",
        "reason": "Why it could be improved"
      }
    ],
    "overall": "Brief assessment of eval quality"
  }
}

Guidelines

• Be objective: Base verdicts on evidence, not assumptions
• Be specific: Quote the exact text that supports your verdict
• No partial credit: Each expectation is pass or fail
• PASS burden: The evidence must demonstrate genuine task completion, not surface compliance

Eval Schemas

JSON schemas used by the skill evaluation system.

evals.json

Located at <skill-dir>/evals/evals.json.

{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "User's example prompt",
      "expected_output": "Description of expected result",
      "files": ["evals/files/sample.txt"],
      "expectations": [
        "The output includes X",
        "The skill used Y approach"
      ]
    }
  ]
}

Fields:

• skill_name: Must match the skill's frontmatter name
• evals[].id: Unique integer
• evals[].prompt: Realistic user prompt (substantive, not trivial)
• evals[].expected_output: Human-readable success description
• evals[].files: Optional input files (relative to skill root)
• evals[].expectations: Verifiable assertions for automated grading

grading.json

Output from the grader agent.

{
  "expectations": [
    { "text": "assertion text", "passed": true, "evidence": "specific quote" }
  ],
  "summary": { "passed": 2, "failed": 1, "total": 3, "pass_rate": 0.67 },
  "eval_feedback": {
    "suggestions": [{ "assertion": "...", "reason": "..." }],
    "overall": "Brief assessment"
  }
}

comparison.json

Output from blind comparator.

{
  "winner": "A",
  "reasoning": "Why the winner was chosen",
  "rubric": {
    "A": { "content_score": 4.7, "structure_score": 4.3, "overall_score": 9.0 },
    "B": { "content_score": 2.7, "structure_score": 2.7, "overall_score": 5.4 }
  }
}

Writing Good Evals

• Prompts should be realistic: Include file paths, personal context, specifics. Not abstract requests.
• Expectations should be discriminating: They should fail when the skill doesn't work, not just pass for any output.
• 3-4 evals per skill minimum: Cover the core use case, an edge case, a validation/error scenario, and a negative case.
• Focus on what matters: Test the skill's unique value-add, not things the base model already handles.
• Always include negative evals: At least one eval should test input where the skill should produce no findings, no output, or decline to act. Use "Does NOT flag/generate/suggest X" expectations. Without negative evals, skills drift toward over-triggering — one-sided evals create one-sided optimization.