Eval Mcp

Evaluate MCP Tools

Tool descriptions are prompt engineering — they land directly in Claude's context window and determine whether Claude picks the right tool with the right arguments. This skill makes tool quality measurable and improvable instead of guesswork.

Three levels of testing, each building on the last: 1. Static Analysis — deterministic schema quality checks (no Claude calls) 2. Selection Testing — does Claude pick the right tool for each intent? 3. Description Optimization — iterative improvement based on confusion patterns

When to Apply

• User wants to check if their MCP tool schemas are well-designed
• User wants to test whether Claude selects the right tools for user intents
• User is debugging tool confusion (Claude picks the wrong tool)
• User wants to optimize tool descriptions for better selection accuracy
• User has finished scaffolding with build-mcp-server and wants to validate quality

Workflow Overview

Phase 1: Connect → Phase 2: Static Analysis → Phase 3: Selection Testing → Phase 4: Optimize
                                                            ↑__________________________|

Phase 4 loops back: apply rewrites → refetch schemas → retest → compare accuracy.

Prerequisites

• Node.js >= 18 — required for the MCP Inspector CLI (npx)
• jq — required for schema analysis scripts
• A running MCP server — the server must respond to tools/list. Use build-mcp-server/scripts/test-server.sh to verify connectivity first.

---

Phase 1 — Connect & Inventory

Connect to the user's MCP server and fetch the tool schemas.

1a: Get connection details

Ask the user how to reach their server:

• HTTP/SSE: URL (e.g., http://localhost:3000/mcp)
• stdio: spawn command (e.g., node dist/server.js)

1b: Fetch tool schemas

bash scripts/fetch-tools.sh <url-or-command> <transport> <workspace>/tools.json

This calls tools/list via the MCP Inspector CLI and saves the schemas.

1c: Display inventory

Show a summary table:

| # | Tool | Description (preview) | Params | Annotations |
|---|------|-----------------------|--------|-------------|
| 1 | search_issues | Search issues by keyword... | 3 | readOnlyHint |
| 2 | create_issue | Create a new issue... | 4 | — |

Flag tool count: 1-15 optimal, 15-30 warning, 30+ excessive (consider search+execute pattern).

1d: Create workspace

Create workspace at {server-name}-eval/ adjacent to the skill directory or in the user's project:

{server-name}-eval/
├── tools.json
├── evals/
│   └── evals.json
└── iteration-N/

---

Phase 2 — Static Analysis

Run deterministic quality checks — no Claude calls needed. This gives immediate feedback during development.

2a: Run analysis

bash scripts/analyze-schemas.sh <workspace>/tools.json <workspace>/iteration-N/static-analysis.json

2b: Display results

Show per-tool quality scores. Read `references/quality-checklist.md` for the criteria being checked.

| Tool | Desc | Params | Schema | Annotations | Overall | Issues |
|------|------|--------|--------|-------------|---------|--------|
| search_issues | 3/3 | 3/3 | 2/3 | 2/3 | 2.5 | No negation |
| create_issue | 1/3 | 1/3 | 0/3 | 0/3 | 0.5 | 4 issues |

2c: Flag sibling pairs

If the analysis found tools with high description overlap, highlight them as confusion risks:

### Sibling Pairs (confusion risk)
| Tool A | Tool B | Overlap | Risk |
|--------|--------|---------|------|
| search_issues | list_issues | 52% | HIGH |

2d: Decision point

If critical issues exist (missing descriptions, zero annotations), recommend fixing them before Phase 3. Static issues create noise in selection testing — fix the obvious problems first, then measure the subtle ones.

If all tools score well, proceed to Phase 3.

---

Phase 3 — Selection Testing

Test whether Claude picks the right tool for each user intent. This is the core eval.

3a: Generate test intents

Read `references/eval-patterns.md` for intent generation patterns.

For each tool, generate:

• 3 should-trigger intents — direct, implicit, and casual phrasings
• 2 should-not-trigger intents — near-miss and keyword overlap

For each sibling pair flagged in Phase 2:

• 1 disambiguation intent per tool — tests whether Claude picks the RIGHT sibling

Present all intents to the user for review. Ask if any should be added, removed, or modified.

3b: Save intents

Save to {workspace}/evals/evals.json:

{
  "server_name": "my-server",
  "generated_from": "tools.json",
  "intents": [
    {
      "id": 1,
      "intent": "Are there any open bugs related to checkout?",
      "expected_tool": "search_issues",
      "type": "should_trigger",
      "target_tool": "search_issues",
      "notes": "Implicit intent — doesn't name the action"
    }
  ]
}

3c: Run selection tests

For each intent, spawn a subagent that receives: 1. The full tool schemas from tools.json (formatted as they'd appear in Claude's context) 2. The user intent text 3. Instructions to select exactly one tool and provide arguments, or decline if no tool fits

The subagent prompt:

You have access to the following MCP tools:

{tool schemas as JSON}

A user sends this message:
"{intent text}"

Which tool would you call? Respond with JSON:
{
  "selected_tool": "tool_name" or null,
  "arguments": { ... } or {},
  "reasoning": "One sentence explaining your choice"
}

If no tool fits the user's request, set selected_tool to null.
Select exactly ONE tool. Do not suggest calling multiple tools.

Save each result to {workspace}/iteration-N/selection/intent-{ID}/result.json.

Launch all selection tests in parallel for efficiency.

3d: Grade results

bash scripts/grade-selection.sh \
  <workspace>/iteration-N/selection \
  <workspace>/evals/evals.json \
  <workspace>/iteration-N/benchmark.json

3e: Display results

## Selection Results — Iteration N

**Accuracy:** 82% (41/50 correct)

| Metric | Count |
|--------|-------|
| Correct | 41 |
| Wrong tool | 5 |
| False accept | 2 |
| False reject | 2 |

### Per-Tool Accuracy
| Tool | Precision | Recall |
|------|-----------|--------|
| search_issues | 0.90 | 0.85 |
| create_issue | 1.00 | 1.00 |

### Worst Confusions
| Expected | Selected Instead | Times |
|----------|-----------------|-------|
| list_issues | search_issues | 3 |
| get_user | find_user_by_email | 2 |

---

Phase 4 — Optimize & Iterate

Analyze confusion patterns and suggest description improvements. Read `references/optimization.md` for rewrite patterns.

4a: Analyze confusions

For each confused pair (from worst_confusions): 1. Read both tools' current descriptions 2. Identify why they're confusing (missing negation, overlapping scope, no cross-reference) 3. Draft a specific rewrite following the disambiguation patterns in optimization.md

4b: Present suggestions

## Suggested Improvements

### search_issues ↔ list_issues (confused 3 times)

**search_issues — Before:**
> Search issues by keyword.

**search_issues — After:**
> Search issues by keyword across title and body. Returns up to `limit` results ranked by relevance. Does NOT filter by status, assignee, or date — use list_issues for structured filtering.

**Reason:** Adding scope boundary and cross-reference to disambiguate from list_issues.

Save to {workspace}/iteration-N/suggestions.json (format defined in optimization.md).

4c: Apply and retest

After the user applies the rewrites to their server code:

1. Restart the server 2. Re-run Phase 1 to refetch tools.json (descriptions may have changed) 3. Re-run Phase 2 for updated static analysis 4. Re-run Phase 3 into iteration-N+1 using the same evals.json 5. Compare accuracy:

## Iteration Comparison

| Metric | Iteration 1 | Iteration 2 | Delta |
|--------|------------|------------|-------|
| Accuracy | 82% | 94% | +12% |
| search↔list confusion | 3 | 0 | -3 |

4d: Iteration guidance

• Change one sibling pair per iteration so you can attribute improvements
• If accuracy plateaus, the remaining confusions may need architectural changes (merging tools, renaming, or restructuring the tool surface)
• Stop when accuracy exceeds 90% or when remaining confusions are in ambiguous edge cases that humans would also struggle with

---

Reference Files

Read these when you reach the relevant phase — not upfront:

• `references/quality-checklist.md` — Testable quality criteria for tool schemas (Phase 2)
• `references/eval-patterns.md` — How to write tool selection test intents (Phase 3)
• `references/optimization.md` — How to improve descriptions from eval results (Phase 4)

Related Skills

• build-mcp-server — Design and scaffold MCP servers (run this first, then eval-mcp to validate)
• build-mcp-app — MCP servers with interactive UI widgets

Gotchas

No known gotchas yet. Append entries as they're discovered during use.

Tool Selection Eval Patterns

How to generate test intents that measure whether Claude picks the right tool. Read this before Phase 3 of the eval workflow.

---

Should-Trigger Intents

Intents that should cause a specific tool to be selected. Generate 3 per tool, varying formality and directness.

Direct Request

Name the action explicitly. Easiest to get right — tests basic description matching.

Tool: create_issue
Intent: "Create a new issue titled 'Login timeout on mobile'"

Implicit Intent

Describe the need without naming the action. Tests whether the description captures the use case, not just keywords.

Tool: search_issues
Intent: "Are there any open bugs related to the checkout flow?"

Casual / Domain Jargon

Use informal language or domain-specific terms. Tests description robustness beyond formal phrasing.

Tool: update_issue
Intent: "Can you bump issue 42 to high priority? It's blocking the release."

Key rule

Never use the tool's exact name in the intent. You're testing whether the description drives selection, not whether Claude can pattern-match on the tool name.

---

Should-NOT-Trigger Intents

Intents that should cause Claude to decline (select no tool). Generate 2 per tool, testing different failure modes.

Near-Miss (Adjacent Capability)

Share vocabulary with a real tool but request something it can't do.

Available: search_issues (searches issues by keyword)
Intent: "Search the pull requests for mentions of the auth refactor"
Why decline: No PR search tool exists. The keyword "search" overlaps but the domain is wrong.

Keyword Overlap (Wrong Semantics)

Use the same verbs/nouns but for a fundamentally different operation.

Available: create_issue, add_comment
Intent: "Create a new Slack channel for the team"
Why decline: "Create" overlaps but the target system is completely different.

Beyond Capability

Request something no tool in the set can do.

Available: [issue tracker tools]
Intent: "Deploy the latest build to staging"
Why decline: No deployment tools exist in this server.

---

Disambiguation Intents

For tool pairs flagged as siblings (high description overlap), generate intents that test whether Claude picks the RIGHT sibling. These are the highest-value tests.

Input-Type Disambiguation

Tools: get_user (by ID), find_user_by_email (by email)
Intent: "Look up the user with email john@example.com"
Expected: find_user_by_email (not get_user)
Why: Tests whether descriptions clarify which input type each accepts.

Scope Disambiguation

Tools: search_issues (keyword search), list_issues (browse with filters)
Intent: "Show me all critical issues from this week"
Expected: list_issues (filter by severity + date, not keyword search)
Why: Tests whether "search" vs "list/filter" distinction is clear.

Action Disambiguation

Tools: update_issue (modify fields), add_comment (append note)
Intent: "Add a note to issue 42 saying the fix is deployed"
Expected: add_comment (not update_issue)
Why: Tests whether "add a note" maps to comment, not field update.

---

Edge Cases

Include 2-3 edge cases per eval set.

Empty / Meaningless Intent

Intent: ""
Expected: none (decline)

Multi-Tool Intent

Intent: "Find the issue about login bugs and add a comment saying it's fixed"
Expected: search_issues (first tool in the sequence)
Note: Selection tests pick ONE tool. Multi-step sequences are out of scope — the test validates the first selection.

Negated Intent

Intent: "Don't create a new issue, just search for existing ones about login"
Expected: search_issues (not create_issue)
Why: Tests whether Claude parses negation correctly and doesn't keyword-match on "create."

---

Intent Writing Guidelines

1. Natural language only. Write as a real user would — include context, vary formality, allow typos if natural. 2. Never use the tool name. Test description quality, not name matching. "Run the search_issues tool" is useless. 3. One intent, one expected tool. Each intent maps to exactly one tool (or none). Multi-tool intents test the first step. 4. Include domain context. "Find the login bug" is generic; "Check if there's already a ticket for the 504 errors on /api/checkout" is realistic. 5. Vary across tools. Don't cluster all intents on the most obvious tool. Less-used tools often have the worst descriptions.

---

How Many Intents?

Keep total under 60 for practical cost. Each intent is one Claude call.

Tool Description Optimization

How to improve MCP tool descriptions based on eval results. Read this during Phase 4 when analyzing confusion patterns and drafting rewrites.

---

The Optimization Loop

Confusion matrix → Identify worst pairs → Rewrite descriptions → Retest → Compare

Each iteration should improve accuracy on the confused pairs without regressing on tools that already work. If a rewrite fixes one confusion but creates another, the descriptions need a different framing.

---

Disambiguation Patterns

Cross-Reference Siblings

When two tools get confused, each description should say when to use the OTHER:

Before (confusing):

get_user — Fetch a user from the database.
find_user — Look up a user in the system.

After (disambiguated):

get_user — Fetch a user by their ID (e.g., usr_abc123). If you only have an email address, use find_user_by_email instead.
find_user_by_email — Look up a user by email address. Returns null if not found. If you already have the user's ID, use get_user.

The key move: each description tells Claude exactly when this tool is wrong and which tool is right instead.

Input-Type Routing

When tools accept different identifier types, make the input type the first distinguishing feature:

get_order — Fetch an order by its order ID (format: ORD-XXXXXX). Returns full order details including line items and shipping status.
lookup_order_by_email — Find all orders for a customer email address. Returns a list of order summaries (ID, date, total) — use get_order for full details.

Scope Boundaries

When tools operate on different scopes of the same entity, define the boundary:

search_issues — Full-text keyword search across issue title and body. Returns up to `limit` results ranked by relevance. Does NOT filter by status, assignee, or date — use list_issues for structured filtering.
list_issues — Browse issues with structured filters (status, assignee, priority, date range). Returns a paginated list sorted by date. For keyword/text search, use search_issues.

---

Negation Patterns

Explicit Exclusion

State what the tool does NOT do when an adjacent capability exists:

search_issues — Search issues by keyword. Does NOT search comments or pull requests — use search_comments / search_prs for those.

Capability Boundary

Define the edge of what the tool can do:

get_issue — Fetch issue metadata (title, status, assignee, labels). Does NOT include the comment thread — use list_comments for that.

System Boundary

When a tool only works within one system:

create_issue — Create a new issue in the project tracker. Does NOT create tickets in Jira or Slack threads — this only operates on the internal tracker.

---

Return Shape Documentation

State the Shape

Tell Claude exactly what comes back:

search_items — Returns a JSON array of {id, title, score} objects, up to `limit` results.

State Truncation

When results can be large, describe the truncation behavior:

Returns up to 50 results. If more exist, the response includes `hasMore: true` and a `cursor` for pagination.

State What's NOT Returned

Prevent follow-up confusion:

Returns issue metadata only (title, status, labels). Does not include the full body or comments — use get_issue for the body and list_comments for the thread.

---

Recovery Hint Patterns

Next-Step Hints

When a tool call fails, the error should suggest what to do next:

if (!item) {
  return {
    isError: true,
    content: [{
      type: "text",
      text: "Item 'xyz' not found. Use search_items to find valid IDs."
    }]
  };
}

Alternative Suggestion

When the user's intent doesn't match this tool:

"This tool only searches by keyword. For filtering by date or status, use list_items with the appropriate filters."

---

The suggestions.json Format

Phase 4 outputs improvement suggestions in this format:

{
  "iteration": 1,
  "suggestions": [
    {
      "tool": "search_issues",
      "field": "description",
      "before": "Search issues by keyword.",
      "after": "Search issues by keyword across title and body. Returns up to `limit` results ranked by relevance. Does NOT filter by status or assignee — use list_issues for structured filtering.",
      "reason": "Confused with list_issues 40% of the time. Adding scope boundary and cross-reference should disambiguate.",
      "confused_with": "list_issues",
      "confusion_rate": 0.4
    }
  ]
}

Fields:

• tool: The tool to modify
• field: Which field to change (description or a param description like param:status)
• before: Current text
• after: Proposed replacement
• reason: Why this change should help (linked to eval data)
• confused_with: The sibling tool causing confusion (if applicable)
• confusion_rate: How often this tool was selected when the sibling was expected

---

Applying Rewrites

1. The suggestions.json file contains before/after text for each tool 2. Find the tool registration in your server source code 3. Replace the description string with the after text 4. Restart the server 5. Re-run Phase 1-3 of eval-mcp into a new iteration 6. Compare accuracy: the confusion rate for the modified pair should drop

Do not apply all suggestions at once. Change one sibling pair per iteration so you can attribute accuracy changes to specific rewrites. If you change everything at once, you can't tell which rewrites helped and which hurt.

MCP Tool Quality Checklist

Testable criteria for MCP tool schemas. Each check has a pass/fail threshold used by scripts/analyze-schemas.sh. Source of truth for these criteria is build-mcp-server/references/tool-design.md.

---

Description Quality

DQ-1: Description Length

Pass: >= 20 characters Fail: Missing or under 20 characters Why: Short descriptions like "Searches for issues" give Claude nothing to disambiguate. Descriptions must say what the tool does, what it returns, and what it doesn't do.

DQ-2: States What the Tool Does

Pass: Description contains an action verb describing the operation Fail: Description is a noun phrase or lacks a clear action Why: Claude reads descriptions as contracts. "Issue searcher" is ambiguous; "Search issues by keyword across title and body" is actionable.

DQ-3: States What the Tool Returns

Pass: Description mentions return/output format (contains "return", "output", "result", "produce", "respond") Fail: No mention of what comes back Why: Claude needs to know whether a tool returns a list, a single item, a confirmation, or structured data to use it correctly in multi-step workflows.

DQ-4: Disambiguates from Siblings

Pass: Description mentions what the tool does NOT do, or when to use a different tool Fail: No negation or cross-reference Why: When two tools overlap, each description should say when to use the OTHER one. Without this, Claude picks based on keyword similarity and frequently confuses siblings.

---

Parameter Schema Quality

PS-1: All Parameters Have Descriptions

Pass: 100% of parameters have a description field Fail: Any parameter lacks a description Why: The .describe() text shows up in the schema Claude sees. Omitting it forces Claude to guess what the parameter means from the name alone.

PS-2: String Parameters Use Constraints

Pass: String parameters use enum, pattern (regex), or format where applicable Fail: Bare string type for a constrained value (IDs, statuses, categories) Why: Tight schemas prevent bad calls. z.string() for an ID lets Claude pass anything; z.string().regex(/^usr_[a-z0-9]{12}$/) validates at the schema level.

PS-3: Number Parameters Have Bounds

Pass: Number parameters have minimum, maximum, or default Fail: Unbounded number with no guidance Why: Without bounds, Claude may pass 0, -1, or 999999. Bounds prevent nonsensical values and communicate intent.

PS-4: Optional Parameters Document Defaults

Pass: Optional parameters describe their default behavior Fail: Optional parameter with no hint about what happens when omitted Why: Claude decides whether to include optional params based on the description. "Defaults to the caller's workspace" tells Claude it can usually skip this param.

---

Annotation Coverage

AC-1: Tool Has At Least One Annotation

Pass: annotations object exists with at least one hint Fail: No annotations at all Why: Annotations drive host UX (auto-approve for readonly, confirm for destructive). Missing annotations means the host assumes worst case.

AC-2: Read-Only Tools Marked

Pass: Tools that don't modify state have readOnlyHint: true Fail: Read-only tool without the annotation Why: Hosts may auto-approve read-only calls, reducing user friction. Missing annotation means unnecessary confirmation prompts.

AC-3: Destructive Tools Marked

Pass: Tools that delete or overwrite have destructiveHint: true Fail: Destructive tool without the annotation Why: Hosts show confirmation dialogs for destructive tools. Missing annotation means a delete could execute without warning.

---

Tool Set Quality

TS-1: Tool Count in Range

Pass: 1-15 tools (optimal), 15-30 (acceptable with warning) Fail: 0 tools or 30+ tools Why: Every tool schema consumes tokens in Claude's context window. 30 tools with rich schemas can eat 3-5k tokens before the conversation starts. Over 30, switch to search+execute.

TS-2: No High Sibling Overlap

Pass: No two tools share > 50% of non-stopword description tokens Fail: A pair exceeds the overlap threshold Why: High overlap means Claude will confuse the pair. Either merge them, rename them, or add disambiguation to both descriptions.

TS-3: Similar Tools Cross-Reference

Pass: Tools flagged as siblings in TS-2 reference each other in descriptions Fail: Similar tools exist without cross-references Why: The pattern from tool-design.md: get_user — Fetch by ID. If you only have an email, use find_user_by_email.

---

Error Handling

EH-1: Descriptions Mention Error Cases

Pass: Description or return docs mention at least one error condition Fail: No mention of failure modes Why: Tools that only describe the happy path leave Claude guessing when things fail. "Returns null if not found" is more useful than silence.

EH-2: Error Returns Include Recovery Hints

Pass: Error responses use isError: true with a next-step suggestion Fail: Bare error message or transport exception Why: The hint turns a dead end into a next step: "Item not found. Use search_items to find valid IDs."

Note: EH-2 is only testable at runtime, not via static schema analysis.