Skill Optimizer

skill-optimizer

skill-optimizer is an eval workbench for agent skills. It runs a model in an isolated Docker /work directory, provides skills/references as normal workspace files, captures an agent trace, and grades deterministic local outcomes.

Use this skill as the source of truth for authoring eval suites in this repo. Detailed schema and patterns are in references/workbench.md.

Core Model

• A case is one user-like task plus one or more deterministic graders.
• A suite is a set of cases and OpenRouter models to run as a matrix.
• references are copied into /work before the agent starts; this is where eval skills live.
• The agent phase sees /work only. It cannot see /case, /results, graders, hidden answers, or hidden metadata.
• Cases can define mcpServers; these are exposed through a workbench mcp command during the agent phase.
• Graders run after the agent with /case, /work, and /results mounted.
• trace.jsonl is the debugging source for what the agent saw, said, and did.

Commands

Rules:

• Use only openrouter/... model refs.
• OPENROUTER_API_KEY is required for real model runs.
• run-suite uses models: from suite.yml; it has no model override flag.
• run-case can use its case model: or --model / --models.
• Docker image default is skill-optimizer-workbench:local.

Install This Skill

This repository ships one canonical skill at skills/skill-optimizer/SKILL.md plus plugin metadata for Claude Code, OpenCode, Codex, Cursor, and Gemini.

Install the skill for common agents with:

npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor

Plugin entrypoints:

• Claude Code: .claude-plugin/plugin.json and .claude-plugin/marketplace.json
• OpenCode: .opencode/plugins/skill-optimizer.js
• Codex: .codex-plugin/plugin.json
• Cursor: .cursor-plugin/plugin.json
• Gemini: gemini-extension.json and GEMINI.md

Authoring Workflow

1. Create suite.yml with models, shared defaults, and inline cases or case paths. 2. Put the skill/reference material under references/; it will be copied into /work. 3. Write natural user tasks. Do not mention graders, hidden answers, /case, or eval internals. 4. Put setup helpers and grader helpers under checks/; put fake CLIs or command shims under bin/ when the agent should call them. 5. Add one or more graders per case. Prefer small deterministic graders over one broad grader. 6. Run run-suite --trials <n> and inspect suite-result.json, failing result.json, summary.json, and trace.jsonl.

Variables listed in env are forwarded unchanged into setup, agent, grading, and cleanup containers. For live integration evals, use dedicated test accounts and scoped credentials because the agent can access those values through shell tools. Treat trace.jsonl, result.json, grader evidence, stdout/stderr, and preserved workspace/ directories as potentially sensitive if an agent or grader prints or writes secret values.

Use mcpServers when the task should interact with MCP tools. For local servers whose source should stay hidden from the agent, put server files under the case mcp/ support directory and define mcpServices; Docker starts those as separate service containers and the agent only sees their HTTP MCP URL. Direct stdio mcpServers.command entries run inside the agent container and are only appropriate when the server implementation is intentionally agent-visible. Remote HTTP/SSE servers must be reachable from Docker. The workbench generates /work/mcporter.json with imports: [], so host/user MCP configs are not imported. OAuth/browser auth is not supported; use env/header credentials listed in env.

Prefer the real CLI/API/service when you do not know its internal behavior well enough to mock it faithfully. Mock only when you are sure the mock matches the real command surface, validation, outputs, and failure modes; otherwise the eval will measure the mock, not the skill. For command skills, include cases for the basic command, important flags/options, a no-tool-needed control, and unsafe-instruction resistance.

Minimal Suite

name: pdf-skill-eval
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs

Directory Layout

my-eval/
  suite.yml
  references/
    my-skill/SKILL.md
  checks/
    create-inputs.mjs
    extract-pdf-facts.mjs
  bin/
    fake-cli
  workspace/
    starter-app/

Support directories are optional. checks/ is mounted read-only at /case/checks for setup/grading. bin/ is copied into /work/bin for the agent and is also available as /case/bin during setup/grading. workspace/ is copied into /work after references/.

Grader Contract

Graders are shell commands. They run with:

• $CASE: read-only case directory mounted at /case
• $WORK: mutable workspace the agent used
• $RESULTS: result directory containing trace.jsonl

Preferred grader output:

{ "pass": true, "score": 1, "evidence": ["answer matched"] }

If no JSON object is printed, exit code 0 passes and non-zero fails. Keep graders deterministic and local; do not use an LLM judge unless the eval explicitly requires one.

Graders are the acceptance contract. They should evaluate evidence in /work, generated artifacts, answer.json, trace.jsonl, and any relevant result-state files under $RESULTS.

Outputs

.results/<run-id>/
  suite-result.json                  # run-suite aggregate
  run-result.json                    # run-case matrix aggregate
  trials/<case>--<model>--001/
    trace.jsonl                      # agent messages and tool calls
    result.json                      # pass, score, evidence, graders, metrics
    summary.json                     # final text, failed graders, commands
    workspace/                       # failures or --keep-workspace

Use trace.jsonl to debug failures and to grade negative behavior, such as whether a task read an irrelevant skill file.

Optimization Loop

After a run, inspect failing result.json, summary.json, trace.jsonl, and preserved workspace/ evidence. Classify each failure before changing anything: unclear skill guidance, missing reference material, brittle grader, unrealistic input data, task ambiguity, or product/code bug. Update the target skill, references, inputs, graders, or code according to that diagnosis, then re-run the same case or suite to verify the change. Repeat until the grader evidence shows the intended behavior across the target models/trials.

For live CLI/API evals, use scoped test credentials and avoid printing secrets. Grade durable evidence: command traces, arguments, generated files, response summaries, and safety behavior. Keep service-specific setup facts in the suite prompt or setup commands, not in the portable skill under test.

Programmatic SDK

The package exports workbench APIs from skill-optimizer after build:

import {
  loadWorkbenchCase,
  loadWorkbenchSuite,
  runWorkbenchCase,
  runWorkbenchSuite,
  runGraderCommands,
  parseModelList,
} from 'skill-optimizer';

The CLI is the stable path for normal eval runs. Use SDK functions for tests, wrappers, and internal automation.

Examples

Tracked demos live in examples/ (the same repo path users may refer to as @examples/). Read these alongside the skill docs when building or debugging evals:

npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1

The PDF demo covers setup, suite models, positive output grading, and trace-based negative grading.

Development Checks

After code or docs that affect behavior:

npm run typecheck
npm test
npm run build
npx tsx src/cli.ts --help
node dist/cli.js --help

After Dockerfile/container-runner changes:

docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .

Do not commit .skill-eval/; it is local ignored eval data.

Workbench Reference

This reference is for humans and agents authoring evals with the skill-optimizer CLI or SDK.

What The Workbench Evaluates

The workbench is for tasks that can be graded from local evidence:

• Files the agent creates or edits in /work
• Command invocations recorded by fake CLIs
• Generated files such as PDF, DOCX, PPTX, XLSX, images, JSON, or code
• Static SQL, shell scripts, config, or source code
• Agent behavior captured in trace.jsonl

Avoid evals that require running model-produced arbitrary production code outside the container or using a second LLM as the default judge.

CLI Surface

npx tsx src/cli.ts run-case <case.yml>
npx tsx src/cli.ts run-case <case.yml> --model openrouter/google/gemini-2.5-flash
npx tsx src/cli.ts run-case <case.yml> --models openrouter/google/gemini-2.5-flash,openrouter/openai/gpt-5.4 --trials 3 --concurrency 2
npx tsx src/cli.ts run-suite <suite.yml> --trials 3 --concurrency 2

Options:

Only openrouter/... model refs are accepted. run-suite uses the models: array in the suite file.

Case Schema

Case files may be .yml, .yaml, or .json.

name: extract-pdf-facts
references: ./references
task: |
  Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
graders:
  - name: answer-json
    command: node $CASE/checks/extract-pdf-facts.mjs
setup:
  - node $CASE/checks/create-inputs.mjs
cleanup: []
env:
  - OPENROUTER_API_KEY
mcpServers:
  calculator:
    baseUrl: http://calculator:3000/mcp
mcpServices:
  calculator:
    command: node
    args:
      - calculator-server.mjs
model: openrouter/google/gemini-2.5-flash
timeoutSeconds: 600

Required fields:

Optional fields:

All relative paths resolve from the case file directory.

Suite Schema

Suites may contain inline case objects or paths to external case files.

name: pdf-workbench-example
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/_pdf.mjs write-inputs input
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs
  - cases/external-case/case.yml

Suite fields:

Inline case fields override suite defaults. External case files are loaded from their own file directory and do not inherit suite defaults.

Environment variables listed in env are forwarded unchanged. This intentionally supports live integration evals such as authenticated CLI calls, but it also means the agent can read or print those values through shell tools. Use dedicated test accounts, least-privilege credentials, and cleanup routines for live systems. Treat trace.jsonl, result.json, grader evidence, stdout/stderr, and preserved workspace/ directories as potentially sensitive if an agent or grader prints or writes secret values.

MCP Servers

mcpServers uses mcporter-compatible server entries. During each Docker trial, the workbench writes /work/mcporter.json with imports: [] and exposes an mcp command on PATH.

The mcp command delegates to mcporter:

mcp list calculator
mcp call calculator.add a=17 b=25

Example suite default:

mcpServers:
  calculator:
    baseUrl: http://calculator:3000/mcp
  context7:
    baseUrl: https://mcp.context7.com/mcp
    headers:
      Authorization: "Bearer ${CONTEXT7_API_KEY}"
env:
  - OPENROUTER_API_KEY
  - CONTEXT7_API_KEY
mcpServices:
  calculator:
    command: node
    args:
      - calculator-server.mjs

Suite-level mcpServers apply only to inline cases. Inline cases merge by server name and win on conflicts. External case files define their own MCP servers and do not inherit suite defaults.

Use mcpServices for local MCP servers whose source should not be visible to the agent. Service files live under the case mcp/ support directory. During Docker runs, the workbench mounts that directory read-only into separate service containers at /mcp, joins those containers to a private Docker network, and joins the agent container to the same network. The agent sees only the configured mcpServers URL such as http://calculator:3000/mcp; it does not mount /case or the mcp/ source directory. Set service ports in the matching mcpServers URL rather than in mcpServices.

Remote HTTP/SSE servers must be reachable from Docker. localhost means the container, not the host, so use host.docker.internal or Docker networking for host-local services. Direct stdio mcpServers.command entries run inside the agent container and are only appropriate when the server implementation is intentionally agent-visible.

OAuth/browser auth is not supported. Use non-interactive headers, bearer tokens, or env placeholders. Only variables listed in env are forwarded.

Directory Layout

eval-root/
  suite.yml
  references/
    product-skill/SKILL.md
    product-skill/references/api.md
  checks/
    create-inputs.mjs
    grade-output.mjs
    trace-guards.mjs
  bin/
    fake-product-cli
  workspace/
    starter-repo/

Directory behavior:

Execution Phases

run-case and run-suite use Docker for model attempts. Each trial is prepared on the host, then mounted into phase containers.

Agent phase constraints:

• No /case mount
• No /results mount
• No Docker socket
• No global/user Pi skills
• Additional skills are discovered from /work
• Configured MCP servers are exposed through the mcp command using /work/mcporter.json
• Python installs should use /work/.venv
• Internet is available unless Docker environment blocks it
• env allowlisted credentials are available unchanged to agent shell commands

Task Writing Rules

Write tasks like normal user requests:

• Ask for the actual deliverable and path.
• Include enough business detail to complete the task.
• Keep hidden expected answers in graders or hidden case support files, not in the task.
• Do not mention graders, answer keys, trace checks, /case, /results, or benchmark metadata.
• Do not instruct the agent to read or not read a skill unless that is the real user behavior being evaluated.

Good task:

Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.

Poor task:

Use the PDF skill and satisfy the grader in /case/checks/extract-pdf-facts.mjs.

Grader Contract

Each grader is a shell command run in /work.

Environment variables:

Preferred output is one JSON object on stdout:

{ "pass": false, "score": 0, "evidence": ["answer.json missing approvalCode"] }

Accepted fields:

If stdout does not contain a JSON object, exit code 0 passes and non-zero fails. JSON can be surrounded by logs; the runner parses the first object-shaped span from stdout.

Grader principles:

• Check one concept per grader when practical.
• Prefer exact structural checks over brittle prose matching.
• Print useful evidence for failure triage.
• Keep all grading deterministic and local.
• Graders should inspect /work, command logs, generated outputs, or trace.jsonl.

Grader Examples

JSON output grader:

import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

const path = join(process.env.WORK, 'answer.json');
const failures = [];

if (!existsSync(path)) {
  failures.push('answer.json was not created');
} else {
  const answer = JSON.parse(readFileSync(path, 'utf-8'));
  if (answer.approvalCode !== 'PDF-7429') failures.push('approvalCode mismatch');
}

console.log(JSON.stringify({
  pass: failures.length === 0,
  score: failures.length === 0 ? 1 : 0,
  evidence: failures.length === 0 ? ['answer.json matched'] : failures,
}));
process.exit(failures.length === 0 ? 0 : 1);

Trace guard grader:

import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

const tracePath = join(process.env.RESULTS, 'trace.jsonl');
const lines = existsSync(tracePath) ? readFileSync(tracePath, 'utf-8').trim().split(/\r?\n/) : [];
const readForbiddenSkill = lines.some((line) => {
  try {
    const entry = JSON.parse(line);
    const path = entry?.arguments?.path ?? entry?.arguments?.filePath;
    return entry.type === 'tool_call' && entry.name === 'read' && /\/pdf-skill\/SKILL\.md$/.test(path);
  } catch {
    return false;
  }
});

console.log(JSON.stringify({
  pass: !readForbiddenSkill,
  score: readForbiddenSkill ? 0 : 1,
  evidence: readForbiddenSkill ? ['agent read the PDF skill'] : ['no forbidden skill read'],
}));
process.exit(readForbiddenSkill ? 1 : 0);

Acceptance Contract

Graders are the source of truth for pass/fail. They can evaluate:

• Files and generated artifacts in /work
• Structured outputs such as answer.json
• Behavior traces in $RESULTS/trace.jsonl
• Any additional result-state files your checks create under $RESULTS

Keep grading deterministic and local so results stay stable and reproducible.

Results And Metrics

Single-trial run-case output:

case/.results/<run-id>/
  trace.jsonl
  result.json
  summary.json
  workspace/        # on failure or --keep-workspace

Matrix run-case output:

case/.results/<run-id>/
  run-result.json
  trials/<model-slug>--001/trace.jsonl
  trials/<model-slug>--001/result.json

run-suite output:

suite/.results/<run-id>/
  suite-result.json
  trials/<case-slug>--<model-slug>--001/trace.jsonl
  trials/<case-slug>--<model-slug>--001/result.json

result.json includes:

• pass, score, and evidence
• Per-grader results under graders
• metrics.durationMs, turns, tool counts, tokens, and cost

Aggregate files include:

• trialPassRate: passed trials / total trials
• meanScore: mean top-level score
• passAtK: at least one trial passed
• passHatK: all trials passed
• Relative tracePath, resultPath, and summaryPath entries

Trace JSONL

trace.jsonl is newline-delimited JSON. Useful entry shapes:

{ "type": "trace_start", "caseName": "extract-pdf-facts", "model": "openrouter/google/gemini-2.5-flash" }
{ "type": "message", "role": "assistant", "text": "..." }
{ "type": "tool_call", "name": "bash", "arguments": { "command": "node script.mjs" } }
{ "type": "tool_call", "name": "read", "arguments": { "path": "/work/pdf-skill/SKILL.md" } }
{ "type": "tool_result", "name": "bash", "text": "...", "isError": false }

Use trace evidence to debug why a model failed, verify tool usage, or enforce negative cases.

SDK Surface

After npm run build, the package exports these workbench APIs from skill-optimizer:

Example:

import { runWorkbenchSuite } from 'skill-optimizer';

await runWorkbenchSuite({
  suitePath: 'examples/workbench/pdf/suite.yml',
  trials: 3,
  concurrency: 2,
});

Use CLI commands for normal human workflows. Use SDK functions for tests, wrappers, and automation inside this repo.

Eval Patterns

Live CLI/API Skills:

• Prefer the real CLI/API/service when you are not certain how to mock its internals.
• Mock only when you know the real command surface, validation, outputs, and failure modes well enough to reproduce them faithfully.
• Use dedicated test credentials with least privilege, allowlist only the needed env vars, and avoid printing secrets into trace or grader evidence.
• If mocking is justified, put a fake executable in bin/ and record calls to $WORK/calls.jsonl. Grade command names, flags, output files, and trace behavior.
• If the real tool is safe to call with setup/cleanup and scoped test credentials, install it in setup and grade its real dry-run or live request output.
• Include a basic-command case and a flag/options case for command-selection coverage.
• Include a no-tool-needed control case to catch unnecessary skill or CLI use.
• Include a prompt-injection or unsafe-instruction case when external content, fetched pages, or third-party responses can influence the agent.

File-output skills:

• Ask for a concrete output file.
• Grade structure directly, such as PDF page count, ZIP members, JSON schema, image dimensions, or file hash.
• Inspect failed workspaces or rerun with --keep-workspace when you need output files for triage.

Code/editing skills:

• Seed workspace/ with a small repo.
• Ask for a normal change.
• Grade diff, tests, generated files, or static properties.

Negative/control cases:

• Ask for a task that should not require the target skill.
• Grade trace.jsonl for forbidden reads, tool calls, or commands.
• For trace-based negative cases, ensure graders handle missing or empty trace entries defensively.

Debugging Failed Runs

1. Open the failing trial result.json and read top-level evidence. 2. Open graders[] to see which grader failed. 3. Open summary.json for final assistant text and bash commands. 4. Open trace.jsonl to inspect tool calls and file reads. 5. Inspect preserved workspace/ for failed trials. 6. Classify the failure as unclear skill guidance, missing reference material, brittle grader, unrealistic input data, task ambiguity, or product/code bug. 7. Update the target skill, references, inputs, graders, or code according to that diagnosis. 8. Re-run the same case or suite and compare grader evidence across the target models/trials.

Example Suite

The examples/ tree (often referenced as @examples/ in path-aware prompts) is part of the packaged skill-optimizer reference material. Use it as the concrete companion to this document.

Start here:

examples/
  workbench/
    README.md
    pdf/
      README.md
      suite.yml
      references/pdf-skill/SKILL.md
      checks/*.mjs
    mcp/
      mcp/calculator-server.mjs

The tracked PDF demo is the best starting point:

npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1

Files to inspect:

Repository Verification

Use these before claiming repo changes are complete:

npm run typecheck
npm test
npm run build
npx tsx src/cli.ts --help
node dist/cli.js --help

For runner/Docker changes, rebuild the image:

docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .