Vitest Evals

vitest-evals

Use the harness-backed API as the only authoring model.

First Steps

1. Read the package, app, or eval file being changed. 2. Identify the runtime target, then open only the needed reference. 3. Keep suites close to Vitest: one harness per describeEval(...), explicit run(...) inside each test, ordinary expect(...) assertions over the returned result.

Reference Router

Runtime Defaults

• Import describeEval(...), judges, and helpers from vitest-evals.
• Bind exactly one harness to a suite.
• Call run(input) where the test should execute the system.
• Assert on result.output for app-facing behavior.
• Use toolCalls(result) and message helpers for normalized session assertions.
• Use spans(result), spansByKind(result, kind), and failedSpans(result) for span assertions.
• Keep HarnessRun, NormalizedSession, usage, artifacts, and tool records JSON-serializable.
• Keep judge model calls on judges. Use createJudge("Name", assess) for

custom judges; use the provider-helper overload only when multiple judges reuse setup and need curried run options.

• Put scenario-owned criteria on the input value. Put direct-check expected

values in Vitest case rows. Pass per-case judge criteria through explicit matcher options, and suite-wide criteria through judge config.

• Custom judges should use createJudge(...) for stable reporter labels.

Verification

Prefer the smallest command that covers the edited files:

vitest-evals Skill

Install the whole skills/vitest-evals directory as one skill. Keep SKILL.md, SPEC.md, SOURCES.md, and references/ together because runtime instructions use relative reference paths.

This skill is intentionally focused on the harness-backed vitest-evals API: suite authoring, custom harnesses, first-party AI SDK and Pi AI harnesses, judges, replay, normalized run data, and targeted verification.

Custom Harness

Open this when no first-party harness adapter fits the application runtime.

Contract

import {
  normalizeContent,
  normalizeMetadata,
  toJsonValue,
  type Harness,
  type HarnessRun,
} from "vitest-evals/harness";

const appHarness: Harness<AppInput, AppOutput> = {
  name: "app",
  run: async (input, context): Promise<HarnessRun<AppOutput>> => {
    const appResult = await runApp(input, {
      signal: context.signal,
    });

    const output = toJsonValue(appResult.decision);

    return {
      output,
      session: {
        messages: [
          { role: "user", content: normalizeContent(input) },
          {
            role: "assistant",
            content: normalizeContent(appResult.reply),
            metadata: normalizeMetadata({ channel: appResult.channel }),
          },
        ],
        metadata: normalizeMetadata({ caseId: appResult.caseId }),
      },
      usage: appResult.usage ?? {},
      errors: [],
    };
  },
};

Required Fields

Implementation Rules

• Run the app through its normal entrypoint.
• Inject context.signal and test doubles where the app supports them.
• Use context.setArtifact(name, value) for JSON-safe diagnostics that should appear on the run.
• Convert unknown values with toJsonValue(...), normalizeContent(...), or normalizeMetadata(...).
• Attach a partial run to thrown errors with attachHarnessRunToError(...) when meaningful trace data exists.
• Put text that should be judged as text in run.output; put transcript text in assistant messages.

Choose A Custom Harness When

AI SDK Harness

Open this for ai package integrations such as generateText, generateObject, tools, or AI SDK-style agents.

Install And Import

npm install -D ai vitest-evals @vitest-evals/harness-ai-sdk

import { aiSdkHarness, type AiSdkToolset } from "@vitest-evals/harness-ai-sdk";

Default generateText Pattern

import { generateText, stepCountIs } from "ai";

const harness = aiSdkHarness({
  tools,
  run: ({ input, runtime }) =>
    generateText({
      model,
      prompt: input,
      tools: runtime.tools,
      stopWhen: stepCountIs(5),
      temperature: 0,
    }),
  output: ({ result }) => parseDomainOutput(result.text),
});

Options

Agent factories receive { input, context } before execution so apps can derive instructions or seeded state without side-channel setup.

When LLM-backed judges need shared provider setup, keep that helper in the judge module. Do not put judge-model calls on the app harness:

const verdict = await generateText({
  model: rubricModel,
  prompt: formatJudgePrompt(ctx),
}).then((result) => result.text);

Normalization Behavior

• Reads steps to build assistant messages, tool calls, tool results, and metadata.
• Reads totalUsage or usage; if absent, aggregates step usage where available.
• Preserves provider and model from the last step when present.
• Uses result.session or result.trace directly when they already match NormalizedSession.
• Adds runtime tool calls that were executed through wrapped tools but did not appear in provider steps.
• Attaches partial runs to thrown errors so failed executions can still expose tool traces.

Tool Rules

• Define tool inputs and outputs so they can be normalized to JSON.
• Set replay: true or a replay config only on tools with execute(...).
• Provider-executed tools without execute(...) cannot be recorded automatically.
• Replay rejects async iterable tool outputs; return JSON-safe values for recorded tools.

Use Cases

Pi AI Harness

Open this for Pi AI or Pi Mono-style agents.

Install And Import

npm install -D vitest-evals @vitest-evals/harness-pi-ai

import { piAiHarness, type PiAiRuntime, type PiAiToolset } from "@vitest-evals/harness-pi-ai";

Default Agent Pattern

const harness = piAiHarness({
  agent: () => createRefundAgent(),
});

agent can be an existing agent instance or a per-run factory. Factories receive { input, context } when per-run instructions, native tool closures, or seeded artifacts are needed before instrumentation.

If the agent needs a custom entrypoint:

const harness = piAiHarness({
  agent: () => createRefundAgent(),
  run: ({ agent, input, runtime }) => agent.execute(input, runtime),
});

Options

Runtime Surface

run receives { agent, input, context, runtime }.

Tool Inference

• The harness looks for runtime toolsets on the agent and nested agent, state, or initialState objects.
• It also instruments native tool arrays with { name, execute }.
• Instrumentation is restored after a run and reapplied after agent resets.
• Explicit tools overrides runtime tools but native tool tracing can still be captured.

Normalization Behavior

• Starts sessions with a user message for the input.
• Uses recorded runtime events as normalized messages.
• Adds tool calls from wrapped runtime tools and native tool instrumentation.
• Reads usage from usage or metrics, and adds tool call counts when absent.
• Uses result.session or result.trace directly when they already match NormalizedSession.
• Attaches partial runs to thrown errors.

Use Cases

Judges And Assertions

Open this when adding or reviewing judges, suite-level scoring, or explicit toSatisfyJudge(...) assertions.

Judge Context

Every judge receives:

Custom Judge Pattern

import {
  createJudge,
} from "vitest-evals";

const RefundRubricJudge = createJudge<
  string,
  RefundOutput,
  { expectedStatus: string }
>(
  "RefundRubricJudge",
  async (ctx) => {
    const verdict = await callJudgeModel({
      prompt: formatRubric({
        input: ctx.input,
        output: ctx.output,
        expectedStatus: ctx.expectedStatus,
      }),
    });

    return parseVerdict(verdict);
  },
);

Automatic Vs Explicit

Built-In Judges

Matcher Context Rules

• expect(result).toSatisfyJudge(...) reuses the fixture-backed run context.
• expect(result.output).toSatisfyJudge(...) can reuse the exact run when the output object came from that run.
• Raw values outside an eval context need explicit input, session, run,

harness, or custom judge params when the judge depends on them.

• Normalized sessions infer output from the latest assistant message content.
• Calling ctx.harness.run(...) inside a judge executes the app again; only do this when the judge intentionally needs a second run.

Review Checklist

• Custom judges should usually use createJudge("Name", assess).
• Scores are numbers or null; failure behavior is controlled by thresholds.
• Rationale or parsed judge output is placed under metadata.
• LLM-backed judges provide the judge prompt/rubric text. Shared provider setup

belongs in a judge-side helper, not on the app harness.

Suite Authoring

Open this when writing or reviewing a normal harness-backed eval suite.

Required Shape

import { expect } from "vitest";
import { describeEval, toolCalls } from "vitest-evals";

describeEval("refund agent", { harness }, (it) => {
  it("approves a refundable invoice", async ({ run }) => {
    const result = await run("Refund invoice inv_123");

    expect(result.output).toMatchObject({ status: "approved" });
    expect(toolCalls(result).map((call) => call.name)).toEqual([
      "lookupInvoice",
      "createRefund",
    ]);
  });
});

Suite Rules

Core Options

Normalized Run Checklist

• run.output is JSON-serializable or omitted.
• run.session.messages contains user, assistant, and tool records worth reporting.
• run.usage includes provider/model/token/tool data when available.
• run.artifacts contains only JSON-safe diagnostics set through context.setArtifact(...).
• run.errors is an array, even on success.

Reporter Setup

For direct eval file runs, include the custom reporter:

pnpm exec vitest run path/to/file.eval.ts -c vitest.config.ts --reporter=./packages/vitest-evals/src/reporter.ts

Tool Replay

Open this when configuring or debugging tool recording and replay.

Vitest Env

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    env: {
      VITEST_EVALS_REPLAY_MODE: "auto",
      VITEST_EVALS_REPLAY_DIR: ".vitest-evals/recordings",
    },
  },
});

Modes

Tool Opt-In

AI SDK:

const tools = {
  lookupInvoice: {
    replay: true,
    inputSchema,
    execute: lookupInvoice,
  },
};

Pi AI:

const tools = {
  lookupInvoice: {
    replay: true,
    execute: lookupInvoice,
  },
};

Replay Config

Use an object when the default cache key or recording needs adjustment:

replay: {
  version: "v2",
  key: (args) => ({
    args,
    tenant: args.tenant,
  }),
  sanitize: (recording) => ({
    ...recording,
    metadata: undefined,
  }),
}

Recording Rules

• Recordings live under <replay-dir>/<tool-name>/<cache-key>.json.
• Cache keys include tool name, normalized key input, and optional version.
• Inputs and outputs must be JSON-serializable.
• Tool errors are recorded and replayed as errors.
• Replay metadata is attached to tool call metadata as metadata.replay.
• Recorded metadata includes status, recordingPath, and cacheKey.

Harness Notes

Troubleshooting

Open this for failing evals, missing reporter data, or uncertain verification.

Failure Matrix

Verification Map

Review Checklist

• New exported functions have brief JSDoc when code is added.
• Docs and examples use the same import paths as package exports.
• Provider-key-dependent evals are gated with skipIf.
• Test assertions cover output, tool traces, and usage when the change affects them.
• Normalized data can be serialized with JSON.stringify(...).

Utilities

Open this when assertions need normalized message, tool-call, or span history.

Session Helpers

Use these helpers with a HarnessRun such as result. They also accept a normalized session when that is the only value available.

const calls = toolCalls(result);

expect(calls.map((call) => call.name)).toEqual([
  "lookupInvoice",
  "createRefund",
]);
expect(latestAssistantMessageContent(result)).toContain("approved");

Trace Helpers

Use these helpers with a HarnessRun such as result when behavior is represented by spans. They also accept a normalized trace array when that is the only value available.

expect(spansByKind(result, "tool").map((span) => span.name)).toContain(
  "lookupInvoice",
);
expect(failedSpans(result)).toHaveLength(0);

vitest-evals Skill Sources

Synthesis Summary

Source Inventory

Decisions

Coverage Matrix

Trigger Optimization

Should trigger:

• "write a vitest-evals suite for my refund agent"
• "wire an ai-sdk generateText app into vitest-evals"
• "make a pi-ai harness eval"
• "add a custom judge using JudgeContext"
• "why are tool calls missing from my vitest-evals reporter output?"
• "configure vitest-evals replay"
• "create docs for harness-backed vitest-evals"

Should not trigger:

• "write a generic Vitest unit test"
• "optimize this model prompt"
• "create a spreadsheet of eval results"
• "build a React dashboard"
• "debug Anthropic authentication outside the eval harness"
• "write a generic AI SDK tutorial"

Final description: Use when authoring, reviewing, or debugging harness-backed vitest-evals suites, custom Harness adapters, first-party ai-sdk or pi-ai harness integrations, judges, replay, reporter-facing normalized run data, or examples and docs for these APIs.

Retrieval Stopping Rationale

Source coverage includes package docs, source types, first-party harness implementations, harness tests, demo evals, replay code, and repo policy docs. Additional retrieval is low-yield until API options, package names, or repo installation conventions change.

Open gaps

• Review skills, .agents, plugin manifests, and repo install docs before adding any registration file.
• Confirm future repo docs or skill installer conventions before adding an install script or catalog entry.
• Retrieve package manifests plus packages/*/src/index.ts before changing version-sensitive guidance.

Changelog

• Created initial self-contained vitest-evals skill with harness-specific references and shared authoring guidance.

vitest-evals Specification

Intent

This skill helps agents author, review, and debug harness-backed vitest-evals work without needing to rediscover the package shape from source every time. It is optimized for product API correctness, trace quality, and small targeted verification.

Scope

In scope:

• harness-backed eval suites using describeEval(...)
• custom Harness adapters
• first-party @vitest-evals/harness-ai-sdk and @vitest-evals/harness-pi-ai
• judges, explicit judge matchers, reporter-facing normalized data, and replay
• examples, docs, tests, and package work that changes those APIs

Out of scope:

• unrelated Vitest test authoring
• provider SDK setup that is not part of a harness
• broad application design outside the runtime seam being evaluated
• alternate suite APIs outside the harness-backed model

Users And Trigger Context

• Primary users: coding agents working in this repo or in apps adopting the package.
• Common user requests: create an eval, wire an AI SDK app, wire a Pi agent, add a judge, add replay, debug missing tool calls, document harness usage.
• Should not trigger for: generic unit tests, generic prompt writing, spreadsheet or slide work, or frontend-only tasks.

Runtime Contract

• Required first actions: inspect the touched code, choose the runtime target, open the matching reference.
• Required outputs: implementation or review guidance that uses the harness-backed API only, plus targeted verification.
• Non-negotiable constraints: normalized run data remains JSON-serializable,

suite tests call run(...) explicitly, and judge model calls stay on judges or judge-side helpers rather than the app harness.

• Expected bundled files loaded at runtime: SKILL.md first, then one or more focused files under references/.

Source And Evidence Model

Authoritative sources:

• package source and tests under packages/vitest-evals
• first-party harness source and tests under packages/harness-ai-sdk and packages/harness-pi-ai
• demo evals under apps/demo-ai-sdk and apps/demo-pi
• repo docs that describe the harness-backed product shape

Useful improvement sources:

• positive examples: demo evals, passing harness tests, reviewer-approved docs
• negative examples: review feedback, test failures, reporter regressions
• commit logs/changelogs: use when API behavior has changed recently
• issue or PR feedback: use when it clarifies user-facing ergonomics
• eval results: use when examples or reporter output drift

Data that must not be stored:

• secrets
• customer data
• private URLs or identifiers not needed for reproduction
• raw provider transcripts unless redacted and required for a reproducible example

Reference Architecture

• SKILL.md contains activation rules, routing, defaults, and verification commands.
• references/ contains focused runtime lookup files by authoring need.
• SOURCES.md contains provenance, decisions, coverage, gaps, and changelog.
• SPEC.md contains this maintenance contract.
• scripts/ and assets/ are intentionally absent until repeatable validation or templates are needed.

Evaluation

• Lightweight validation: run the skill validator, inspect routing coverage, and search new files for excluded API guidance.
• Deeper evaluation: run holdout prompts when trigger behavior or reference coverage changes materially.
• Holdout examples: keep examples in SOURCES.md unless they become large enough for references/evidence/.
• Acceptance gates: all supported harnesses have a direct reference, every reference is routed from SKILL.md, and no runtime guidance points outside the harness-backed model.

Known Limitations

• The skill does not automate installation; it is installable as a self-contained directory.
• Provider-specific model credentials and environment setup remain application concerns.
• API details must be refreshed when package options, peer ranges, or reporter behavior change.

Maintenance Notes

• Update SKILL.md when trigger language, routing, or universal defaults change.
• Update SOURCES.md when source evidence, decisions, coverage, or gaps change.
• Update harness reference files when a harness package option, runtime contract, normalization rule, or replay behavior changes.
• Add references/evidence/ only when repeated positive or negative examples become useful for future revisions.