Evals Write Spec

Write Eval Specs

Spec File Anatomy

Eval specs use the evaluate Playwright fixture (not test). A spec file follows this structure:

import { evaluate, tags, selectEvaluators, type Example, type TaskOutput } from '@kbn/evals';

evaluate.describe('Suite name', { tag: tags.serverless.observability.complete }, () => {
  evaluate.beforeAll(async ({ fetch, log }) => {
    // one-time setup: install docs, create agents, load archives
  });

  evaluate.afterAll(async ({ fetch, log }) => {
    // teardown: uninstall docs, delete agents, unload archives
  });

  evaluate('test name', async ({ executorClient, connector }) => {
    await executorClient.runExperiment(
      { dataset, task },
      evaluators
    );
  });
});

When a suite has a custom src/evaluate.ts, import from there instead of @kbn/evals:

import { evaluate } from '../src/evaluate';

Tags

Every evaluate.describe must have a tag. Common choices:

Import tags from @kbn/scout or @kbn/evals (re-exported).

Datasets

A dataset is an array of examples with typed input, output (expected), and optional metadata:

type MyExample = Example<
  { question: string },
  { expectedAnswer: string },
  { tags?: string[] }
>;

const dataset = {
  name: 'my-dataset',
  description: 'What this dataset tests',
  examples: [
    {
      input: { question: 'What is 2+2?' },
      output: { expectedAnswer: '4' },
      metadata: { tags: ['math'] },
    },
  ],
};

Keep datasets focused. For local iteration, use --grep to run a subset:

node scripts/evals start --grep "my test name"

Tasks

The task function receives an example and returns the output to evaluate:

task: async ({ input }) => {
  const result = await someKibanaApi(input.question);
  return { answer: result.content };
}

Tasks can use any fixture available in the evaluate callback: fetch, inferenceClient, connector, esClient, kbnClient, or custom fixtures like chatClient.

Evaluators

There are two ways to provide evaluators to runExperiment:

1. Inline array -- pass evaluator objects directly (simple suites) 2. `selectEvaluators` -- typed wrapper that enforces Example/TaskOutput generics

CODE Evaluators

Deterministic, no LLM call. Use for binary checks:

{
  name: 'NonEmpty',
  kind: 'CODE',
  evaluate: async ({ output }) => ({
    score: output?.documents?.length > 0 ? 1 : 0,
  }),
}

LLM-as-Judge Criteria

Use evaluators.criteria(criteriaArray) for subjective quality checks. The judge LLM scores each criterion:

evaluators.criteria([
  'The response correctly identifies the top users.',
  'The response includes risk scores.',
]).evaluate({ input, output, expected, metadata })

Correctness Analysis

Compares output against expected answer:

evaluators.correctnessAnalysis().evaluate({ input, output, expected, metadata })

Groundedness Analysis

Checks if output is grounded in provided context:

evaluators.groundednessAnalysis().evaluate({ input, output, expected, metadata })

Trace-Based Evaluators

Available from evaluators.traceBasedEvaluators:

• inputTokens, outputTokens, cachedTokens -- token usage
• toolCalls -- number of tool calls
• latency -- span latency in seconds

These read from the tracing ES cluster and require EDOT to be running.

RAG Evaluators

For retrieval-augmented generation with ground truth:

import { createPrecisionAtKEvaluator, createRecallAtKEvaluator, createF1AtKEvaluator } from '@kbn/evals';

See evaluator-patterns.md for full examples.

Available Fixtures

The evaluateDataset Pattern

For suites with many specs that share the same task + evaluator wiring, extract a reusable helper:

`src/evaluate_dataset.ts`:

import type { DefaultEvaluators, EvalsExecutorClient } from '@kbn/evals';
import type { MyChatClient } from './chat_client';

export type EvaluateDataset = (opts: {
  dataset: { name: string; description: string; examples: MyExample[] };
}) => Promise<void>;

export function createEvaluateDataset({
  chatClient, evaluators, executorClient,
}: {
  chatClient: MyChatClient;
  evaluators: DefaultEvaluators;
  executorClient: EvalsExecutorClient;
}): EvaluateDataset {
  return async ({ dataset }) => {
    await executorClient.runExperiment(
      {
        dataset,
        task: async ({ input }) => {
          const response = await chatClient.converse({ messages: [{ message: input.question }] });
          return { messages: response.messages, steps: response.steps };
        },
      },
      [myCriteriaEvaluator, myToolCallsEvaluator]
    );
  };
}

In the spec:

import { evaluate as base } from '../src/evaluate';
import type { EvaluateDataset } from '../src/evaluate_dataset';
import { createEvaluateDataset } from '../src/evaluate_dataset';

const evaluate = base.extend<{ evaluateDataset: EvaluateDataset }, {}>({
  evaluateDataset: [
    ({ chatClient, evaluators, executorClient }, use) => {
      use(createEvaluateDataset({ chatClient, evaluators, executorClient }));
    },
    { scope: 'test' },
  ],
});

evaluate.describe('My suite', { tag: tags.serverless.search }, () => {
  evaluate('my test', async ({ evaluateDataset }) => {
    await evaluateDataset({ dataset: { name: '...', description: '...', examples: [...] } });
  });
});

Setup and Teardown

Use evaluate.beforeAll / evaluate.afterAll for expensive one-time operations:

• Install product docs: POST to /internal/product_doc_base/install
• Create agents/rules: Use fetch or kbnClient
• Load ES archives: Use esArchiver.load(archivePath) (requires custom fixture)

Always clean up in afterAll -- delete agents, uninstall docs, unload archives.

Running Locally

# Full interactive flow
node scripts/evals start

# Specify model and judge
node scripts/evals start --model <connector-id> --judge <connector-id>

# Filter to a specific test
node scripts/evals start --grep "my test name"

# Run directly (services already running)
node scripts/evals run --model <connector-id> --judge <connector-id>

Common Mistakes

• Forgetting the tag on evaluate.describe -- Scout validates tags at runtime.
• Missing afterAll cleanup -- leftover agents/docs pollute subsequent runs.
• Overly large datasets for local iteration -- use --grep to target a single evaluate() block.
• Importing evaluate from @kbn/evals when the suite has a custom src/evaluate.ts -- you'll miss custom fixtures.
• Using test instead of evaluate -- the evaluate fixture provides all the evals-specific wiring.

References

• Evaluator type examples with real code: references/evaluator-patterns.md
• Suite scaffolding: use the evals-create-suite skill

Evaluator Patterns

Extended examples for each evaluator type, extracted from real eval suites.

CODE Evaluators with selectEvaluators

From llm-tasks -- deterministic checks with typed generics:

import { selectEvaluators, type Example, type TaskOutput } from '@kbn/evals';

type MyExample = Example & {
  input: { searchTerm: string; products?: string[] };
  metadata?: { minDocs?: number; requiredTerms?: string[] };
};

type MyTaskOutput = TaskOutput & {
  success: boolean;
  documents: Array<{ title: string; url: string; content: string }>;
};

await executorClient.runExperiment(
  { dataset, task },
  selectEvaluators<MyExample, MyTaskOutput>([
    {
      name: 'NonEmptyDocuments',
      kind: 'CODE',
      evaluate: async ({ output, metadata }) => {
        const minDocs = typeof metadata?.minDocs === 'number' ? metadata.minDocs : 1;
        const count = output?.documents?.length ?? 0;
        return { score: count >= minDocs ? 1 : 0, metadata: { minDocs, count } };
      },
    },
    {
      name: 'RequiredTermsInContent',
      kind: 'CODE',
      evaluate: async ({ output, metadata }) => {
        const requiredTerms = metadata?.requiredTerms ?? [];
        if (requiredTerms.length === 0) return { score: 1 };

        const text = (output?.documents ?? [])
          .slice(0, 3)
          .map((d) => `${d.title}\n${d.content}`)
          .join('\n');

        const ok = requiredTerms.every((term) =>
          text.toLowerCase().includes(term.toLowerCase())
        );
        return { score: ok ? 1 : 0, metadata: { requiredTerms } };
      },
    },
    {
      name: 'HasValidUrl',
      kind: 'CODE',
      evaluate: async ({ output }) => {
        const urls = (output?.documents ?? []).map((d) => d.url);
        const ok = urls.some((u) => typeof u === 'string' && u.startsWith('https://'));
        return { score: ok ? 1 : 0, metadata: { urls: urls.slice(0, 3) } };
      },
    },
  ])
);

Key points:

• selectEvaluators<ExampleType, OutputType>([...]) gives type safety on input, output, expected, metadata.
• kind: 'CODE' means no LLM call -- fast and deterministic.
• Return metadata in the result to aid debugging.

LLM-as-Judge Criteria via evaluators.criteria

From security-solution-evals -- the criteria evaluator delegates to the judge LLM:

const mainCriteriaResult = await evaluators
  .criteria([
    'The response correctly identifies the top users.',
    'The response includes risk scores for each user.',
    'The response includes risk levels for each user.',
  ])
  .evaluate({ input, output, expected, metadata });

Each criterion is evaluated independently. The judge returns a score (0 or 1) and explanation per criterion.

Wrapping Criteria in a Reusable Evaluator

function createCriteriaEvaluator({ evaluators }: { evaluators: DefaultEvaluators }) {
  return {
    name: 'Criteria',
    kind: 'LLM' as const,
    evaluate: async ({ input, output, expected, metadata }: {
      input: MyExample['input'];
      output: MyTaskOutput;
      expected: MyExample['output'];
      metadata: MyExample['metadata'];
    }) => {
      const criteria = expected.criteria ?? [];
      if (criteria.length === 0) {
        return { score: 1, label: 'PASS', explanation: 'No criteria specified.' };
      }
      return evaluators.criteria(criteria).evaluate({ input, expected, output, metadata });
    },
  };
}

Tool Call Evaluator

From security-solution-evals -- checks that specific tools were invoked:

function createToolCallsEvaluator({ evaluators }: { evaluators: DefaultEvaluators }) {
  return {
    name: 'ToolCalls',
    kind: 'LLM' as const,
    evaluate: async ({ input, output, expected, metadata }) => {
      const toolCalls = expected.toolCalls ?? [];
      const steps = output.steps ?? [];

      if (toolCalls.length === 0) {
        return { score: 1, label: 'PASS', explanation: 'No tool call assertions.' };
      }

      const results = [];
      for (const assertion of toolCalls) {
        const called = steps.some(
          (s) => s.type === 'tool_call' && s.tool_id === assertion.id
        );

        if (!called) {
          results.push({
            score: 0,
            label: 'FAIL',
            explanation: `Tool "${assertion.id}" was not called.`,
          });
          continue;
        }

        if (assertion.criteria?.length) {
          const criteriaResult = await evaluators
            .criteria(assertion.criteria)
            .evaluate({ input, expected: { criteria: assertion.criteria }, output, metadata });
          results.push(criteriaResult);
        } else {
          results.push({ score: 1, label: 'PASS', explanation: `Tool "${assertion.id}" called.` });
        }
      }

      const allPassed = results.every((r) => r.label === 'PASS');
      const scores = results.map((r) => r.score ?? 0);
      const avg = scores.reduce((a, b) => a + b, 0) / scores.length;

      return {
        score: allPassed ? avg : 0,
        label: allPassed ? 'PASS' : 'FAIL',
        explanation: results.map((r) => r.explanation).join(' '),
      };
    },
  };
}

Dataset examples with tool call assertions:

{
  input: { question: 'Which users have the highest risk scores?' },
  output: {
    criteria: [
      'Return 10 users with the highest risk scores.',
      'Return the risk levels of those users.',
    ],
    toolCalls: [
      {
        id: 'security.entity_analytics.risk_score',
        criteria: ['The ES|QL query should sort by risk score descending.'],
      },
    ],
  },
}

RAG Evaluators

For retrieval quality with ground truth documents:

import {
  createPrecisionAtKEvaluator,
  createRecallAtKEvaluator,
  createF1AtKEvaluator,
  createRagEvaluators,
} from '@kbn/evals';
import type { GroundTruth, RetrievedDoc } from '@kbn/evals';

The createRagEvaluators factory creates all three at once:

const ragEvals = createRagEvaluators({
  k: 5,
  extractRetrievedDocs: (output) =>
    output.documents.map((d) => ({ id: d.id, content: d.content })),
  extractGroundTruth: (expected) => expected.groundTruth,
});

Ground truth in datasets uses document IDs mapped to relevance scores:

{
  input: { question: 'How do I set up payments?' },
  output: {
    expected: 'You can start accepting payments using Wix Payments...',
    groundTruth: {
      knowledge_base: {
        'doc_hash_abc123': 1,
        'doc_hash_def456': 1,
      },
    },
  },
}

Trace-Based Evaluators

These are pre-built and available from evaluators.traceBasedEvaluators. They query the tracing ES cluster for spans matching the current trace ID:

const { inputTokens, outputTokens, cachedTokens, toolCalls, latency } =
  evaluators.traceBasedEvaluators;

Each returns a numeric score:

• inputTokens / outputTokens / cachedTokens -- token counts from gen_ai.usage.* span attributes
• toolCalls -- count of tool call spans
• latency -- total span duration in seconds

To use trace-based evaluators, EDOT must be running and TRACING_ES_URL must point to an ES instance receiving traces.

Combining Multiple Evaluator Types

A common pattern passes both CODE and LLM evaluators to runExperiment:

await executorClient.runExperiment(
  { dataset, task },
  [
    createCriteriaEvaluator({ evaluators }),
    createToolCallsEvaluator({ evaluators }),
    {
      name: 'HasResponse',
      kind: 'CODE',
      evaluate: async ({ output }) => ({
        score: output?.messages?.length > 0 ? 1 : 0,
      }),
    },
  ]
);

The executor runs all evaluators for each example and aggregates scores in the final report.