Agentic Eval First Development

Agentic Eval-First Development

Evals are infrastructure, not afterthoughts. Define success criteria before writing prompts or task logic. The eval becomes the spec.

Framework: Data → Task → Scores

Every eval has exactly three components:

1. Data — Golden dataset of inputs (the test cases) 2. Task — The operation being evaluated (LLM call, agent workflow, MCP pipeline) 3. Scores — Categorical rubric that maps outputs to normalized 0–1 values

Step 1: Define the PRD (Data & Scores)

Build the Golden Dataset

Collect or generate 10–20 representative inputs covering the full range of expected usage.

• Use a high-reasoning model to autogenerate diverse test cases if manual examples are unavailable
• Intentionally include inputs expected to fail — these map current model limitations
• Store as JSON or JSONL for reproducibility. See references/golden-dataset-template.md for the format

Define the Scoring Rubric

Use categorical scoring (Options A/B/C) rather than asking for raw numbers. Raw numeric scores drift across evaluators and models.

• Every score must include a written rationale explaining the grade
• All scores normalize to 0–1 for cross-model comparison. See references/scoring-rubrics.md for rubric templates
• Run scripts/normalize_scores.py to convert categorical results to normalized values

Example categorical scorer:

A (1.0) — Fully correct, well-structured, addresses all aspects
B (0.5) — Partially correct or missing key elements
C (0.0) — Incorrect, off-topic, or harmful

Step 2: Configure the Task (The Harness)

The task is the operation under evaluation.

1. Tool Pruning — If using MCP, limit available tools to only what's necessary. Models select incorrect tools when overwhelmed with options 2. System Prompt — Define initial instructions based on success criteria from Step 1 (e.g., "don't ask clarifying questions," "respond in JSON") 3. Isolation — Each eval run must be independent. No shared state between test cases

Step 3: Execute the Flywheel Loop

┌─────────────────────────────────────────┐
│  OFFLINE: Run golden dataset locally    │
│  → Identify gaps → Refine prompt/tools  │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│  ONLINE: Deploy scorers to production   │
│  → Monitor real user logs               │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│  CLOSE THE LOOP: Production failures    │
│  → Add back to golden dataset           │
└─────────────────────────────────────────┘

1. Offline iteration — Run experiments locally against the golden dataset. Iterate on prompts, tools, and model selection until scores stabilize 2. Online validation — Deploy scorers to production monitoring real user logs 3. Close the loop — When online score (e.g., 0.3) < offline score (e.g., 0.75), identify production failures and add them to the golden dataset

When to Stop Iterating

• Offline scores plateau across 3+ consecutive runs
• Online/offline gap is < 0.1
• Remaining failures are edge cases outside the product's scope

Troubleshooting

Key Principle

The eval is the durable asset. Models change, prompts evolve, agent frameworks get replaced — but a well-built eval survives all of it. When switching models, re-run the eval; don't re-do the product thinking.

Golden Dataset Template

Format

Store golden datasets as JSONL (one JSON object per line) for streaming and append-friendly workflows.

Minimal Schema

{"id": "test_001", "input": "What is the capital of France?", "expected": "Paris", "tags": ["factual", "easy"], "source": "manual"}
{"id": "test_002", "input": "Explain quantum entanglement to a 5-year-old", "expected": null, "tags": ["creative", "hard"], "source": "manual"}
{"id": "test_003", "input": "", "expected": null, "tags": ["edge-case", "empty-input"], "source": "manual"}

Field Definitions

Coverage Guidelines

A good golden dataset of 10–20 items should include:

• 40–50% typical/happy-path inputs
• 20–30% edge cases (empty input, very long input, ambiguous phrasing)
• 10–20% adversarial inputs (prompt injection attempts, out-of-scope requests)
• 10–20% known failure cases from production

Adding Production Failures

When closing the flywheel loop, append production failures with source "production":

{"id": "prod_047", "input": "the actual user input that failed", "expected": null, "tags": ["production-failure", "tool-selection"], "source": "production", "metadata": {"discovered": "2026-03-15", "online_score": 0.0, "failure_mode": "selected wrong MCP tool"}}

Autogeneration Prompt

When generating test cases with a high-reasoning model:

Generate {N} diverse test inputs for the following task: {task_description}

Requirements:
- Include easy, medium, and hard difficulty levels
- Include at least 2 edge cases (empty input, ambiguous phrasing, very long input)
- Include at least 1 adversarial input (out-of-scope request or prompt injection attempt)
- Tag each input with: difficulty, topic, and expected failure mode (if any)
- Output as JSONL with fields: id, input, expected (null if open-ended), tags, source="autogenerated"

Scoring Rubric Templates

Binary Rubric (Pass/Fail)

Use for factual correctness, constraint adherence, or safety checks.

{
  "rubric_type": "binary",
  "categories": {
    "A": { "label": "Pass", "score": 1.0, "criteria": "Output meets all specified requirements" },
    "C": { "label": "Fail", "score": 0.0, "criteria": "Output fails one or more requirements" }
  }
}

Three-Level Rubric (Standard)

Use for most quality evaluations. The default choice.

{
  "rubric_type": "categorical_3",
  "categories": {
    "A": { "label": "Excellent", "score": 1.0, "criteria": "Fully correct, well-structured, addresses all aspects of the input" },
    "B": { "label": "Partial", "score": 0.5, "criteria": "Partially correct or missing key elements, but demonstrates understanding" },
    "C": { "label": "Poor", "score": 0.0, "criteria": "Incorrect, off-topic, harmful, or fails to address the input" }
  }
}

Five-Level Rubric (Granular)

Use when fine-grained quality distinctions matter (e.g., content generation, summarization).

{
  "rubric_type": "categorical_5",
  "categories": {
    "A": { "label": "Excellent", "score": 1.0, "criteria": "Exceptional quality, exceeds expectations on all dimensions" },
    "B": { "label": "Good", "score": 0.75, "criteria": "Meets expectations with minor issues" },
    "C": { "label": "Adequate", "score": 0.5, "criteria": "Acceptable but with notable gaps or weaknesses" },
    "D": { "label": "Below Average", "score": 0.25, "criteria": "Significant issues that undermine usefulness" },
    "F": { "label": "Unacceptable", "score": 0.0, "criteria": "Fails to meet minimum quality standards" }
  }
}

Multi-Dimension Rubric

Use when evaluating multiple independent quality axes. Final score = weighted average.

{
  "rubric_type": "multi_dimension",
  "dimensions": [
    {
      "name": "Correctness",
      "weight": 0.4,
      "categories": { "A": 1.0, "B": 0.5, "C": 0.0 }
    },
    {
      "name": "Completeness",
      "weight": 0.3,
      "categories": { "A": 1.0, "B": 0.5, "C": 0.0 }
    },
    {
      "name": "Style",
      "weight": 0.3,
      "categories": { "A": 1.0, "B": 0.5, "C": 0.0 }
    }
  ]
}

Scorer Output Format

Every scorer invocation must return this structure:

{
  "input_id": "test_001",
  "grade": "B",
  "score": 0.5,
  "rationale": "The response correctly identified the main topic but omitted the secondary constraint about format. The reasoning was sound but incomplete.",
  "dimension_scores": {}
}

The rationale field is mandatory. Scores without rationale cannot be debugged or improved.