Dspy Evaluate

Evaluate Your DSPy Program

Guide the user through measuring AI quality with DSPy's Evaluate class. The pattern: pick a metric, prepare a devset, run the evaluator, interpret results, then feed the same metric into an optimizer.

What is dspy.Evaluate

dspy.Evaluate runs your program on every devset example, scores each with a metric, and reports the aggregate score. It handles threading and progress display. Returns a percentage (0-100).

Built-in metrics

DSPy provides answer_exact_match (normalized string equality) and answer_passage_match (substring check). Both expect an answer field on example and prediction.

SemanticF1

Measures token-level overlap between the predicted and expected answer using an F1 score. More forgiving than exact match — it gives partial credit for answers that are close but not identical:

from dspy.evaluate import SemanticF1

semantic_f1 = SemanticF1()

evaluator = Evaluate(devset=devset, metric=semantic_f1, num_threads=4)
score = evaluator(my_program)

SemanticF1 is a good default metric for open-ended QA tasks where exact match is too strict. It expects a question field on the example plus a response field on both the example and prediction (not answer). Constructor: SemanticF1(threshold=0.66, decompositional=False). The threshold controls the minimum score during optimization (when trace is set).

CompleteAndGrounded

Checks whether the predicted answer is both complete (covers all key claims in the gold answer) and grounded (doesn't hallucinate facts not in the gold answer):

from dspy.evaluate import CompleteAndGrounded

complete_and_grounded = CompleteAndGrounded()

evaluator = Evaluate(devset=devset, metric=complete_and_grounded, num_threads=4)
score = evaluator(my_program)

This is an LM-based metric — it uses the configured LM to judge completeness and groundedness. It expects response and context fields on the prediction, and question and response on the example. Constructor: CompleteAndGrounded(threshold=0.66). Useful for RAG tasks where you care about both recall and precision of facts.

Custom metrics

A metric is def metric(example, prediction, trace=None) returning bool, int, or float. The trace parameter is None during evaluation but set during optimization (use this to apply stricter requirements during training).

Multi-field scoring

def metric(example, prediction, trace=None):
    fields = ["name", "email", "phone"]
    correct = sum(
        1 for f in fields
        if getattr(prediction, f, "").strip().lower() == getattr(example, f, "").strip().lower()
    )
    return correct / len(fields)

LM-as-judge

For open-ended tasks (summaries, creative writing, complex QA), use an LM to judge quality. Define a signature for the judge, then call it inside your metric:

class AssessAnswer(dspy.Signature):
    """Assess if the predicted answer correctly addresses the question."""
    question: str = dspy.InputField()
    gold_answer: str = dspy.InputField(desc="The reference answer")
    predicted_answer: str = dspy.InputField(desc="The answer to evaluate")
    is_correct: bool = dspy.OutputField(desc="True if the prediction is correct and complete")

def llm_judge_metric(example, prediction, trace=None):
    judge = dspy.Predict(AssessAnswer)
    result = judge(
        question=example.question,
        gold_answer=example.answer,
        predicted_answer=prediction.answer,
    )
    return result.is_correct

Use a separate LM for the judge

To avoid the model grading its own work, use a different (often stronger) LM for the judge:

judge_lm = dspy.LM("openai/gpt-4o")  # or "anthropic/claude-sonnet-4-5-20250929", etc.

def llm_judge_metric(example, prediction, trace=None):
    judge = dspy.Predict(AssessAnswer)
    with dspy.context(lm=judge_lm):
        result = judge(
            question=example.question,
            gold_answer=example.answer,
            predicted_answer=prediction.answer,
        )
    return result.is_correct

Graded judge (float scores)

Return a float instead of a bool for partial credit:

class GradeAnswer(dspy.Signature):
    """Grade the predicted answer on a scale of 0 to 5."""
    question: str = dspy.InputField()
    gold_answer: str = dspy.InputField()
    predicted_answer: str = dspy.InputField()
    score: int = dspy.OutputField(desc="Score from 0 (completely wrong) to 5 (perfect)")
    justification: str = dspy.OutputField(desc="Why this score was given")

def graded_metric(example, prediction, trace=None):
    judge = dspy.ChainOfThought(GradeAnswer)
    result = judge(
        question=example.question,
        gold_answer=example.answer,
        predicted_answer=prediction.answer,
    )
    return result.score / 5.0  # normalize to 0.0-1.0

Composite metrics

Combine multiple signals into a single score with weights:

def composite_metric(example, prediction, trace=None):
    # Correctness (primary signal)
    correct = float(prediction.answer.strip().lower() == example.answer.strip().lower())

    # Conciseness (prefer shorter answers)
    concise = float(len(prediction.answer.split()) < 50)

    # Has reasoning (check that the model explained its thinking)
    has_reasoning = float(len(getattr(prediction, "reasoning", "")) > 20)

    # Weighted combination
    return 0.7 * correct + 0.2 * concise + 0.1 * has_reasoning

Mixing exact checks with LM judges

def hybrid_metric(example, prediction, trace=None):
    # Fast exact check
    if prediction.answer.strip().lower() == example.answer.strip().lower():
        return 1.0

    # Fall back to LM judge for partial credit
    judge = dspy.Predict(AssessAnswer)
    result = judge(
        question=example.question,
        gold_answer=example.answer,
        predicted_answer=prediction.answer,
    )
    return 0.5 if result.is_correct else 0.0

Debugging with per-example scores

Evaluate returns an EvaluationResult with .score (aggregate percentage) and .results (list of (example, prediction, score) tuples). Use .results to find failing examples and understand failure patterns.

Common patterns

Trace-aware metrics for optimization

The trace parameter is None during evaluation but set during optimization. Use this to apply stricter requirements during training:

def metric(example, prediction, trace=None):
    correct = prediction.answer.strip().lower() == example.answer.strip().lower()
    if trace is not None:
        # During optimization: also require good reasoning
        has_reasoning = len(getattr(prediction, "reasoning", "")) > 50
        return correct and has_reasoning
    # During evaluation: only check correctness
    return correct

This makes the optimizer filter for traces where the model both got the answer right and showed its work. The result is more robust few-shot demonstrations.

Before-and-after comparison

A common workflow for measuring the impact of optimization:

from dspy.evaluate import Evaluate

evaluator = Evaluate(devset=devset, metric=metric, num_threads=4, display_table=5)

# Baseline
baseline = evaluator(my_program)

# Optimize
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized = optimizer.compile(my_program, trainset=trainset)

# Compare
optimized_result = evaluator(optimized)
print(f"Baseline:  {baseline.score:.1f}%")
print(f"Optimized: {optimized_result.score:.1f}%")
print(f"Delta:     {optimized_result.score - baseline.score:+.1f}%")

Gotchas

1. Claude uses `return_all_scores=True` or `return_outputs=True` which no longer exist. Evaluate now returns an EvaluationResult object. Access .score for the aggregate percentage and .results for per-example (example, prediction, score) tuples. Do not pass return_all_scores or return_outputs. 2. Claude uses `SemanticF1` with `answer` fields but it expects `response`. SemanticF1 looks for response on both the example and prediction, not answer. If your signature uses answer, either rename the field or write a wrapper metric. 3. Claude uses `CompleteAndGrounded` without providing `context`. CompleteAndGrounded expects response and context on the prediction, and question and response on the example. Without context, it cannot check groundedness. 4. Metrics must return a `float` or `bool`, not a string -- returning a string silently breaks scoring. 5. Small dev sets (<30 examples) give unreliable scores -- results can swing 10-20% between runs. Aim for 50+ examples for stable evaluation.

Additional resources

• dspy.Evaluate API docs
• dspy.SemanticF1 API docs
• dspy.CompleteAndGrounded API docs
• For constructor signatures and method reference, see reference.md
• For worked examples (exact match, LM judge, composite), see examples.md

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Need to prepare training and evaluation data? Use /dspy-data
• Ready to optimize with few-shot examples? Use /dspy-bootstrap-few-shot
• Want the best prompt optimization? Use /dspy-miprov2
• For the full measure-improve-verify loop, see /ai-improving-accuracy
• For decomposed RAG evaluation (faithfulness, context precision/recall) see /dspy-ragas
• For worked examples (exact match, LM judge, composite), see examples.md
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Evaluate Examples

Exact Match Evaluation

A simple QA pipeline evaluated with exact match. Shows the full workflow: setup, devset, metric, evaluation, and inspecting failures.

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# Program
qa = dspy.ChainOfThought("question -> answer")

# Devset
devset = [
    dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="What is the largest planet in our solar system?", answer="Jupiter").with_inputs("question"),
    dspy.Example(question="Who wrote Romeo and Juliet?", answer="Shakespeare").with_inputs("question"),
    dspy.Example(question="What is the chemical symbol for gold?", answer="Au").with_inputs("question"),
    dspy.Example(question="What year did World War II end?", answer="1945").with_inputs("question"),
    dspy.Example(question="What is the speed of light in m/s?", answer="299792458").with_inputs("question"),
    dspy.Example(question="What is the smallest prime number?", answer="2").with_inputs("question"),
    dspy.Example(question="What language is DSPy written in?", answer="Python").with_inputs("question"),
    dspy.Example(question="How many continents are there?", answer="7").with_inputs("question"),
    dspy.Example(question="What is the boiling point of water in Celsius?", answer="100").with_inputs("question"),
]

# Metric: normalized exact match
def exact_match(example, prediction, trace=None):
    pred = prediction.answer.strip().lower()
    gold = example.answer.strip().lower()
    return pred == gold

# Evaluate
evaluator = Evaluate(
    devset=devset,
    metric=exact_match,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

result = evaluator(qa)
print(f"\nOverall accuracy: {result.score:.1f}%")

# Inspect failures using result.results (list of (example, prediction, score) tuples)
print("\nFailing examples:")
for i, (example, pred, s) in enumerate(result.results):
    if not s:
        print(f"  [{i}] Q: {example.question}")
        print(f"       Expected: {example.answer}")
        print(f"       Got:      {pred.answer}")

LM-as-Judge Evaluation

Grading open-ended answers where exact match is too strict. Uses a separate LM to judge whether predictions are correct and complete.

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# Program: explain concepts
explainer = dspy.ChainOfThought("concept -> explanation")

# Devset with reference explanations
devset = [
    dspy.Example(
        concept="photosynthesis",
        explanation="The process by which plants convert sunlight, water, and CO2 into glucose and oxygen.",
    ).with_inputs("concept"),
    dspy.Example(
        concept="recursion in programming",
        explanation="A function that calls itself to solve smaller instances of the same problem, with a base case to stop.",
    ).with_inputs("concept"),
    dspy.Example(
        concept="supply and demand",
        explanation="An economic model where price is determined by the relationship between how much of something is available and how much people want it.",
    ).with_inputs("concept"),
    dspy.Example(
        concept="natural selection",
        explanation="Organisms with traits better suited to their environment are more likely to survive and reproduce, passing those traits on.",
    ).with_inputs("concept"),
    dspy.Example(
        concept="HTTP status codes",
        explanation="Three-digit codes returned by web servers indicating the result of a request: 2xx for success, 4xx for client errors, 5xx for server errors.",
    ).with_inputs("concept"),
]

# LM-as-judge signature
class JudgeExplanation(dspy.Signature):
    """Judge whether the predicted explanation correctly covers the key ideas in the reference explanation. Minor wording differences are fine — focus on factual accuracy and completeness."""
    concept: str = dspy.InputField()
    reference_explanation: str = dspy.InputField(desc="The gold-standard explanation")
    predicted_explanation: str = dspy.InputField(desc="The explanation to evaluate")
    is_correct: bool = dspy.OutputField(desc="True if the prediction covers the key ideas accurately")
    reasoning: str = dspy.OutputField(desc="Brief explanation of the judgment")

# Use a stronger model as the judge
judge_lm = dspy.LM("openai/gpt-4o")  # or "anthropic/claude-sonnet-4-5-20250929", etc.

def llm_judge(example, prediction, trace=None):
    judge = dspy.ChainOfThought(JudgeExplanation)
    with dspy.context(lm=judge_lm):
        result = judge(
            concept=example.concept,
            reference_explanation=example.explanation,
            predicted_explanation=prediction.explanation,
        )
    return result.is_correct

# Evaluate
evaluator = Evaluate(
    devset=devset,
    metric=llm_judge,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

result = evaluator(explainer)
print(f"\nJudge accuracy: {result.score:.1f}%")

Composite Metric Evaluation

Combining correctness, conciseness, and safety into a single weighted score. Demonstrates how to build metrics that balance multiple quality dimensions.

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# Program: answer health questions
health_qa = dspy.ChainOfThought("question -> answer")

# Devset
devset = [
    dspy.Example(
        question="What are common symptoms of the flu?",
        answer="Fever, cough, body aches, fatigue, and sometimes vomiting or diarrhea.",
    ).with_inputs("question"),
    dspy.Example(
        question="How much water should an adult drink daily?",
        answer="About 8 cups (2 liters) per day, though needs vary by activity level and climate.",
    ).with_inputs("question"),
    dspy.Example(
        question="What is the recommended amount of sleep for adults?",
        answer="7-9 hours per night.",
    ).with_inputs("question"),
    dspy.Example(
        question="What are the benefits of regular exercise?",
        answer="Improved cardiovascular health, better mood, weight management, stronger bones, and reduced disease risk.",
    ).with_inputs("question"),
    dspy.Example(
        question="What causes seasonal allergies?",
        answer="Immune system overreaction to pollen, mold spores, or other airborne allergens.",
    ).with_inputs("question"),
]

# Safety check signature
class CheckSafety(dspy.Signature):
    """Check if a health answer is safe — does not give specific medical diagnoses, dosage recommendations, or advice to skip professional medical consultation."""
    question: str = dspy.InputField()
    answer: str = dspy.InputField()
    is_safe: bool = dspy.OutputField(desc="True if the answer is safe and appropriately general")

# Correctness check signature
class CheckCorrectness(dspy.Signature):
    """Check if the predicted answer captures the key facts from the reference answer."""
    question: str = dspy.InputField()
    reference_answer: str = dspy.InputField()
    predicted_answer: str = dspy.InputField()
    is_correct: bool = dspy.OutputField(desc="True if key facts are covered accurately")

judge_lm = dspy.LM("openai/gpt-4o")  # or "anthropic/claude-sonnet-4-5-20250929", etc.

def composite_metric(example, prediction, trace=None):
    # 1. Correctness (0.6 weight) — LM judge
    with dspy.context(lm=judge_lm):
        correctness_judge = dspy.Predict(CheckCorrectness)
        correctness_result = correctness_judge(
            question=example.question,
            reference_answer=example.answer,
            predicted_answer=prediction.answer,
        )
    correct = float(correctness_result.is_correct)

    # 2. Conciseness (0.2 weight) — heuristic
    word_count = len(prediction.answer.split())
    if word_count <= 50:
        concise = 1.0
    elif word_count <= 100:
        concise = 0.5
    else:
        concise = 0.0

    # 3. Safety (0.2 weight) — LM judge
    with dspy.context(lm=judge_lm):
        safety_judge = dspy.Predict(CheckSafety)
        safety_result = safety_judge(
            question=example.question,
            answer=prediction.answer,
        )
    safe = float(safety_result.is_safe)

    # Weighted composite
    score = 0.6 * correct + 0.2 * concise + 0.2 * safe

    # During optimization, require all three to pass
    if trace is not None:
        return correct and concise >= 0.5 and safe

    return score

# Evaluate
evaluator = Evaluate(
    devset=devset,
    metric=composite_metric,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

result = evaluator(health_qa)
print(f"\nComposite score: {result.score:.1f}%")

# Breakdown per example using result.results
for i, (example, pred, score) in enumerate(result.results):
    status = "PASS" if score >= 0.8 else "WARN" if score >= 0.5 else "FAIL"
    print(f"  [{status}] {example.question[:60]}... score={score:.2f}")

Condensed from dspy.ai/api/evaluation/Evaluate/, SemanticF1, and CompleteAndGrounded. Verify against upstream for latest.

dspy.Evaluate — API Reference

Constructor

dspy.Evaluate(
    *,
    devset,                    # list[Example] (required)
    metric=None,               # Callable | None
    num_threads=None,          # int | None
    display_progress=False,    # bool
    display_table=False,       # bool | int (int truncates columns)
    max_errors=None,           # int | None
    provide_traceback=None,    # bool | None
    failure_score=0.0,         # float
    save_as_csv=None,          # str | None
    save_as_json=None,         # str | None
)

__call__()

result = evaluator(
    program,                   # dspy.Module (required)
    metric=None,               # override metric
    devset=None,               # override devset
    num_threads=None,          # override threads
    display_progress=None,     # override progress
    display_table=None,        # override table
)

Returns: EvaluationResult with:

• .score — aggregate percentage (0-100)
• .results — list of (example, prediction, score) tuples

Note: return_all_scores and return_outputs are removed. Per-example results are always in .results.

Metric function signature

def metric(example, prediction, trace=None):
    """
    Args:
        example: dspy.Example with ground truth
        prediction: dspy.Prediction from the program
        trace: None during evaluation, set during optimization

    Returns:
        bool, int, or float
    """

Built-in metrics

SemanticF1

from dspy.evaluate import SemanticF1

semantic_f1 = SemanticF1(threshold=0.66, decompositional=False)

Expected fields: question and response on both example and prediction.

CompleteAndGrounded

from dspy.evaluate import CompleteAndGrounded

complete_and_grounded = CompleteAndGrounded(threshold=0.66)

Expected fields: question and response on example; response and context on prediction.

Other built-ins