Ai Improving Accuracy

Measure and Improve Your AI

Guide the user through measuring how well their AI works, then systematically improving it. This is a loop: define "good" -> measure -> improve -> verify.

The Workflow

1. Define what "good" means — write a metric 2. Measure current quality — run an evaluation 3. Improve — choose an optimizer, run it 4. Verify — re-evaluate to confirm improvement 5. Iterate or ship

Step 1: Understand the problem

Ask the user: 1. What does your AI get wrong? (wrong answers, wrong format, inconsistent, too slow?) 2. Do you have labeled examples? (how many? what format?) 3. How do you know when an answer is good? (exact match, partial credit, human judgment?) 4. Have you tried optimization before? (if yes, what and what happened?)

If the user does not have labeled data, point them to /ai-generating-data first.

Step 2: Define what "good" means (write a metric)

A metric takes an expected answer and the AI answer, and returns a score.

Exact match (simplest)

def metric(example, prediction, trace=None):
    return prediction.answer == example.answer

Normalized match (handles capitalization/whitespace)

def metric(example, prediction, trace=None):
    return prediction.answer.strip().lower() == example.answer.strip().lower()

Partial credit (for multi-field outputs)

def metric(example, prediction, trace=None):
    fields = ["name", "email", "phone"]
    correct = sum(
        1 for f in fields
        if getattr(prediction, f, "").lower() == getattr(example, f, "").lower()
    )
    return correct / len(fields)

F1 score (for text overlap)

def metric(example, prediction, trace=None):
    gold_tokens = set(example.answer.lower().split())
    pred_tokens = set(prediction.answer.lower().split())
    if not gold_tokens or not pred_tokens:
        return float(gold_tokens == pred_tokens)
    precision = len(gold_tokens & pred_tokens) / len(pred_tokens)
    recall = len(gold_tokens & pred_tokens) / len(gold_tokens)
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

AI-as-judge (for open-ended tasks)

When exact match is too strict (summaries, creative tasks, open-ended Q&A):

class AssessQuality(dspy.Signature):
    """Assess if the predicted answer is correct and complete."""
    question: str = dspy.InputField()
    gold_answer: str = dspy.InputField()
    predicted_answer: str = dspy.InputField()
    is_correct: bool = dspy.OutputField()

def metric(example, prediction, trace=None):
    judge = dspy.Predict(AssessQuality)
    result = judge(
        question=example.question,
        gold_answer=example.answer,
        predicted_answer=prediction.answer,
    )
    return result.is_correct

Training-aware metric

The trace parameter is not None during optimization. Use it for stricter requirements during training:

def metric(example, prediction, trace=None):
    correct = prediction.answer == example.answer
    if trace is not None:
        # During optimization, also require good reasoning
        has_reasoning = len(prediction.reasoning) > 50
        return correct and has_reasoning
    return correct

Step 3: Measure current quality (run evaluation)

import dspy
from dspy.evaluate import Evaluate

devset = [
    dspy.Example(question="What is DSPy?", answer="A framework for LM programs").with_inputs("question"),
    # 50-200+ examples for reliable evaluation
]

evaluator = Evaluate(
    devset=devset,
    metric=metric,
    num_threads=4,
    display_progress=True,
    display_table=5,   # show 5 example results
)

baseline_score = evaluator(my_program)
print(f"Baseline: {baseline_score}")

Step 4: Improve (choose an optimizer)

Stacking tip: Run BootstrapFewShot first, then MIPROv2 on the result. Bootstrap finds good examples, then MIPRO refines the instructions.

BootstrapFewShot (start here)

Fast, cheap. Finds good examples by running your program and keeping successful traces.

optimizer = dspy.BootstrapFewShot(
    metric=metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,   # default is 16 — lower for small datasets
)
optimized = optimizer.compile(my_program, trainset=trainset)

MIPROv2 (recommended for most cases)

Optimizes both instructions and examples. Best general-purpose optimizer.

optimizer = dspy.MIPROv2(
    metric=metric,
    auto="medium",    # "light" (default), "medium", "heavy"
)
optimized = optimizer.compile(my_program, trainset=trainset)

BootstrapFinetune (maximum quality)

Fine-tunes model weights. Requires 500+ examples and a fine-tunable model:

optimizer = dspy.BootstrapFinetune(metric=metric, num_threads=24)
optimized = optimizer.compile(my_program, trainset=trainset)

For the full fine-tuning workflow, see /ai-fine-tuning.

When optimization plateaus

Optimized prompts are model-specific. If you change models, re-run your optimizer.

Step 5: Verify and ship

optimized_score = evaluator(optimized)
print(f"Baseline: {baseline_score:.1f}%")
print(f"Optimized: {optimized_score:.1f}%")
print(f"Improvement: {optimized_score - baseline_score:.1f}%")

# Save
optimized.save("optimized_program.json")

# Load later
my_program = MyProgram()
my_program.load("optimized_program.json")

Gotchas

• Claude writes metrics that return strings instead of floats or bools. DSPy metrics must return a numeric score (float 0.0-1.0) or a boolean. Returning a string like "correct" silently breaks evaluation — the score will be 0 for every example.
• Claude forgets `.with_inputs()` on evaluation Examples. Every dspy.Example must call .with_inputs("field1", ...) to mark input fields. Without this, the evaluator passes all fields (including the expected output) to the program, inflating scores because the model sees the answer.
• Claude uses the same data for training and evaluation. Always split into trainset and devset. Evaluating on training data gives misleadingly high scores — the optimizer may have memorized those exact examples.
• AI-as-judge metrics are slow and expensive during optimization. Each training example triggers a separate LM call for the judge. For a 200-example trainset with MIPROv2 auto="medium", this can add thousands of extra LM calls. Use exact-match or F1 metrics during optimization, then validate with AI-as-judge on the final result.
• `display_table` reveals metric bugs that the score hides. A 75% score looks reasonable, but display_table=10 might show the metric gives credit for completely wrong answers that happen to match on whitespace. Always inspect individual predictions before trusting aggregate scores.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Cost reduction once quality is good -- see /ai-cutting-costs
• Production monitoring to track quality after deployment -- see /ai-monitoring
• Experiment tracking to log and compare optimization runs -- see /ai-tracking-experiments
• Generating data when you need more training examples -- see /ai-generating-data
• Fixing errors when the AI crashes or throws exceptions -- see /ai-fixing-errors
• Signatures for defining typed input/output contracts -- see /dspy-signatures
• Optimizers for detailed API on MIPROv2 and BootstrapFewShot -- see /dspy-optimizers
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• For optimizer details and metric patterns, see reference.md

Accuracy Improvement Reference

Condensed from dspy.ai/api/optimizers/ and dspy.ai/api/evaluation/. Verify against upstream for latest.

Optimizer Comparison Table

Choosing by data size

Optimizer Details

BootstrapFewShot

How it works: 1. Runs your program on training examples 2. Keeps traces where the metric scored high 3. Uses those traces as few-shot examples in prompts

dspy.BootstrapFewShot(
    metric=metric,
    max_bootstrapped_demos=4,         # generated few-shot examples (default: 4)
    max_labeled_demos=16,             # labeled examples from trainset (default: 16)
    max_rounds=1,                     # bootstrapping rounds (default: 1)
)

BootstrapFewShotWithRandomSearch

Same as BootstrapFewShot but tries multiple random configurations:

dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    num_candidate_programs=8,         # configurations to try
    max_labeled_demos=16,             # default: 16
)

MIPROv2

How it works: 1. Generates candidate instructions using an LM 2. Generates candidate few-shot examples 3. Uses Bayesian optimization to find the best combination

dspy.MIPROv2(
    metric=metric,
    auto="medium",                    # "light" | "medium" | "heavy"
    # Or manual control:
    # num_candidates=10,
    # init_temperature=0.7,
    # num_trials=30,
)

Auto settings:

• "light": ~10 trials, good for quick iteration
• "medium": ~30 trials, balanced quality vs cost
• "heavy": ~100 trials, best quality

GEPA

Generates, evaluates, and proposes alternative instructions using an evolutionary approach.

dspy.GEPA()
# Usage: optimizer.compile(program, trainset=trainset, metric=metric)

BootstrapFinetune

How it works: 1. Bootstraps training data from successful traces 2. Fine-tunes the LM on this data 3. Returns a program using the fine-tuned model

dspy.BootstrapFinetune(
    metric=metric,
    num_threads=24,
)

Requirements:

• Sufficient training data (500+)
• A fine-tunable model (OpenAI GPT models, or open-source via Together/Anyscale)
• Budget for fine-tuning API costs

BetterTogether

Jointly optimizes prompts and fine-tuned weights.

dspy.BetterTogether(metric=metric)
optimized = optimizer.compile(program, trainset=trainset)

Composing optimizers

You can chain optimizers:

# First, optimize instructions with GEPA
opt1 = dspy.GEPA()
step1 = opt1.compile(program, trainset=trainset, metric=metric)

# Then, optimize few-shot examples
opt2 = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
step2 = opt2.compile(step1, trainset=trainset)

Metric Patterns

Classification metrics

# Accuracy
def accuracy(example, pred, trace=None):
    return pred.label == example.label

# Multi-label F1
def multilabel_f1(example, pred, trace=None):
    gold = set(example.labels)
    predicted = set(pred.labels)
    if not gold and not predicted:
        return 1.0
    tp = len(gold & predicted)
    precision = tp / len(predicted) if predicted else 0
    recall = tp / len(gold) if gold else 0
    if precision + recall == 0:
        return 0.0
    return 2 * precision * recall / (precision + recall)

Extraction metrics

# Field-level accuracy
def field_accuracy(example, pred, trace=None):
    fields = ["name", "date", "amount"]
    scores = []
    for f in fields:
        expected = getattr(example, f, None)
        predicted = getattr(pred, f, None)
        if expected is not None:
            scores.append(
                float(str(predicted).strip().lower() == str(expected).strip().lower())
            )
    return sum(scores) / len(scores) if scores else 0.0

# Pydantic model comparison
def model_accuracy(example, pred, trace=None):
    gold = example.extracted
    predicted = pred.extracted
    if type(gold) != type(predicted):
        return 0.0
    fields = gold.model_fields.keys()
    correct = sum(1 for f in fields if getattr(gold, f) == getattr(predicted, f))
    return correct / len(fields)

AI-as-judge variants

# Binary judge
class IsCorrect(dspy.Signature):
    """Is the answer correct?"""
    question: str = dspy.InputField()
    expected: str = dspy.InputField()
    actual: str = dspy.InputField()
    correct: bool = dspy.OutputField()

# Graded judge (0-5 scale)
class GradeAnswer(dspy.Signature):
    """Grade the answer quality on a 0-5 scale."""
    question: str = dspy.InputField()
    expected: str = dspy.InputField()
    actual: str = dspy.InputField()
    grade: int = dspy.OutputField(desc="Score from 0 (wrong) to 5 (perfect)")

def graded_metric(example, pred, trace=None):
    judge = dspy.Predict(GradeAnswer)
    result = judge(question=example.question, expected=example.answer, actual=pred.answer)
    return result.grade / 5.0

# Multi-criteria judge
class AssessMulti(dspy.Signature):
    """Assess the answer on multiple criteria."""
    question: str = dspy.InputField()
    expected: str = dspy.InputField()
    actual: str = dspy.InputField()
    factually_correct: bool = dspy.OutputField()
    well_structured: bool = dspy.OutputField()
    concise: bool = dspy.OutputField()

def multi_criteria_metric(example, pred, trace=None):
    judge = dspy.Predict(AssessMulti)
    result = judge(question=example.question, expected=example.answer, actual=pred.answer)
    return (result.factually_correct * 0.6 + result.well_structured * 0.2 + result.concise * 0.2)

Data Loading Patterns

# From CSV
import csv
with open("data.csv") as f:
    reader = csv.DictReader(f)
    examples = [dspy.Example(**row).with_inputs("question") for row in reader]

# From JSON
import json
with open("data.json") as f:
    data = json.load(f)
examples = [dspy.Example(**item).with_inputs("input_field") for item in data]

# From HuggingFace
from datasets import load_dataset
ds = load_dataset("squad", split="validation[:200]")
examples = [
    dspy.Example(question=x["question"], answer=x["answers"]["text"][0]).with_inputs("question")
    for x in ds
]

# Train/dev split
import random
random.seed(42)
random.shuffle(examples)
split = int(0.8 * len(examples))
trainset, devset = examples[:split], examples[split:]

Evaluate class options

from dspy.evaluate import Evaluate

evaluator = Evaluate(
    devset=devset,           # evaluation dataset
    metric=metric,           # scoring function
    num_threads=4,           # parallel threads
    display_progress=True,   # progress bar
    display_table=5,         # show N example results in table
    max_errors=5,            # stop after N errors
)

Troubleshooting

Optimization doesn't improve score:

• Check your metric — is it measuring the right thing?
• Check your data — are labels correct?
• Try a different optimizer
• Add more training data

Optimization is too expensive:

• Use auto="light" with MIPROv2
• Start with BootstrapFewShot
• Use a cheaper LM for optimization, then transfer to the target LM

Optimized program is worse than baseline:

• Overfitting — reduce max_bootstrapped_demos
• Bad metric — validate metric scores manually
• Use a validation set to check for overfitting

Key Methods on Optimized Programs

After optimization, the returned program supports:

# Save and load
optimized.save("optimized_v1.json")
program = MyProgram()
program.load("optimized_v1.json")

# Batch processing
results = optimized.batch(devset, num_threads=4)

Evaluation Best Practices

1. Minimum devset size: 50 examples for rough estimates, 200+ for reliable comparisons 2. Statistical significance: A 2% improvement on 50 examples might be noise; on 500 it's likely real 3. Error analysis: Look at failures, not just the score — they reveal what to fix 4. Metric validation: Manually score 20 examples and compare to your metric 5. Version tracking: Log scores with timestamps, model versions, and optimizer settings