Dspy Gepa

Instruction Optimization with dspy.GEPA

Guide the user through using dspy.GEPA to automatically discover better instructions for their DSPy programs through reflective evolution.

What is dspy.GEPA

dspy.GEPA is a DSPy optimizer that evolves the instruction text in your program's predictors. Rather than adding few-shot examples (like BootstrapFewShot) or tuning model weights (like BootstrapFinetune), GEPA iteratively proposes, evaluates, and refines the natural-language instructions that guide each LM call.

Benchmark results from the GEPA paper (arxiv 2507.19457) show strong performance:

• 93% on MATH via the DSPy adapter vs 67% with basic DSPy (no optimization)
• 10+ percentage points over MIPROv2 across six benchmark tasks (+12 on AIME-2025)
• 20-100 examples needed, vs 100K-512K rollouts for RL approaches like GRPO
• GEPA also has MCP and RAG adapters beyond DSPy, but the DSPy adapter is the focus of this skill

Key properties:

• Tunes instructions only -- no few-shot demos are injected into prompts, keeping them compact
• Uses textual feedback -- a reflection LM reads execution traces and failure feedback to propose better instructions, not just scalar scores
• Maintains a Pareto frontier -- tracks multiple candidate programs that excel on different subsets, then merges the best traits
• Works with 20-100 examples -- needs less data than MIPROv2 or BootstrapFewShotWithRandomSearch
• Supports per-predictor feedback -- metrics can return targeted feedback for individual predictors in multi-step pipelines

When to use GEPA

Use dspy.GEPA when:

• You have 20-100 labeled examples (fewer than what MIPROv2 needs to shine)
• You want to optimize instructions without adding few-shot examples to the prompt
• Your task has interpretable failure modes you can describe in natural language
• You have a multi-step pipeline and want per-predictor instruction tuning
• You want compact prompts (no demo bloat) while still improving quality

Do not use GEPA when:

• You have no way to provide textual feedback on failures -- use dspy.BootstrapFewShot instead
• You need the best possible prompt optimization and have 200+ examples -- use dspy.MIPROv2
• You want to tune model weights -- use dspy.BootstrapFinetune
• Your task is trivially solved without instruction tuning -- use dspy.Predict or dspy.ChainOfThought directly

Basic usage

Three things are needed: a DSPy program, a feedback metric, and a training set.

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))  # or "anthropic/claude-sonnet-4-5-20250929", etc.

# 1. Define your program
classify = dspy.ChainOfThought("text -> label")

# 2. Define a feedback metric
# GEPA metrics can return a float OR a dict with score + feedback text
def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    score = float(pred.label == gold.label)
    feedback = "" if score == 1.0 else f"Expected '{gold.label}', got '{pred.label}'."
    return {"score": score, "feedback": feedback}

# 3. Prepare training data
trainset = [
    dspy.Example(text="Great product!", label="positive").with_inputs("text"),
    dspy.Example(text="Terrible service.", label="negative").with_inputs("text"),
    # ... 20-100 examples
]

# 4. Optimize
gepa = dspy.GEPA(
    metric=metric,
    reflection_lm=dspy.LM("openai/gpt-4o", temperature=1.0, max_tokens=4096),  # use a strong model for reflection
    auto="light",
)
optimized = gepa.compile(classify, trainset=trainset)

# 5. Use the optimized program
result = optimized(text="This exceeded my expectations!")
print(result.label)

# 6. Save for later
optimized.save("optimized_classifier.json")

Constructor parameters

dspy.GEPA(
    metric,                              # GEPAFeedbackMetric (required)
    *,
    auto=None,                           # "light", "medium", or "heavy"
    max_full_evals=None,                 # int -- full validation passes allowed
    max_metric_calls=None,               # int -- total metric invocations allowed
    reflection_lm=None,                  # LM for proposing new instructions
    reflection_minibatch_size=3,         # examples per reflection step
    candidate_selection_strategy="pareto",  # "pareto" or "current_best"
    skip_perfect_score=True,             # skip examples already scoring perfectly
    add_format_failure_as_feedback=False, # include format errors in feedback
    instruction_proposer=None,           # custom proposal function
    component_selector="round_robin",    # which predictor to improve next
    use_merge=True,                      # merge successful variants
    max_merge_invocations=5,             # merge attempt limit
    num_threads=None,                    # parallel evaluation threads
    failure_score=0.0,                   # score for failed examples
    perfect_score=1.0,                   # score that counts as perfect
    log_dir=None,                        # directory for optimization logs
    track_stats=False,                   # return detailed metadata
    track_best_outputs=False,            # retain best outputs per task
    seed=0,                              # reproducibility seed
)

Key parameters explained

Budget control

Exactly one of these three must be set:

• `auto` -- preset budget ("light", "medium", "heavy")
• `max_full_evals` -- number of full passes over the validation set
• `max_metric_calls` -- total number of metric invocations

Start with auto="light" for quick experiments, then move to "medium" or "heavy" for production.

Writing feedback metrics

GEPA metrics are more expressive than standard DSPy metrics. They accept additional keyword arguments for trace-level feedback:

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """
    Args:
        gold: The expected Example (ground truth)
        pred: The model's Prediction
        trace: Full program execution trace (optional)
        pred_name: Name of the predictor being optimized (optional)
        pred_trace: Sub-trace for just this predictor (optional)

    Returns:
        float -- simple score
        OR dict -- {"score": float, "feedback": str}
    """

Returning textual feedback

The key advantage of GEPA over other optimizers is that metrics can explain why an output failed. The reflection LM reads this feedback to propose better instructions.

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    if pred.answer == gold.answer:
        return {"score": 1.0, "feedback": ""}

    feedback_parts = []
    if len(pred.answer) > 200:
        feedback_parts.append("Answer is too verbose. Keep it under 200 characters.")
    if gold.answer.lower() not in pred.answer.lower():
        feedback_parts.append(f"Answer should contain '{gold.answer}'.")

    return {
        "score": 0.0,
        "feedback": " ".join(feedback_parts),
    }

Write feedback that is actionable -- describe what the instruction should encourage or discourage. Vague feedback like "wrong answer" does not help the reflection LM.

Name the failing axis with specifics

Good feedback tells the reflection LM _which_ quality dimension failed and _what_ the correct behavior looks like. Compare:

• Bad: "Wrong answer" -- the reflection LM has no direction
• Bad: "Score: 0" -- no feedback at all
• Good: "Faithfulness: summary claims 'revenue doubled' but the article says 'revenue grew 15%'. The instruction should emphasize only stating facts from the source text."
• Good: "Format: output used bullet points instead of prose. The instruction should specify narrative paragraph format."

When your examples contain metadata beyond the core input (e.g., expected categories, known edge cases, trap fields), use that metadata in the metric to give structural feedback. For example, if an example has example.edge_case = "sarcasm", the metric can say "This review uses sarcasm -- the instruction should warn about positive words with negative intent." This gives the reflection LM a pattern to fix, not just a score to chase.

Feedback engineering principles

The GEPA paper (arxiv 2507.19457) shows that task-specific feedback yields 20-30% better evolved prompts than generic feedback. Maximize your feedback quality:

• Structure feedback with three parts: what aspect failed, why it failed, and what correct behavior looks like. Example: "Faithfulness failed: the summary invented a statistic. The instruction should require citing source sentences."
• Explain causal patterns, not just symptoms. "Wrong answer" gives the reflection LM nothing to work with. "The model confused sarcasm for praise because the review used positive adjectives with negative intent" gives it a pattern to fix.
• Use example metadata in feedback. If your examples have edge case labels, difficulty tags, or domain annotations, reference them: "This is a sarcasm edge case -- the instruction should warn about positive words with negative intent."
• Be specific to the task domain. Generic feedback like "be more accurate" is nearly useless. Domain-specific feedback like "dates must be in ISO 8601 format, not US date format" gives the reflection LM a concrete fix.

Model-size configuration tips

The GEPA paper provides guidance on scaling parameters to model size and budget:

Smaller models benefit from larger populations (more diversity to explore) and more generations (more refinement steps). Larger models converge faster and need fewer candidates.

How GEPA works internally

Understanding the algorithm helps you write better metrics and choose parameters:

1. Initialize -- seeds the candidate pool with the unoptimized program 2. Select candidate -- picks a program from the Pareto frontier (diverse strengths) 3. Sample minibatch -- draws reflection_minibatch_size examples from trainset 4. Collect traces and feedback -- runs the candidate, captures execution traces and metric feedback 5. Select component -- picks which predictor to improve (round-robin by default) 6. Reflect and mutate -- the reflection_lm reads traces + feedback and proposes a revised instruction 7. Evaluate -- scores the new candidate on the minibatch; if promising, validates on the full set 8. Update frontier -- adds the candidate to the Pareto frontier if it is non-dominated 9. Merge -- combines the best predictors from different candidate lineages into one program 10. Terminate -- returns the best aggregate performer when the budget is exhausted

The Pareto frontier is the key innovation: rather than keeping only the single best candidate, GEPA maintains candidates that excel on different subsets. This prevents the optimizer from overfitting to one failure pattern while ignoring others.

GEPA vs MIPROv2 -- when to use which

Paper context: The GEPA paper (arxiv 2507.19457) reports GEPA outperforming MIPROv2 by 10+ percentage points across six benchmark tasks. However, MIPROv2 also tunes few-shot demonstrations, which GEPA does not -- for tasks where in-context examples are critical, MIPROv2 may still be the better choice.

Rule of thumb: Start with GEPA when you have fewer than 200 examples, want compact prompts, or can provide rich textual feedback in your metric. Move to MIPROv2 if you need few-shot demos in the prompt or have 200+ examples.

Providing a validation set

If you have a separate validation set, pass it to compile:

optimized = gepa.compile(
    classify,
    trainset=trainset,
    valset=valset,
)

Without a valset, GEPA uses the trainset for both training and validation. This can lead to overfitting but is useful for test-time search (optimizing for a specific batch of inputs).

Inference-time search

GEPA can be used at inference time to find the best instructions for a specific batch of tasks:

gepa = dspy.GEPA(
    metric=metric,
    reflection_lm=dspy.LM("openai/gpt-4o", temperature=1.0, max_tokens=4096),  # use a strong model for reflection
    auto="light",
    track_stats=True,
    track_best_outputs=True,
)

# Pass the same data as both trainset and valset
result = gepa.compile(program, trainset=tasks, valset=tasks)

# Access the best output for each task
best_per_task = result.detailed_results.best_outputs_valset

What GEPA does NOT optimize

GEPA only tunes the instruction string (the Signature docstring). Everything else in your prompt is fixed during optimization:

This matters most for structured output tasks where Pydantic field descriptions carry significant guidance for the LM. If your output schema has Field(description="Invoice date in YYYY-MM-DD format"), GEPA will never touch that description -- even if it's the source of failures.

Workaround: flatten field descriptions into the instruction

To bring field descriptions into GEPA's optimization surface, serialize them into the instruction before optimizing, then extract back out:

import dspy
import json
from pydantic import BaseModel, Field

# 1. Your original Pydantic model
class Invoice(BaseModel):
    vendor: str = Field(description="Company name of the vendor")
    date: str = Field(description="Invoice date in YYYY-MM-DD format")
    total: float = Field(description="Total amount due")

# 2. Serialize field descriptions into the instruction
field_guidance = "\n".join(
    f"- {name}: {info.description}"
    for name, info in Invoice.model_fields.items()
    if info.description
)

class ParseInvoice(dspy.Signature):
    # GEPA will optimize this entire docstring, including the field guidance
    f"""Extract invoice data from raw text.

    Output field guidelines:
    {field_guidance}"""

    text: str = dspy.InputField()
    invoice: Invoice = dspy.OutputField()

# 3. Optimize -- GEPA now sees and can rewrite the field guidance
gepa = dspy.GEPA(metric=metric, reflection_lm=reflection_lm, auto="medium")
optimized = gepa.compile(ParseInvoice, trainset=trainset)

# 4. After optimization, inspect the optimized instruction
# to see how GEPA refined the field guidance
dspy.inspect_history(n=1)

Limitations of this workaround:

• The optimized field guidance lives in the instruction string, not back in the Pydantic model. The Pydantic model still validates types, but its description fields remain unchanged.
• You must manually inspect the optimized instruction to see what GEPA changed about the field descriptions.
• For simple schemas (2-3 fields), this adds complexity with little benefit -- GEPA can usually compensate through the instruction alone.

When this is worth doing:

• Complex Pydantic models with 5+ fields where field descriptions carry important formatting or semantic guidance
• Structured output tasks where the LM consistently misinterprets specific fields despite good top-level instructions
• When field-level desc strings are doing heavy lifting (e.g., date formats, enum explanations, nested object guidance)

Gotchas

1. Claude writes GEPA metrics that return only a float. GEPA can use plain float scores, but its key advantage is textual feedback. When the metric returns {"score": 0.0, "feedback": "Expected positive but got negative; the review is sarcastic"}, the reflection LM uses that feedback to propose better instructions. Without feedback, GEPA degrades to blind search. Always return a dict with both score and feedback. 2. Claude uses a weak model as the reflection LM. The quality of proposed instructions depends entirely on the reflection model. Using gpt-4o-mini or a small local model for reflection produces generic, unhelpful instruction changes. Use a strong model (GPT-4o, Claude Sonnet) for reflection_lm -- the task LM can be cheaper. 3. Claude starts with `auto="heavy"` before validating the metric. A broken or noisy metric wastes the entire optimization budget. Start with auto="light" to verify the metric produces meaningful scores and feedback, then scale up to "medium" or "heavy" for production runs. 4. Claude does not run `dspy.Evaluate` before and after GEPA. Without a baseline measurement, there is no way to know if GEPA actually improved anything. Always evaluate the unoptimized program first, then compare against the optimized version. 5. Claude expects GEPA to optimize Pydantic field descriptions. GEPA only tunes the signature docstring (instruction). InputField(desc=...), OutputField(desc=...), and Pydantic Field(description=...) are never modified. If field descriptions are causing failures, flatten them into the instruction before optimizing (see the workaround in this skill).

When GEPA does not improve anything

If your optimized program scores the same as the baseline, GEPA is working correctly -- it is just not finding anything to fix.

The saturation diagnostic

GEPA improves instructions by reflecting on failures. If every minibatch is all-correct, the reflection LM never fires and the instructions stay unchanged. This means the task is saturated for the current task LM -- the model already solves it without better instructions.

Signs of saturation:

• Baseline score == optimized score (often both near 100%)
• Optimization finishes quickly with no instruction changes
• track_stats=True shows zero reflection calls

Three fixes for saturation

1. Harden the examples -- add adversarial, ambiguous, or edge-case examples that the model currently gets wrong. If your trainset is too easy, GEPA has no signal to work with. 2. Weaken the task LM -- use a smaller or cheaper model as the task LM. Counterintuitively, smaller models (1.2B-8B parameters) often benefit MORE from GEPA than larger ones. A 1.2B model can see 25+ point lifts on tasks where 8B+ models already saturate. The reflection LM should still be strong (GPT-4o, Claude Sonnet). 3. Accept the task is solved -- if your model already handles the task well, optimization is unnecessary. Ship it.

Practical strategy with free-tier models

Smaller models paired with GEPA can match larger models at zero cost. Free-tier models on OpenRouter (e.g., small Qwen or Llama variants) work as task LMs while a strong model handles reflection. This lets you run optimization loops with no API spend on the task LM side. Set seed=0 for reproducibility.

# Weaker task LM + strong reflection LM = maximum GEPA signal
task_lm = dspy.LM("openrouter/qwen/qwen3-1.7b:free", seed=0)
reflection_lm = dspy.LM("openai/gpt-4o", temperature=1.0, max_tokens=4096)
dspy.configure(lm=task_lm)

gepa = dspy.GEPA(metric=metric, reflection_lm=reflection_lm, auto="medium")
optimized = gepa.compile(program, trainset=trainset)

Additional resources

• dspy.GEPA API docs
• DSPy optimizer selection guide
• For constructor signatures and method reference, see reference.md
• For worked examples (sentiment classification, multi-step summarization), see examples.md

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Watch GEPA optimization in real time -- see /ai-watching-optimization
• Improving accuracy with other optimizers -- see /ai-improving-accuracy
• MIPROv2 for instruction + few-shot optimization -- see /dspy-miprov2
• Chain of thought reasoning as the inner module -- see /dspy-chain-of-thought
• Evaluating programs before and after optimization -- see /dspy-evaluate
• Iterative self-improvement at inference time -- see /dspy-refine
• Signatures and Pydantic outputs -- see /dspy-signatures for field descriptions, typed outputs, and gotchas about what optimizers can/cannot tune
• VizPy (commercial alternative for instruction optimization) -- see /dspy-vizpy
• For worked examples, see examples.md
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

dspy.GEPA Examples

Example 1: Instruction optimization for classification

A sentiment classifier optimized with GEPA. The feedback metric tells the reflection LM exactly what went wrong when the classifier mislabels an example, so GEPA can propose instructions that address common failure patterns like sarcasm or mixed sentiment.

from typing import Literal
import dspy
from dspy.evaluate import Evaluate

# Configure LMs
task_lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
reflection_lm = dspy.LM("openai/gpt-4o", temperature=1.0, max_tokens=4096)
dspy.configure(lm=task_lm)

# Define the classifier
class SentimentClassifier(dspy.Signature):
    """Classify the sentiment of a customer review."""
    review: str = dspy.InputField(desc="A customer review")
    sentiment: Literal["positive", "negative", "neutral"] = dspy.OutputField(
        desc="The sentiment of the review"
    )

classify = dspy.ChainOfThought(SentimentClassifier)

# Prepare training data (~50 examples)
trainset = [
    dspy.Example(
        review="Absolutely love this product! Best purchase I've made all year.",
        sentiment="positive",
    ).with_inputs("review"),
    dspy.Example(
        review="Broke after two days. Complete waste of money.",
        sentiment="negative",
    ).with_inputs("review"),
    dspy.Example(
        review="It works fine. Nothing special but gets the job done.",
        sentiment="neutral",
    ).with_inputs("review"),
    dspy.Example(
        review="Oh sure, because crashing every five minutes is a 'feature'.",
        sentiment="negative",
    ).with_inputs("review"),
    dspy.Example(
        review="The packaging was nice but the product itself is mediocre.",
        sentiment="neutral",
    ).with_inputs("review"),
    # ... add more examples to reach ~50
]

# Hold out some examples for validation
valset = trainset[40:]
trainset = trainset[:40]


# Define a feedback metric
def sentiment_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """Score the prediction and provide actionable feedback on failures."""
    correct = pred.sentiment == gold.sentiment

    if correct:
        return {"score": 1.0, "feedback": ""}

    # Build targeted feedback for the reflection LM
    feedback_parts = []

    # Detect common failure patterns
    review_lower = gold.review.lower()
    if gold.sentiment == "negative" and pred.sentiment == "positive":
        if any(word in review_lower for word in ["sure", "great", "love", "amazing"]):
            feedback_parts.append(
                "This review uses sarcasm -- positive words with negative intent. "
                "The instruction should tell the model to watch for sarcastic tone."
            )
        else:
            feedback_parts.append(
                f"Misclassified negative review as positive. "
                f"The review expresses dissatisfaction."
            )

    if gold.sentiment == "neutral" and pred.sentiment != "neutral":
        feedback_parts.append(
            "This review has mixed or mild signals. The instruction should clarify "
            "that reviews without strong positive or negative language are neutral."
        )

    if gold.sentiment == "positive" and pred.sentiment == "negative":
        feedback_parts.append(
            "Misclassified a genuinely positive review as negative. "
            "The instruction should not over-correct for sarcasm."
        )

    if not feedback_parts:
        feedback_parts.append(
            f"Expected '{gold.sentiment}' but predicted '{pred.sentiment}'. "
            f"Review: '{gold.review[:80]}...'"
        )

    return {"score": 0.0, "feedback": " ".join(feedback_parts)}


# Evaluate baseline
evaluator = Evaluate(devset=valset, metric=sentiment_metric, num_threads=4)
baseline_score = evaluator(classify)
print(f"Baseline score: {baseline_score}")

# Optimize with GEPA
gepa = dspy.GEPA(
    metric=sentiment_metric,
    reflection_lm=reflection_lm,
    auto="medium",
)
optimized_classify = gepa.compile(classify, trainset=trainset, valset=valset)

# Evaluate optimized program
optimized_score = evaluator(optimized_classify)
print(f"Optimized score: {optimized_score}")
print(f"Improvement: {baseline_score} -> {optimized_score}")

# Save the optimized program
optimized_classify.save("optimized_sentiment.json")

# Use it
result = optimized_classify(review="Yeah right, 'premium quality' that falls apart in a week.")
print(f"Sentiment: {result.sentiment}")
print(f"Reasoning: {result.reasoning}")

What this demonstrates:

• Feedback metric with failure analysis -- the metric detects sarcasm, mixed signals, and over-correction patterns, giving the reflection LM concrete guidance
• Structured feedback -- instead of just "wrong", the feedback says _why_ the instruction should change (e.g., "watch for sarcastic tone")
• Baseline comparison -- evaluating before and after GEPA to measure the actual improvement
• Class-based signature -- SentimentClassifier with typed Literal output constrains the label space
• Separate validation set -- prevents overfitting by holding out examples from training

Example 2: Instruction tuning for a generation task

A summarization pipeline where GEPA optimizes instructions based on multiple quality dimensions. The feedback metric scores summaries on faithfulness, conciseness, and completeness, giving per-dimension feedback so the reflection LM knows which aspect of the instruction to improve.

import dspy
from dspy.evaluate import Evaluate

# Configure LMs
task_lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
reflection_lm = dspy.LM("openai/gpt-4o", temperature=1.0, max_tokens=4096)
dspy.configure(lm=task_lm)


# Define a two-step summarization pipeline
class ExtractKeyPoints(dspy.Signature):
    """Extract the key points from an article."""
    article: str = dspy.InputField(desc="The full article text")
    key_points: str = dspy.OutputField(desc="Bullet-pointed list of key facts")


class WriteSummary(dspy.Signature):
    """Write a concise summary from key points."""
    key_points: str = dspy.InputField(desc="Extracted key points")
    summary: str = dspy.OutputField(desc="A 2-3 sentence summary")


class Summarizer(dspy.Module):
    def __init__(self):
        self.extract = dspy.ChainOfThought(ExtractKeyPoints)
        self.summarize = dspy.ChainOfThought(WriteSummary)

    def forward(self, article):
        extraction = self.extract(article=article)
        return self.summarize(key_points=extraction.key_points)


# Prepare training data
# Each example has an article and a reference summary
trainset = [
    dspy.Example(
        article=(
            "Researchers at MIT have developed a new battery technology that "
            "could double the range of electric vehicles. The solid-state "
            "battery uses a lithium-metal anode and a ceramic electrolyte, "
            "eliminating the risk of fire associated with liquid electrolytes. "
            "The team published their findings in Nature Energy and expects "
            "commercial production within five years."
        ),
        reference_summary=(
            "MIT researchers created a solid-state battery with a lithium-metal "
            "anode and ceramic electrolyte that could double EV range while "
            "eliminating fire risk. Commercial production is expected within "
            "five years."
        ),
    ).with_inputs("article"),
    dspy.Example(
        article=(
            "The European Central Bank raised interest rates by 0.25 percentage "
            "points to 4.5%, marking the tenth consecutive increase. ECB "
            "President Christine Lagarde cited persistent inflation in services "
            "and food prices. Markets reacted negatively, with the Euro Stoxx "
            "50 falling 1.2% on the announcement."
        ),
        reference_summary=(
            "The ECB raised rates by 0.25 points to 4.5% for the tenth "
            "straight increase, citing persistent inflation. European stocks "
            "fell 1.2% in response."
        ),
    ).with_inputs("article"),
    # ... add more examples to reach ~50
]

valset = trainset[40:]
trainset = trainset[:40]


# Define a multi-dimensional feedback metric
def summary_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """Score summaries on faithfulness, conciseness, and completeness."""
    summary = pred.summary
    article = gold.article
    reference = gold.reference_summary

    score = 0.0
    feedback_parts = []

    # Dimension 1: Conciseness (0.3 points)
    # Summary should be shorter than 40% of the article
    summary_words = len(summary.split())
    article_words = len(article.split())
    ratio = summary_words / max(article_words, 1)

    if ratio <= 0.4:
        score += 0.3
    elif ratio <= 0.6:
        score += 0.15
        feedback_parts.append(
            f"Summary is {summary_words} words ({ratio:.0%} of article). "
            f"Aim for under 40% of the article length. "
            f"The instruction should emphasize brevity."
        )
    else:
        feedback_parts.append(
            f"Summary is too long ({summary_words} words, {ratio:.0%} of article). "
            f"The instruction should strongly emphasize conciseness and "
            f"mention a target of 2-3 sentences."
        )

    # Dimension 2: Completeness (0.4 points)
    # Check if key terms from the reference appear in the summary
    ref_words = set(reference.lower().split())
    summary_words_set = set(summary.lower().split())
    # Filter to meaningful words (>4 chars)
    key_ref_words = {w for w in ref_words if len(w) > 4}
    if key_ref_words:
        overlap = len(key_ref_words & summary_words_set) / len(key_ref_words)
        score += 0.4 * overlap
        if overlap < 0.5:
            missing = key_ref_words - summary_words_set
            feedback_parts.append(
                f"Summary misses key information. Missing terms: "
                f"{', '.join(list(missing)[:5])}. "
                f"The instruction should emphasize capturing all main facts."
            )

    # Dimension 3: Faithfulness (0.3 points)
    # Basic check: summary should not introduce words absent from article
    article_words_set = set(article.lower().split())
    summary_unique = set(summary.lower().split())
    novel_words = summary_unique - article_words_set
    # Filter to meaningful novel words
    novel_meaningful = {w for w in novel_words if len(w) > 5 and w.isalpha()}
    if len(novel_meaningful) <= 2:
        score += 0.3
    elif len(novel_meaningful) <= 5:
        score += 0.15
        feedback_parts.append(
            f"Summary introduces terms not in the article: "
            f"{', '.join(list(novel_meaningful)[:3])}. "
            f"The instruction should say to only use information from the source."
        )
    else:
        feedback_parts.append(
            f"Summary adds too much information not in the article. "
            f"Novel terms: {', '.join(list(novel_meaningful)[:5])}. "
            f"The instruction must emphasize faithfulness to the source text."
        )

    # Per-predictor feedback for multi-step pipeline
    if pred_name == "extract" and feedback_parts:
        feedback_parts.insert(
            0, "The key-point extraction step may be missing important facts. "
        )
    elif pred_name == "summarize" and feedback_parts:
        feedback_parts.insert(
            0, "The summary-writing step needs improvement. "
        )

    feedback = " ".join(feedback_parts) if feedback_parts else ""
    return {"score": score, "feedback": feedback}


# Evaluate baseline
summarizer = Summarizer()
evaluator = Evaluate(devset=valset, metric=summary_metric, num_threads=4)
baseline_score = evaluator(summarizer)
print(f"Baseline score: {baseline_score}")

# Optimize with GEPA
gepa = dspy.GEPA(
    metric=summary_metric,
    reflection_lm=reflection_lm,
    auto="medium",
    reflection_minibatch_size=5,  # more context per reflection for generation
)
optimized_summarizer = gepa.compile(summarizer, trainset=trainset, valset=valset)

# Evaluate optimized pipeline
optimized_score = evaluator(optimized_summarizer)
print(f"Optimized score: {optimized_score}")
print(f"Improvement: {baseline_score} -> {optimized_score}")

# Save the optimized pipeline
optimized_summarizer.save("optimized_summarizer.json")

# Use it
result = optimized_summarizer(
    article=(
        "SpaceX successfully launched its Starship rocket on its third test "
        "flight, reaching orbital velocity for the first time. The vehicle "
        "re-entered the atmosphere but broke apart before landing. CEO Elon "
        "Musk called it a 'huge step forward' and said the next flight would "
        "attempt a controlled ocean landing within three months."
    )
)
print(f"Summary: {result.summary}")

What this demonstrates:

• Multi-step pipeline -- GEPA optimizes instructions for both extract and summarize predictors independently via round-robin component selection
• Per-predictor feedback -- the metric uses pred_name to tailor feedback to the specific step being optimized
• Multi-dimensional scoring -- conciseness (0.3), completeness (0.4), and faithfulness (0.3) are scored separately with targeted feedback for each
• Actionable feedback per dimension -- instead of just "bad summary", the metric says exactly which dimension failed and what the instruction should emphasize
• Larger minibatch -- reflection_minibatch_size=5 gives the reflection LM more context for generation tasks where quality is subjective
• Baseline comparison -- measuring improvement before and after optimization to validate the effort

Example 3: Diagnosing saturation

When GEPA returns the same score as the baseline, the task is saturated -- the model already solves it without better instructions. This example shows how to detect saturation and fix it by switching to a weaker task LM.

import dspy
from dspy.evaluate import Evaluate

# --- Step 1: Run GEPA and see no improvement ---

# A strong task LM that already handles the task well
task_lm = dspy.LM("openai/gpt-4o-mini")
reflection_lm = dspy.LM("openai/gpt-4o", temperature=1.0, max_tokens=4096)
dspy.configure(lm=task_lm)

classify = dspy.ChainOfThought("text -> label: Literal['positive', 'negative', 'neutral']")

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    correct = pred.label == gold.label
    if correct:
        return {"score": 1.0, "feedback": ""}
    return {
        "score": 0.0,
        "feedback": f"Expected '{gold.label}', got '{pred.label}'. Review: '{gold.text[:60]}...'",
    }

# Assume trainset and valset are prepared with ~50 examples
# trainset = [...]
# valset = [...]

evaluator = Evaluate(devset=valset, metric=metric, num_threads=4)

baseline_score = evaluator(classify)
print(f"Baseline: {baseline_score}")  # e.g., 96.0

gepa = dspy.GEPA(
    metric=metric,
    reflection_lm=reflection_lm,
    auto="medium",
    track_stats=True,
)
optimized = gepa.compile(classify, trainset=trainset, valset=valset)

optimized_score = evaluator(optimized)
print(f"Optimized: {optimized_score}")  # e.g., 96.0 -- same as baseline

# Diagnosis: baseline == optimized means every minibatch was all-correct,
# so the reflection LM was never called. The task is saturated.

# --- Step 2: Fix by using a weaker task LM ---

# Switch to a smaller model that will struggle more, giving GEPA signal
weak_lm = dspy.LM("openrouter/qwen/qwen3-1.7b:free", seed=0)
dspy.configure(lm=weak_lm)

classify_weak = dspy.ChainOfThought("text -> label: Literal['positive', 'negative', 'neutral']")

weak_baseline = evaluator(classify_weak)
print(f"Weak model baseline: {weak_baseline}")  # e.g., 62.0

gepa = dspy.GEPA(
    metric=metric,
    reflection_lm=reflection_lm,  # reflection LM stays strong
    auto="medium",
)
optimized_weak = gepa.compile(classify_weak, trainset=trainset, valset=valset)

weak_optimized = evaluator(optimized_weak)
print(f"Weak model optimized: {weak_optimized}")  # e.g., 87.0 -- 25-point lift
print(f"Lift: {weak_optimized - weak_baseline:.1f} points")

What this demonstrates:

• Saturation diagnostic -- baseline == optimized means every minibatch was all-correct and GEPA had no failures to reflect on
• Weaker task LM as a fix -- switching from gpt-4o-mini to a 1.7B model creates optimization signal, enabling large lifts
• Strong reflection LM stays -- the reflection LM should always be a capable model regardless of the task LM
• Free-tier reproducibility -- using seed=0 and free OpenRouter models for cost-zero optimization loops

Condensed from dspy.ai/api/optimizers/GEPA/. Verify against upstream for latest.

dspy.GEPA — API Reference

Inherits from: Teleprompter

Constructor

dspy.GEPA(
    metric,                              # GEPAFeedbackMetric (required)
    *,
    # Budget (exactly one required)
    auto=None,                           # "light", "medium", or "heavy"
    max_full_evals=None,                 # int -- full validation passes allowed
    max_metric_calls=None,               # int -- total metric invocations allowed

    # Reflection
    reflection_lm=None,                  # LM for proposing new instructions
    reflection_minibatch_size=3,         # examples per reflection step
    candidate_selection_strategy="pareto",  # "pareto" or "current_best"
    skip_perfect_score=True,             # skip examples already scoring perfectly
    add_format_failure_as_feedback=False, # include format errors in feedback
    instruction_proposer=None,           # custom ProposalFn
    component_selector="round_robin",    # which predictor to improve next

    # Merging
    use_merge=True,                      # merge successful variants
    max_merge_invocations=5,             # merge attempt limit

    # Evaluation
    num_threads=None,                    # parallel evaluation threads
    failure_score=0.0,                   # score for failed examples
    perfect_score=1.0,                   # score that counts as perfect

    # Logging
    log_dir=None,                        # directory for optimization logs
    track_stats=False,                   # return detailed metadata
    track_best_outputs=False,            # retain best outputs per task
    use_wandb=False,                     # W&B integration
    use_mlflow=False,                    # MLflow integration

    # Reproducibility
    seed=0,                              # random seed
)

compile()

optimized = gepa.compile(
    student,              # dspy.Module (required)
    *,
    trainset,             # list[Example] (required)
    valset=None,          # list[Example] | None -- auto-uses trainset if None
)

Returns: Optimized dspy.Module with improved instructions. If track_stats=True, result has a detailed_results attribute of type DspyGEPAResult.

Note: The teacher parameter exists in the signature but is not currently supported (assert teacher is None).

Key methods

GEPAFeedbackMetric protocol

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """
    Args:
        gold: Expected Example (ground truth)
        pred: Model Prediction
        trace: Full program execution trace
        pred_name: Name of the predictor being optimized
        pred_trace: Sub-trace for this predictor

    Returns:
        float -- simple score
        OR {"score": float, "feedback": str} -- score with textual feedback
    """

DspyGEPAResult (when track_stats=True)

Key properties:

• best_idx — index of best candidate
• best_candidate — the best optimized module
• candidates — all candidate modules
• val_aggregate_scores — scores per candidate
• best_outputs_valset — best outputs per task (if track_best_outputs=True)

What GEPA does NOT optimize