Dspy Bootstrap Few Shot

Bootstrap Few-Shot Demonstrations

Guide the user through using dspy.BootstrapFewShot to automatically generate and select high-quality few-shot demonstrations for their DSPy program. This is the simplest optimizer and the recommended first step before trying heavier optimizers.

What is BootstrapFewShot

dspy.BootstrapFewShot takes your program, a training set, and a metric, then:

1. Runs your program on each training example 2. Keeps the traces (input/output pairs) where the metric passes 3. Attaches the best traces as few-shot demonstrations to your program's predictors

The result is a copy of your program with working examples baked into the prompt — so the LM sees "here's how I solved similar problems" every time it runs.

optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized = optimizer.compile(my_program, trainset=trainset)

When to use it

• First optimizer to try — it is fast, simple, and often gives a meaningful lift
• You have ~50+ labeled examples (fewer can work but results vary)
• You want to add few-shot demonstrations without hand-writing them
• You want a quick baseline before trying heavier optimizers

Basic usage

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# 1. Define your program
qa = dspy.ChainOfThought("question -> answer")

# 2. Prepare your data (mark inputs with .with_inputs())
trainset = [
    dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"),
    # ... ~50+ examples
]

devset = [
    dspy.Example(question="Who wrote Hamlet?", answer="Shakespeare").with_inputs("question"),
    # ... held-out examples for evaluation
]

# 3. Define a metric
def metric(example, prediction, trace=None):
    return prediction.answer.strip().lower() == example.answer.strip().lower()

# 4. Evaluate baseline
evaluator = Evaluate(devset=devset, metric=metric, num_threads=4)
baseline = evaluator(qa)
print(f"Baseline: {baseline:.1f}%")

# 5. Optimize
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# 6. Evaluate optimized program
improved = evaluator(optimized_qa)
print(f"Optimized: {improved:.1f}%")

Key parameters

optimizer = dspy.BootstrapFewShot(
    metric=metric,                # Scoring function(example, prediction, trace) -> bool/float
    max_bootstrapped_demos=4,     # Max bootstrapped (generated) demos per predictor. Default: 4
    max_labeled_demos=16,         # Max labeled (from trainset) demos per predictor. Default: 16
    max_rounds=1,                 # Number of bootstrap rounds. Default: 1
    max_errors=None,              # Error tolerance. Default: None (uses dspy.settings.max_errors)
    metric_threshold=None,        # Numerical threshold for accepting bootstrap examples
    teacher_settings=None,        # Config dict for the teacher model (e.g., {"lm": teacher_lm})
)

What the parameters control

• `max_bootstrapped_demos` — How many auto-generated demonstrations to include in the prompt. These come from running the program on training examples and keeping traces that pass the metric. Start with 4, increase to 8 if you have a complex task.

• `max_labeled_demos` — How many examples from your trainset to include directly as demonstrations (without running through the program first). These are simpler input/output pairs. Set to 0 if you only want bootstrapped demos.

• `max_rounds` — Number of bootstrapping iterations. In each round, the optimizer runs the program (with any demos from previous rounds) and collects new passing traces. More rounds can find better demos but take longer. Usually 1 is sufficient.

• `max_errors` — How many failed examples to tolerate before the optimizer stops. Defaults to None (uses dspy.settings.max_errors). Increase if your task is noisy or the metric is strict.

• `metric_threshold` — Numerical threshold for accepting bootstrap examples. When set, only traces scoring above this threshold become demos. Useful when your metric returns floats rather than booleans.

• `teacher_settings` — Configuration dict for a teacher model. Pass {"lm": teacher_lm} to use a stronger model for generating traces while the student uses a cheaper model.

How bootstrapping works

Understanding the process helps you debug when results are unexpected.

Round 1: 1. The optimizer picks examples from trainset 2. For each example, it runs your program to get a prediction 3. It scores the prediction with your metric(example, prediction, trace) 4. If the metric passes, the full trace (inputs + outputs, including intermediate reasoning) is saved as a candidate demo 5. The best max_bootstrapped_demos traces are attached to each predictor

Round 2+ (if `max_rounds > 1`): 1. The program now has demos from round 1 2. The optimizer runs the program again on more training examples 3. New passing traces are collected — these are often better because the program already has some demos 4. The demo set is updated with the best traces so far

The result: Your program's predictors now have few-shot demonstrations in their prompts. When the program runs, the LM sees these worked examples before processing the new input.

Trace-aware metrics

The trace parameter in your metric is None during evaluation but set during optimization. Use this to apply stricter filtering during bootstrapping:

def metric(example, prediction, trace=None):
    correct = prediction.answer.strip().lower() == example.answer.strip().lower()
    if trace is not None:
        # During optimization: require good reasoning too
        has_reasoning = len(getattr(prediction, "reasoning", "")) > 50
        return correct and has_reasoning
    # During evaluation: only check correctness
    return correct

This ensures bootstrapped demos have both correct answers and clear reasoning, producing higher-quality demonstrations.

Saving and loading optimized programs

After optimization, save the program so you don't have to re-optimize every time:

# Save
optimized_qa.save("optimized_qa.json")

# Load later
loaded_qa = dspy.ChainOfThought("question -> answer")
loaded_qa.load("optimized_qa.json")

# Use it
result = loaded_qa(question="What is the capital of Japan?")

For custom modules:

class MyPipeline(dspy.Module):
    def __init__(self):
        self.step1 = dspy.ChainOfThought("question -> search_query")
        self.step2 = dspy.ChainOfThought("question, search_query -> answer")

    def forward(self, question):
        query = self.step1(question=question)
        return self.step2(question=question, search_query=query.search_query)

# Save after optimization
optimized_pipeline.save("pipeline.json")

# Load
loaded = MyPipeline()
loaded.load("pipeline.json")

The saved file contains the few-shot demonstrations for each predictor. The program structure itself is defined in code — save and load only handle the learned demos and parameters.

Using with multi-step programs

BootstrapFewShot works on every predictor in your program. For a multi-step pipeline, each step gets its own demonstrations:

class RAG(dspy.Module):
    def __init__(self):
        self.generate_query = dspy.ChainOfThought("question -> search_query")
        self.generate_answer = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        query = self.generate_query(question=question)
        # Assume some retrieval step here
        context = retrieve(query.search_query)
        return self.generate_answer(context=context, question=question)

optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized_rag = optimizer.compile(RAG(), trainset=trainset)
# Both generate_query and generate_answer now have bootstrapped demos

When to upgrade to a heavier optimizer

BootstrapFewShot is a great starting point, but you may want to upgrade if:

A typical progression:

1. BootstrapFewShot — fast, first pass (~50 examples) 2. BootstrapFewShotWithRandomSearch — better demo selection (~200 examples) 3. MIPROv2 — full prompt optimization (~200 examples) 4. BootstrapFinetune — weight tuning (~500+ examples)

Troubleshooting

No demos were bootstrapped:

• Your metric may be too strict — check that at least some training examples pass
• Run a quick evaluation on your trainset to see the pass rate
• Lower the bar in your metric or fix data quality issues

Accuracy didn't improve (or got worse):

• Try increasing max_bootstrapped_demos (e.g., 8)
• Try setting max_labeled_demos=0 to only use bootstrapped demos
• Check that your trainset is representative of the task
• Ensure your devset is held out (not overlapping with trainset)

Optimization is slow:

• Reduce trainset size (50-100 examples is often enough)
• Use a faster/cheaper LM for bootstrapping, then evaluate with the target LM
• Reduce max_rounds to 1

Gotchas

• Claude sets `max_labeled_demos` too high, bloating the prompt. The default is 16, which adds up to 16 raw input/output pairs from the trainset to the prompt. For tasks with long inputs, this can consume most of the context window. Start with max_labeled_demos=4 and increase only if accuracy improves.
• Claude forgets `.with_inputs()` on training examples. Every dspy.Example in the trainset must call .with_inputs("field1", "field2") to mark which fields are inputs vs labels. Without it, the optimizer cannot distinguish inputs from expected outputs and bootstrapping silently produces garbage demos.
• Claude overlaps trainset and devset. If the devset contains examples also in the trainset, evaluation scores are inflated because the optimizer has already seen those examples. Always use a held-out devset with no overlap.
• Claude uses a strict exact-match metric that rejects most traces. If fewer than ~10% of training examples pass the metric, barely any demos get bootstrapped. Check your metric pass rate on the trainset first. Relax the metric (e.g., use containment instead of exact match) or fix data quality before optimizing.
• Claude does not compare baseline vs optimized scores. Without a baseline evaluation, there is no way to know if optimization helped or hurt. Always evaluate the unoptimized program on the devset first, then compare after optimization.

Additional resources

• dspy.BootstrapFewShot API docs
• reference.md — constructor parameters, compile() method, key behaviors
• examples.md — QA optimization, classification with trace-aware metrics

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Need to prepare training data? Use /dspy-data
• Need to write a metric or run evaluation? Use /dspy-evaluate
• Want to try random search over demo sets? Use /dspy-bootstrap-rs
• Want the best prompt optimization? Use /dspy-miprov2
• For the full measure-improve-verify loop, see /ai-improving-accuracy
• For worked examples, see examples.md
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

BootstrapFewShot Examples

QA Optimization with Exact Match Metric

A question-answering pipeline optimized with BootstrapFewShot. Shows the full workflow: baseline evaluation, optimization, and before/after comparison.

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# Program
qa = dspy.ChainOfThought("question -> answer")

# Training set (~50 examples for bootstrapping)
trainset = [
    dspy.Example(question="What is the chemical symbol for water?", answer="H2O").with_inputs("question"),
    dspy.Example(question="What planet is closest to the Sun?", answer="Mercury").with_inputs("question"),
    dspy.Example(question="What is the square root of 144?", answer="12").with_inputs("question"),
    dspy.Example(question="Who painted the Mona Lisa?", answer="Leonardo da Vinci").with_inputs("question"),
    dspy.Example(question="What is the capital of Japan?", answer="Tokyo").with_inputs("question"),
    dspy.Example(question="How many legs does a spider have?", answer="8").with_inputs("question"),
    dspy.Example(question="What gas do plants absorb from the atmosphere?", answer="Carbon dioxide").with_inputs("question"),
    dspy.Example(question="What is the largest ocean on Earth?", answer="Pacific Ocean").with_inputs("question"),
    dspy.Example(question="Who developed the theory of relativity?", answer="Albert Einstein").with_inputs("question"),
    dspy.Example(question="What is the freezing point of water in Fahrenheit?", answer="32").with_inputs("question"),
    dspy.Example(question="What is the powerhouse of the cell?", answer="Mitochondria").with_inputs("question"),
    dspy.Example(question="How many bones are in the adult human body?", answer="206").with_inputs("question"),
    dspy.Example(question="What is the hardest natural substance?", answer="Diamond").with_inputs("question"),
    dspy.Example(question="What language has the most native speakers?", answer="Mandarin Chinese").with_inputs("question"),
    dspy.Example(question="What is the speed of sound in m/s at sea level?", answer="343").with_inputs("question"),
    dspy.Example(question="Who wrote The Great Gatsby?", answer="F. Scott Fitzgerald").with_inputs("question"),
    dspy.Example(question="What is the atomic number of carbon?", answer="6").with_inputs("question"),
    dspy.Example(question="What is the tallest mountain on Earth?", answer="Mount Everest").with_inputs("question"),
    dspy.Example(question="What year was the internet invented?", answer="1969").with_inputs("question"),
    dspy.Example(question="What is the currency of the United Kingdom?", answer="Pound sterling").with_inputs("question"),
]

# Held-out dev set (never used for training)
devset = [
    dspy.Example(question="What is the capital of Australia?", answer="Canberra").with_inputs("question"),
    dspy.Example(question="What element does 'O' represent on the periodic table?", answer="Oxygen").with_inputs("question"),
    dspy.Example(question="How many sides does a hexagon have?", answer="6").with_inputs("question"),
    dspy.Example(question="Who invented the telephone?", answer="Alexander Graham Bell").with_inputs("question"),
    dspy.Example(question="What is the largest mammal?", answer="Blue whale").with_inputs("question"),
    dspy.Example(question="What is the boiling point of water in Celsius?", answer="100").with_inputs("question"),
    dspy.Example(question="What continent is Brazil on?", answer="South America").with_inputs("question"),
    dspy.Example(question="How many planets are in the solar system?", answer="8").with_inputs("question"),
    dspy.Example(question="Who wrote 1984?", answer="George Orwell").with_inputs("question"),
    dspy.Example(question="What is the chemical symbol for gold?", answer="Au").with_inputs("question"),
]

# Metric: normalized exact match
def exact_match(example, prediction, trace=None):
    pred = prediction.answer.strip().lower()
    gold = example.answer.strip().lower()
    return pred == gold

# Evaluate baseline
evaluator = Evaluate(
    devset=devset,
    metric=exact_match,
    num_threads=4,
    display_progress=True,
    display_table=5,
)
baseline_score = evaluator(qa)
print(f"Baseline: {baseline_score:.1f}%")

# Optimize with BootstrapFewShot
optimizer = dspy.BootstrapFewShot(
    metric=exact_match,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,
)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# Evaluate optimized program
optimized_score = evaluator(optimized_qa)
print(f"Optimized: {optimized_score:.1f}%")
print(f"Delta: {optimized_score - baseline_score:+.1f}%")

# Save for production use
optimized_qa.save("optimized_qa.json")

Classification with Bootstrapped Demos

A sentiment classifier optimized with BootstrapFewShot. Demonstrates typed outputs, a classification metric, and trace-aware filtering.

import dspy
from typing import Literal
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# Signature with typed output
class ClassifySentiment(dspy.Signature):
    """Classify the sentiment of a product review."""
    review: str = dspy.InputField(desc="A product review from a customer")
    sentiment: Literal["positive", "negative", "neutral"] = dspy.OutputField(desc="The sentiment label")

# Program
classifier = dspy.ChainOfThought(ClassifySentiment)

# Training set
trainset = [
    dspy.Example(review="Absolutely love this product! Best purchase I've made all year.", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Broke after two days. Complete waste of money.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="It works fine. Nothing special but gets the job done.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="The quality exceeded my expectations. Highly recommend!", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Terrible customer service. Product arrived damaged.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="Average product. Decent for the price.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="This changed my life! Can't imagine going back.", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Doesn't work as advertised. Very disappointed.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="It's okay. Not great, not terrible.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="Five stars! Everything I wanted and more.", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Returned it the same day. Awful quality.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="Meets basic expectations. Would consider buying again.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="My whole family loves it. Ordering more as gifts!", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Stopped working after a week. No response from support.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="Solid product. Does what it says.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="Incredible value for money. Blown away!", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Flimsy and cheap. Not worth half the price.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="Pretty standard. Works as expected.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="The best in its category. Perfection!", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Worst purchase I've ever made. Stay away.", sentiment="negative").with_inputs("review"),
]

devset = [
    dspy.Example(review="Great build quality and fast shipping!", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Not what I expected. The description was misleading.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="It's a standard product. Nothing to complain about.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="Love the design and it works perfectly.", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Arrived broken. Requesting a refund.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="Functional. Does the basics well enough.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="Outstanding! This is premium quality.", sentiment="positive").with_inputs("review"),
    dspy.Example(review="Poor materials. Feels like it'll break any moment.", sentiment="negative").with_inputs("review"),
    dspy.Example(review="Middle of the road. Neither impressed nor disappointed.", sentiment="neutral").with_inputs("review"),
    dspy.Example(review="Exceeded all my expectations. A must-buy!", sentiment="positive").with_inputs("review"),
]

# Trace-aware metric: during bootstrapping, require that reasoning
# mentions the key sentiment signals from the review
def classify_metric(example, prediction, trace=None):
    correct = prediction.sentiment == example.sentiment
    if trace is not None:
        # During optimization: also require reasoning
        reasoning = getattr(prediction, "reasoning", "")
        has_reasoning = len(reasoning) > 20
        return correct and has_reasoning
    return correct

# Evaluate baseline
evaluator = Evaluate(
    devset=devset,
    metric=classify_metric,
    num_threads=4,
    display_progress=True,
    display_table=5,
)
baseline_score = evaluator(classifier)
print(f"Baseline: {baseline_score:.1f}%")

# Optimize — use only bootstrapped demos (no raw labeled demos)
# This forces all demos to include reasoning traces
optimizer = dspy.BootstrapFewShot(
    metric=classify_metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=0,   # only bootstrapped demos with reasoning
)
optimized_classifier = optimizer.compile(classifier, trainset=trainset)

# Evaluate optimized program
optimized_score = evaluator(optimized_classifier)
print(f"Optimized: {optimized_score:.1f}%")
print(f"Delta: {optimized_score - baseline_score:+.1f}%")

# Save for production use
optimized_classifier.save("optimized_classifier.json")

# Load later
loaded = dspy.ChainOfThought(ClassifySentiment)
loaded.load("optimized_classifier.json")
result = loaded(review="This product is a game changer!")
print(f"Sentiment: {result.sentiment}")

Condensed from dspy.ai/api/optimizers/BootstrapFewShot/. Verify against upstream for latest.

dspy.BootstrapFewShot — API Reference

Constructor

dspy.BootstrapFewShot(
    metric=None,                 # Callable | None
    metric_threshold=None,       # float | None
    teacher_settings=None,       # dict | None
    max_bootstrapped_demos=4,    # int
    max_labeled_demos=16,        # int
    max_rounds=1,                # int
    max_errors=None,             # int | None
)

Inheritance

BootstrapFewShot extends Teleprompter.

Methods

compile()

optimizer.compile(
    student,                # dspy.Module (required)
    *,
    teacher=None,           # dspy.Module | None
    trainset,               # list[Example] (required)
) -> Module

Orchestrates bootstrapping: prepares student/teacher, maps predictors, runs bootstrap rounds, and attaches the best demonstrations to each predictor.

Returns: The compiled student module with _compiled = True and bootstrapped demos attached to each predictor.

get_params()

optimizer.get_params() -> dict[str, Any]

Returns all configuration parameters as a dictionary.

Key behaviors

• Each bootstrap round uses a fresh LM instance with temperature=1.0 to bypass caches and gather diverse traces
• Successfully bootstrapped examples are accepted immediately; rounds continue only until one succeeds per example
• When teacher_settings is provided, the teacher model generates traces while the student receives the resulting demos
• The compile() method returns a copy of the student — the original module is not modified
• Bootstrapped demos include intermediate fields (e.g., reasoning from ChainOfThought), while labeled demos are raw input/output pairs