Dspy Bootstrap Rs

Optimize Few-Shot Demos with dspy.BootstrapFewShotWithRandomSearch

Guide the user through using DSPy's BootstrapFewShotWithRandomSearch optimizer to find the best set of few-shot demonstrations for their program. This optimizer runs BootstrapFewShot multiple times with different random seeds and keeps the candidate program that scores highest on a metric.

What it is

BootstrapFewShotWithRandomSearch (also known as BootstrapRS) is a prompt optimizer that searches over multiple candidate sets of few-shot demonstrations to find the best one. It wraps BootstrapFewShot and runs it repeatedly with different random subsets of training examples, then evaluates each candidate program on a held-out portion of the trainset.

trainset ──> [ BootstrapFewShot run 1 ] ──> candidate program 1 ──┐
         ──> [ BootstrapFewShot run 2 ] ──> candidate program 2 ──┤
         ──> [ BootstrapFewShot run 3 ] ──> candidate program 3 ──┼──> evaluate all ──> best program
         ──> ...                                                  │
         ──> [ BootstrapFewShot run N ] ──> candidate program N ──┘

How it improves on BootstrapFewShot

BootstrapFewShot runs once: it bootstraps demonstrations from your training data, picks a fixed set, and returns a single optimized program. The result depends heavily on which examples happened to be selected and which traces succeeded. You might get lucky or unlucky.

BootstrapFewShotWithRandomSearch removes that luck factor. It runs the bootstrap process multiple times (controlled by num_candidate_programs), each time with a different random sample of training examples. Each candidate program gets scored on a validation set, and the optimizer returns the highest-scoring one.

The trade-off is straightforward: more compute for more reliable results.

Basic usage

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or any LiteLLM-supported provider
dspy.configure(lm=lm)

# 1. Define your program
qa = dspy.ChainOfThought("question -> answer")

# 2. Prepare training data (50-200+ examples recommended)
trainset = [
    dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="What is 2 + 2?", answer="4").with_inputs("question"),
    # ... more examples
]

# 3. Define a metric
def metric(example, prediction, trace=None):
    return prediction.answer.strip().lower() == example.answer.strip().lower()

# 4. Optimize with random search
optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,
    num_candidate_programs=16,
)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# 5. Use the optimized program
result = optimized_qa(question="What is the capital of Germany?")
print(result.answer)

# 6. Save for later
optimized_qa.save("optimized_qa.json")

Key parameters

dspy.BootstrapFewShotWithRandomSearch(
    metric,                          # Scoring function: (example, prediction, trace) -> float|bool
    max_bootstrapped_demos=4,        # Max demos generated by running the program on training examples
    max_labeled_demos=16,            # Max demos taken directly from labeled training data
    num_candidate_programs=16,       # How many random bootstrap runs to try
    num_threads=None,                # Threads for parallel evaluation of candidates
    stop_at_score=None,              # Early-stop if a candidate reaches this score
    metric_threshold=None,           # Min metric score for a bootstrapped demo to be kept
)

max_bootstrapped_demos vs max_labeled_demos

These two parameters control where demonstrations come from:

• Bootstrapped demos are generated by running your program on training examples and keeping the traces where the metric passes. These are powerful because they show the LM its own successful reasoning patterns, including intermediate steps like chain-of-thought reasoning.

• Labeled demos are taken directly from your training data as input-output pairs. They don't include intermediate reasoning steps, but they're reliable because they use your gold-standard answers.

The optimizer includes up to max_bootstrapped_demos bootstrapped demos plus up to max_labeled_demos labeled demos in each candidate program's prompt.

Guidance:

• Start with max_bootstrapped_demos=4, max_labeled_demos=4 for most tasks.
• Increase max_labeled_demos (up to 8-16) if you have high-quality labeled data and your model benefits from more examples.
• Increase max_bootstrapped_demos (up to 4-8) if your task involves chain-of-thought or multi-step reasoning where seeing worked examples helps.
• Keep the total number of demos reasonable -- too many demos bloat the prompt and can hurt performance or exceed context limits.

How random search works

Each candidate program is built by a separate BootstrapFewShot run. The randomness comes from:

1. Shuffled training data: Each run sees a different random ordering of training examples, so different examples get bootstrapped. 2. Different demo subsets: The random ordering means each candidate ends up with a different combination of bootstrapped and labeled demos.

After all candidate programs are generated, the optimizer evaluates each one on a validation set (a portion of your trainset that was held out). The candidate with the highest validation score wins.

This is conceptually similar to hyperparameter random search: instead of searching over learning rates or layer sizes, you're searching over which few-shot demos to include in the prompt.

Computational cost

The cost scales linearly with num_candidate_programs:

Each candidate program requires: 1. One BootstrapFewShot run (bootstrapping demos from trainset) 2. One evaluation pass over the validation set

Cost estimate: If a single BootstrapFewShot run costs ~$0.50, then 16 candidate programs costs ~$8. With a larger trainset or more expensive model, plan for $5-$20.

Tip: Start with num_candidate_programs=8 to get a quick sense of how much random search helps, then increase to 16 or 25 if the improvement justifies the cost.

When to use BootstrapFewShotWithRandomSearch

Use BootstrapFewShotWithRandomSearch when:

• You have 50-200+ training examples
• Basic BootstrapFewShot gives inconsistent results across runs
• You want better few-shot demos without optimizing instructions
• You have budget for 10-20x the cost of a single BootstrapFewShot run
• You want a solid middle ground between BootstrapFewShot and MIPROv2

Use BootstrapFewShot instead when:

• You have fewer than 50 examples
• You want the fastest possible optimization
• Budget is very tight
• You're just prototyping and will optimize more later

Use MIPROv2 instead when:

• You want to optimize instructions and demos together (BootstrapRS only optimizes demos)
• You have 200+ examples and budget for a thorough search
• You've already tried BootstrapRS and want to push further
• You want the best prompt optimization DSPy offers

Quick & cheap          Solid middle ground         Best quality
BootstrapFewShot  -->  BootstrapFewShotWithRS  -->  MIPROv2
~$0.50                 ~$5-20                       ~$5-50
Few-shot demos only    Few-shot demos (searched)    Instructions + few-shot demos
1 candidate            N candidates                 Bayesian optimization

Using a teacher model

Use a larger model to generate high-quality bootstrapped demos, then deploy with a cheaper student model:

teacher_lm = dspy.LM("openai/gpt-4o")  # or any LiteLLM-supported provider
student_lm = dspy.LM("openai/gpt-4o-mini")  # or any LiteLLM-supported provider

dspy.configure(lm=student_lm)

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    num_candidate_programs=16,
    teacher_settings={"lm": teacher_lm},
)
optimized = optimizer.compile(my_program, trainset=trainset)
# optimized runs on student_lm but uses demos generated by teacher_lm

Early stopping with stop_at_score

Skip evaluating remaining candidates once a "good enough" program is found:

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    num_candidate_programs=25,
    stop_at_score=95.0,  # stop as soon as a candidate scores >= 95%
)

This is useful when you set num_candidate_programs high but want to save cost if an early candidate is already excellent.

Passing an optimized program to further optimization

You can stack optimizers. Run BootstrapRS first to find great demos, then pass the result to MIPROv2 to refine instructions on top:

# Step 1: Find best demos with random search
bootstrap_optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,
    num_candidate_programs=16,
)
bootstrapped = bootstrap_optimizer.compile(my_program, trainset=trainset)

# Step 2: Refine instructions with MIPROv2
mipro_optimizer = dspy.MIPROv2(metric=metric, auto="medium")
final = mipro_optimizer.compile(bootstrapped, trainset=trainset)

Gotchas

1. Claude uses the same data for training and validation. BootstrapRS evaluates candidates on a held-out validation set. If you pass all data as trainset without a separate valset, the optimizer splits it internally, but you get no control over the split. Pass valset explicitly for reproducible results: optimizer.compile(program, trainset=trainset, valset=devset). 2. Claude sets `num_candidate_programs` too low. With num_candidate_programs=3 the random search barely explores the space. The default of 16 is a good starting point. Fewer than 8 rarely finds materially better demos than plain BootstrapFewShot. 3. Claude sets `max_labeled_demos=16` with multi-step pipelines. Each predictor in the pipeline gets up to max_labeled_demos + max_bootstrapped_demos demos. A 3-step pipeline with 16+4 demos per step = 60 demos total, which can blow past context limits. Use 2-4 demos per type for multi-step pipelines. 4. Claude forgets the `candidate_programs` attribute on the result. The optimized program has a candidate_programs attribute containing all scored candidates. This is useful for inspecting how much variance exists and whether more search would help. 5. Claude runs BootstrapRS with fewer than 50 training examples. With fewer than ~50 examples, the random search has too little data to meaningfully differentiate candidates. Use plain BootstrapFewShot instead, or collect more data.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• BootstrapFewShot for the simpler single-run version -- see /ai-improving-accuracy
• MIPROv2 for instruction + demo optimization -- see /ai-improving-accuracy
• Evaluate for measuring quality with metrics and devsets -- see /dspy-evaluate
• Data handling for preparing training sets -- see /dspy-data
• Improving accuracy for the full optimization decision framework -- see /ai-improving-accuracy
• For worked examples, see examples.md
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• dspy.BootstrapFewShotWithRandomSearch API docs
• DSPy optimizer selection guide
• For constructor signatures and method reference, see reference.md
• For worked examples (QA optimization, multi-step pipeline), see examples.md

dspy-bootstrap-rs -- Worked Examples

Example 1: QA optimization with random search

Optimize a question-answering module by searching over multiple candidate demo sets. This shows the basic end-to-end workflow: prepare data, define a metric, run the optimizer, and compare against baseline.

import dspy
from dspy.evaluate import Evaluate

# --- Setup ---

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# --- Data ---

dataset = [
    ("What language is DSPy written in?", "Python"),
    ("Who developed the transformer architecture?", "Google"),
    ("What does LLM stand for?", "Large Language Model"),
    ("What year was GPT-3 released?", "2020"),
    ("What framework does DSPy build on for LM calls?", "LiteLLM"),
    ("What is retrieval-augmented generation?", "Combining retrieval with generation"),
    ("What is the purpose of few-shot prompting?", "Providing examples to guide the model"),
    ("What does RLHF stand for?", "Reinforcement Learning from Human Feedback"),
    ("What is a token in NLP?", "A unit of text processed by the model"),
    ("What is prompt engineering?", "Designing inputs to get better LM outputs"),
    ("What is chain-of-thought prompting?", "Asking the model to show reasoning steps"),
    ("What is fine-tuning?", "Training a pre-trained model on task-specific data"),
    ("What does zero-shot mean?", "Performing a task without any examples"),
    ("What is an embedding?", "A dense vector representation of text"),
    ("What is attention in transformers?", "A mechanism for weighing token relevance"),
    ("What is beam search?", "A decoding strategy that keeps top-k candidates"),
    ("What is temperature in LM sampling?", "A parameter controlling output randomness"),
    ("What is a system prompt?", "Instructions that set the LM's behavior"),
    ("What is grounding in AI?", "Connecting model outputs to factual sources"),
    ("What does RAG stand for?", "Retrieval-Augmented Generation"),
]

examples = [
    dspy.Example(question=q, answer=a).with_inputs("question")
    for q, a in dataset
]

# Split into train and dev sets
trainset = examples[:15]
devset = examples[15:]


# --- Metric ---

def answer_match(example, prediction, trace=None):
    """Check if the predicted answer matches the gold answer (case-insensitive)."""
    pred = prediction.answer.strip().lower()
    gold = example.answer.strip().lower()
    # Exact match or gold answer is contained in the prediction
    return gold in pred


# --- Baseline ---

qa = dspy.ChainOfThought("question -> answer")

evaluator = Evaluate(
    devset=devset,
    metric=answer_match,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

baseline_score = evaluator(qa)
print(f"Baseline score: {baseline_score:.1f}%")


# --- Optimize with BootstrapFewShotWithRandomSearch ---

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=answer_match,
    max_bootstrapped_demos=4,     # Up to 4 program-generated demos
    max_labeled_demos=4,          # Up to 4 labeled demos from trainset
    num_candidate_programs=10,    # Try 10 different random demo sets
    num_threads=4,
)

optimized_qa = optimizer.compile(qa, trainset=trainset)


# --- Evaluate optimized program ---

optimized_score = evaluator(optimized_qa)
print(f"Baseline:  {baseline_score:.1f}%")
print(f"Optimized: {optimized_score:.1f}%")
print(f"Improvement: {optimized_score - baseline_score:+.1f}%")


# --- Save ---

optimized_qa.save("optimized_qa.json")

Key points:

• The optimizer tries 10 different random subsets of demos (num_candidate_programs=10) and picks the best-performing one
• The metric uses containment (gold in pred) rather than strict equality to handle minor formatting differences -- a common practical choice for QA tasks
• Training and dev sets are split so the optimizer doesn't overfit to the evaluation data
• Start with num_candidate_programs=10 for a quick run; increase to 16-25 for more thorough search

Example 2: Multi-step pipeline optimization

Optimize a two-stage pipeline (extract key facts, then generate an answer) where each stage has its own predictor that gets its own set of demos. BootstrapRS finds demos for all predictors in the pipeline simultaneously.

import dspy
from dspy.evaluate import Evaluate


# --- Define a multi-step pipeline ---

class FactThenAnswer(dspy.Module):
    """Two-step QA: first extract relevant facts, then answer based on facts."""

    def __init__(self):
        self.extract_facts = dspy.ChainOfThought(
            "context, question -> key_facts: str"
        )
        self.answer = dspy.ChainOfThought(
            "key_facts, question -> answer: str"
        )

    def forward(self, context, question):
        facts = self.extract_facts(context=context, question=question)
        return self.answer(key_facts=facts.key_facts, question=question)


# --- Setup ---

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)


# --- Data ---

trainset = [
    dspy.Example(
        context="The Eiffel Tower was built in 1889 for the World's Fair. It stands 330 meters tall and is located in Paris, France. Gustave Eiffel's company designed and built it.",
        question="How tall is the Eiffel Tower?",
        answer="330 meters",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="Python was created by Guido van Rossum and first released in 1991. It emphasizes code readability and supports multiple programming paradigms.",
        question="Who created Python?",
        answer="Guido van Rossum",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="The human genome contains approximately 3 billion base pairs of DNA. The Human Genome Project was completed in 2003 after 13 years of work.",
        question="How many base pairs are in the human genome?",
        answer="approximately 3 billion",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="Tesla was founded in 2003 by Martin Eberhard and Marc Tarpenning. Elon Musk joined as chairman in 2004 and became CEO in 2008.",
        question="When was Tesla founded?",
        answer="2003",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="The speed of light in a vacuum is 299,792,458 meters per second. Einstein's theory of special relativity states nothing can travel faster than light.",
        question="What is the speed of light?",
        answer="299,792,458 meters per second",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="Water boils at 100 degrees Celsius at standard atmospheric pressure. At higher altitudes, the boiling point decreases due to lower air pressure.",
        question="At what temperature does water boil at standard pressure?",
        answer="100 degrees Celsius",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="The Amazon River is about 6,400 km long, making it the second longest river in the world after the Nile. It flows through South America.",
        question="How long is the Amazon River?",
        answer="about 6,400 km",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="Mount Everest is 8,849 meters above sea level, making it the highest peak on Earth. It is located in the Himalayas on the border of Nepal and Tibet.",
        question="How high is Mount Everest?",
        answer="8,849 meters",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="The Great Wall of China stretches over 21,000 km. Construction began in the 7th century BC, and the most well-known sections were built during the Ming Dynasty.",
        question="How long is the Great Wall of China?",
        answer="over 21,000 km",
    ).with_inputs("context", "question"),
    dspy.Example(
        context="Jupiter is the largest planet in our solar system with a diameter of 139,820 km. It has at least 95 known moons, including the four large Galilean moons.",
        question="How many known moons does Jupiter have?",
        answer="at least 95",
    ).with_inputs("context", "question"),
]

devset = trainset[7:]   # last 3 for evaluation
trainset = trainset[:7] # first 7 for training


# --- Metric ---

def answer_match(example, prediction, trace=None):
    """Check if the gold answer is contained in the prediction."""
    pred = prediction.answer.strip().lower()
    gold = example.answer.strip().lower()
    return gold in pred


# --- Baseline ---

pipeline = FactThenAnswer()

evaluator = Evaluate(
    devset=devset,
    metric=answer_match,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

baseline_score = evaluator(pipeline)
print(f"Baseline: {baseline_score:.1f}%")


# --- Optimize both stages with BootstrapRS ---

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=answer_match,
    max_bootstrapped_demos=2,     # Fewer demos per stage to keep prompts manageable
    max_labeled_demos=2,
    num_candidate_programs=16,    # Search over 16 candidate demo combinations
    num_threads=4,
)

optimized_pipeline = optimizer.compile(pipeline, trainset=trainset)


# --- Evaluate ---

optimized_score = evaluator(optimized_pipeline)
print(f"Baseline:  {baseline_score:.1f}%")
print(f"Optimized: {optimized_score:.1f}%")
print(f"Improvement: {optimized_score - baseline_score:+.1f}%")


# --- Inspect what the optimizer chose ---

# Each predictor in the pipeline gets its own demos
print("\n--- Extract Facts demos ---")
for demo in optimized_pipeline.extract_facts.demos:
    print(f"  Q: {demo.get('question', 'N/A')[:60]}...")

print("\n--- Answer demos ---")
for demo in optimized_pipeline.answer.demos:
    print(f"  Q: {demo.get('question', 'N/A')[:60]}...")


# --- Save ---

optimized_pipeline.save("optimized_pipeline.json")

Key points:

• The pipeline has two predictors (extract_facts and answer), and the optimizer finds demos for both simultaneously -- each predictor gets its own demo set
• max_bootstrapped_demos=2 and max_labeled_demos=2 are lower than the single-module example because each stage adds demos to the prompt, and a two-stage pipeline needs to fit within context limits
• Bootstrapped demos are especially valuable here because the extract_facts step produces intermediate key_facts that only exist in successful traces -- labeled data alone wouldn't include these
• The optimizer evaluates end-to-end: a candidate is scored by how well the full pipeline's final answer matches, not by how well individual stages perform
• With num_candidate_programs=16, the optimizer tries 16 different combinations of demos across both stages and picks the combination that yields the best end-to-end score

Condensed from dspy.ai/api/optimizers/BootstrapFewShotWithRandomSearch/. Verify against upstream for latest.

dspy.BootstrapFewShotWithRandomSearch — API Reference

Constructor

dspy.BootstrapFewShotWithRandomSearch(
    metric,
    teacher_settings=None,
    max_bootstrapped_demos=4,
    max_labeled_demos=16,
    max_rounds=1,
    num_candidate_programs=16,
    num_threads=None,
    max_errors=None,
    stop_at_score=None,
    metric_threshold=None,
)

Methods

compile()

optimizer.compile(
    student,
    *,
    teacher=None,
    trainset,
    valset=None,
    restrict=None,
    labeled_sample=True,
)

Optimizes the student program by running BootstrapFewShot multiple times with different random seeds and returning the best candidate.

Returns: Optimized dspy.Module with a candidate_programs attribute containing all scored candidates.

get_params()

optimizer.get_params() -> dict[str, Any]

Returns all configuration parameters as a dictionary.

Candidate initialization strategies

The optimizer evaluates candidates across these strategies:

Result attributes

The returned optimized program has:

Relationship to BootstrapFewShot

BootstrapFewShotWithRandomSearch wraps BootstrapFewShot. It shares the same bootstrap logic but runs it num_candidate_programs times and picks the best result. Key differences: