Dspy Ensemble

Combine Programs with dspy.Ensemble

Guide the user through using DSPy's Ensemble optimizer to combine multiple optimized programs into a single ensemble that aggregates their outputs. This is useful when you have run several optimization passes (different optimizers, different hyperparameters, different random seeds) and want to combine them for more robust predictions.

What is Ensemble

dspy.Ensemble is an optimizer (teleprompter) that takes a list of DSPy programs and returns a single EnsembledProgram. When you call the ensembled program, it runs each constituent program on the same inputs and aggregates the results using a reduce function you provide.

Program A ──┐
Program B ──┼──> Run all ──> reduce_fn ──> Single output
Program C ──┘

Unlike other optimizers that tune prompts or weights, Ensemble does not change the programs themselves. It combines their outputs at inference time.

When to use Ensemble

• You ran multiple optimization passes (e.g., several BootstrapFewShot runs with different seeds) and want to combine the best of each
• You want majority voting -- run several programs and pick the most common answer for higher reliability
• You want to average numeric outputs -- combine scores or probabilities from multiple models
• Different optimizers produced different strengths -- one program is good at precision, another at recall, and you want both
• You need a quick reliability boost -- ensembling is a well-known technique to reduce variance

Do not use Ensemble when:

• You only have one program (nothing to ensemble)
• Latency is critical -- ensembling runs every program, multiplying your inference time
• Cost is a hard constraint -- you pay for every program in the ensemble
• Your programs produce complex structured outputs that are hard to aggregate

Basic usage

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# 1. Define your base program
qa = dspy.ChainOfThought("question -> answer")

# 2. Create a training set and metric
trainset = [
    dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="What is 2 + 2?", answer="4").with_inputs("question"),
    # ... more examples
]

def exact_match(example, pred, trace=None):
    return pred.answer.strip().lower() == example.answer.strip().lower()

# 3. Run multiple optimization passes to get different programs
programs = []
for i in range(3):
    optimizer = dspy.BootstrapFewShot(
        metric=exact_match,
        max_bootstrapped_demos=4,
        max_labeled_demos=4,
    )
    optimized = optimizer.compile(qa, trainset=trainset)
    programs.append(optimized)

# 4. Combine with Ensemble using majority voting
ensemble_optimizer = dspy.Ensemble(reduce_fn=dspy.majority, size=None)
ensemble_program = ensemble_optimizer.compile(programs)

# 5. Use the ensemble like any module
result = ensemble_program(question="What is the capital of Germany?")
print(result.answer)

Constructor parameters

dspy.Ensemble(
    reduce_fn=None,     # Function to aggregate outputs from all programs
    size=None,          # How many programs to sample (None = use all)
    deterministic=False,  # Must be False (deterministic mode not yet implemented)
)

compile method

ensemble_optimizer.compile(programs)

Returns an EnsembledProgram that runs the selected programs and applies reduce_fn.

Reduce functions

The reduce function determines how outputs from multiple programs are combined into a single result.

dspy.majority (built-in)

The most common reduce function. It picks the most frequent output value across all programs -- majority voting.

ensemble = dspy.Ensemble(reduce_fn=dspy.majority)

Use dspy.majority when:

• Outputs are categorical (classification labels, short factual answers, yes/no)
• You want the most robust answer -- the one most programs agree on

Custom reduce: averaging numeric outputs

def average_scores(predictions):
    """Average a numeric output field across all predictions."""
    scores = [float(p.score) for p in predictions]
    avg = sum(scores) / len(scores)
    # Return a Prediction-like object with the averaged score
    return predictions[0].__class__(score=str(avg))

ensemble = dspy.Ensemble(reduce_fn=average_scores)

Custom reduce: weighted voting

def weighted_vote(predictions):
    """Pick the answer backed by the most programs, with confidence weighting."""
    from collections import Counter
    votes = Counter(p.answer for p in predictions)
    winner = votes.most_common(1)[0][0]
    # Return a prediction with the winning answer
    return predictions[0].__class__(answer=winner)

ensemble = dspy.Ensemble(reduce_fn=weighted_vote)

No reduce function

If you pass reduce_fn=None, the ensembled program returns the raw list of predictions from all programs. This is useful when you want to implement custom post-processing logic outside the ensemble.

ensemble = dspy.Ensemble(reduce_fn=None)
ensemble_program = ensemble.compile(programs)

# Returns a list of predictions
all_predictions = ensemble_program(question="What is DSPy?")
# Process them yourself
for pred in all_predictions:
    print(pred.answer)

Combining different optimizers

One of the most powerful uses of Ensemble is combining programs from different optimization strategies. Each optimizer may find different strengths.

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

qa = dspy.ChainOfThought("question -> answer")

# Program 1: Optimized with BootstrapFewShot
opt1 = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
prog1 = opt1.compile(qa, trainset=trainset)

# Program 2: Optimized with MIPROv2
opt2 = dspy.MIPROv2(metric=metric, auto="light")
prog2 = opt2.compile(qa, trainset=trainset)

# Program 3: Optimized with BootstrapFewShotWithRandomSearch
opt3 = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    num_candidate_programs=5,
)
prog3 = opt3.compile(qa, trainset=trainset)

# Ensemble all three
ensemble = dspy.Ensemble(reduce_fn=dspy.majority)
combined = ensemble.compile([prog1, prog2, prog3])

result = combined(question="What is the tallest mountain?")
print(result.answer)

This approach works because different optimizers explore different parts of the prompt space. BootstrapFewShot finds good demonstrations, MIPROv2 finds good instructions, and combining them via voting smooths out individual weaknesses.

Sampling with size

When you have many optimized programs (e.g., from a large random search), you can use size to randomly sample a subset at inference time. This reduces cost while still benefiting from diversity.

# You have 10 programs from BootstrapFewShotWithRandomSearch
programs = [...]  # 10 optimized programs

# Only run 3 of them per inference call (randomly sampled)
ensemble = dspy.Ensemble(reduce_fn=dspy.majority, size=3)
ensemble_program = ensemble.compile(programs)

Each call to ensemble_program randomly picks 3 of the 10 programs, runs them, and applies majority voting. This balances diversity against cost.

Cost and latency considerations

Ensemble multiplies your inference cost and latency by the number of programs (or size if set):

Ways to manage this:

• Use `size` to cap the number of programs run per inference call
• Use cheaper models for the ensemble members and reserve expensive models for critical paths
• Ensemble at evaluation time only to pick the single best program, then deploy that one program in production
• Parallelize if your infrastructure supports concurrent LM calls -- the programs are independent

Ensemble vs BestOfN

Both combine multiple outputs, but they work differently:

You can even stack them: ensemble multiple optimized programs, then wrap the ensemble with BestOfN for additional quality.

Gotchas

• Claude passes a single program instead of a list to `compile()`. Ensemble.compile() expects a list[dspy.Module], not a single module. Always wrap even two programs in a list: ensemble.compile([prog1, prog2]).
• Claude forgets that each ensemble member uses its own LM context. Programs optimized under different dspy.configure(lm=...) calls retain their LM binding. You do not need to re-configure the LM before calling the ensemble -- each program already knows which LM to use.
• Claude sets `deterministic=True` expecting reproducible sampling. The deterministic parameter is reserved but not yet implemented -- setting it to True raises an error. Leave it at the default False.
• Claude uses Ensemble when BestOfN is the right tool. Ensemble combines different optimized programs. If you have one program and want to run it multiple times with temperature sampling and pick the best output, use dspy.BestOfN instead.
• Claude builds a custom reduce function that returns a raw string instead of a Prediction. The reduce_fn receives a list of dspy.Prediction objects and must return a dspy.Prediction (or compatible object). Returning a plain string breaks downstream field access.

Additional resources

• Ensemble API docs
• reference.md -- constructor parameters, compile method, reduce function protocol
• examples.md -- worked examples with majority voting and multi-model ensembles

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• BestOfN for picking the best from multiple runs of a single program -- see /dspy-best-of-n
• BootstrapFewShot for generating the programs to ensemble -- see /ai-improving-accuracy
• MIPROv2 for instruction optimization -- see /ai-improving-accuracy
• Evaluate for measuring ensemble quality with metrics -- see /dspy-evaluate
• For worked examples, see examples.md
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

dspy-ensemble -- Worked Examples

Example 1: Majority voting ensemble from multiple optimization runs

Run BootstrapFewShot three times with different random seeds, then combine the optimized programs with majority voting. This is the most common Ensemble pattern -- it smooths out the randomness in optimization and gives more reliable answers.

import dspy
from dspy.evaluate import Evaluate


# --- Signature ---

class FactualQA(dspy.Signature):
    """Answer the question with a short factual response."""
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="A short factual answer")


# --- Metric ---

def exact_match(example, pred, trace=None):
    return pred.answer.strip().lower() == example.answer.strip().lower()


# --- Dataset ---

trainset = [
    dspy.Example(question="What is the chemical symbol for gold?", answer="Au"),
    dspy.Example(question="How many continents are there?", answer="7"),
    dspy.Example(question="What planet is closest to the Sun?", answer="Mercury"),
    dspy.Example(question="What is the boiling point of water in Celsius?", answer="100"),
    dspy.Example(question="Who wrote Romeo and Juliet?", answer="Shakespeare"),
    dspy.Example(question="What is the square root of 144?", answer="12"),
    dspy.Example(question="What gas do plants absorb from the atmosphere?", answer="Carbon dioxide"),
    dspy.Example(question="What is the largest ocean on Earth?", answer="Pacific"),
    dspy.Example(question="How many sides does a hexagon have?", answer="6"),
    dspy.Example(question="What is the freezing point of water in Fahrenheit?", answer="32"),
]
trainset = [ex.with_inputs("question") for ex in trainset]

devset = [
    dspy.Example(question="What is the capital of Japan?", answer="Tokyo"),
    dspy.Example(question="How many legs does a spider have?", answer="8"),
    dspy.Example(question="What element does O represent?", answer="Oxygen"),
    dspy.Example(question="What is the smallest prime number?", answer="2"),
    dspy.Example(question="What continent is Brazil on?", answer="South America"),
]
devset = [ex.with_inputs("question") for ex in devset]


# --- Setup ---

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)


# --- Step 1: Run multiple optimization passes ---

base_program = dspy.ChainOfThought(FactualQA)

optimized_programs = []
for run_idx in range(3):
    optimizer = dspy.BootstrapFewShot(
        metric=exact_match,
        max_bootstrapped_demos=4,
        max_labeled_demos=4,
    )
    optimized = optimizer.compile(base_program, trainset=trainset)
    optimized_programs.append(optimized)
    print(f"Run {run_idx + 1} compiled.")


# --- Step 2: Combine with Ensemble ---

ensemble_optimizer = dspy.Ensemble(reduce_fn=dspy.majority)
ensemble_program = ensemble_optimizer.compile(optimized_programs)


# --- Step 3: Evaluate the ensemble vs individual programs ---

evaluator = Evaluate(
    devset=devset,
    metric=exact_match,
    num_threads=4,
    display_progress=True,
)

# Score each individual program
for i, prog in enumerate(optimized_programs):
    score = evaluator(prog)
    print(f"Program {i + 1} accuracy: {score:.1f}%")

# Score the ensemble
ensemble_score = evaluator(ensemble_program)
print(f"Ensemble accuracy: {ensemble_score:.1f}%")


# --- Step 4: Use the ensemble in production ---

result = ensemble_program(question="What is the capital of Germany?")
print(f"Answer: {result.answer}")

Key points:

• Each BootstrapFewShot run picks different demonstrations due to randomness, so the three programs have different strengths
• dspy.majority counts votes across all three programs and returns the most common answer
• The ensemble typically matches or beats the best individual program because voting corrects occasional errors from any single program
• Evaluate the ensemble on a devset to confirm the improvement is real before deploying

Example 2: Ensemble with different model configurations

Combine programs optimized with different LMs or optimization strategies. Each model brings different capabilities -- a cheaper model may be fast and often correct, while a more capable model catches harder cases. Ensembling them via majority voting gives you the best of both.

import dspy
from dspy.evaluate import Evaluate


# --- Signature ---

class ClassifyIntent(dspy.Signature):
    """Classify the user message into one of the given intent categories."""
    message: str = dspy.InputField(desc="User message to classify")
    categories: str = dspy.InputField(desc="Comma-separated list of valid categories")
    intent: str = dspy.OutputField(desc="The best matching category")


# --- Metric ---

def correct_intent(example, pred, trace=None):
    return pred.intent.strip().lower() == example.intent.strip().lower()


# --- Dataset ---

categories = "billing, technical_support, account, general_inquiry, cancellation"

trainset = [
    dspy.Example(message="I was charged twice this month", categories=categories, intent="billing"),
    dspy.Example(message="My app keeps crashing on startup", categories=categories, intent="technical_support"),
    dspy.Example(message="How do I change my password?", categories=categories, intent="account"),
    dspy.Example(message="What are your business hours?", categories=categories, intent="general_inquiry"),
    dspy.Example(message="I want to cancel my subscription", categories=categories, intent="cancellation"),
    dspy.Example(message="The invoice amount is wrong", categories=categories, intent="billing"),
    dspy.Example(message="I can't connect to the API", categories=categories, intent="technical_support"),
    dspy.Example(message="Update my email address please", categories=categories, intent="account"),
    dspy.Example(message="Do you offer student discounts?", categories=categories, intent="general_inquiry"),
    dspy.Example(message="Please stop my auto-renewal", categories=categories, intent="cancellation"),
]
trainset = [ex.with_inputs("message", "categories") for ex in trainset]

devset = [
    dspy.Example(message="Why is my bill higher this month?", categories=categories, intent="billing"),
    dspy.Example(message="The website shows a 500 error", categories=categories, intent="technical_support"),
    dspy.Example(message="I need to update my shipping address", categories=categories, intent="account"),
    dspy.Example(message="What payment methods do you accept?", categories=categories, intent="general_inquiry"),
    dspy.Example(message="I'd like to close my account", categories=categories, intent="cancellation"),
]
devset = [ex.with_inputs("message", "categories") for ex in devset]


# --- Setup: two different LMs ---

fast_lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
strong_lm = dspy.LM("openai/gpt-4o")  # or "anthropic/claude-sonnet-4-5-20250929", etc.


# --- Step 1: Optimize a program with the fast LM ---

dspy.configure(lm=fast_lm)
base_program = dspy.ChainOfThought(ClassifyIntent)

opt_fast = dspy.BootstrapFewShot(
    metric=correct_intent,
    max_bootstrapped_demos=4,
)
prog_fast = opt_fast.compile(base_program, trainset=trainset)
print("Fast-model program compiled.")


# --- Step 2: Optimize a program with the strong LM ---

dspy.configure(lm=strong_lm)
base_program_strong = dspy.ChainOfThought(ClassifyIntent)

opt_strong = dspy.MIPROv2(
    metric=correct_intent,
    auto="light",
)
prog_strong = opt_strong.compile(base_program_strong, trainset=trainset)
print("Strong-model program compiled.")


# --- Step 3: Optimize another variant with random search ---

dspy.configure(lm=fast_lm)
base_program_rs = dspy.ChainOfThought(ClassifyIntent)

opt_rs = dspy.BootstrapFewShotWithRandomSearch(
    metric=correct_intent,
    max_bootstrapped_demos=4,
    num_candidate_programs=5,
)
prog_rs = opt_rs.compile(base_program_rs, trainset=trainset)
print("Random-search program compiled.")


# --- Step 4: Ensemble all three ---

ensemble_optimizer = dspy.Ensemble(reduce_fn=dspy.majority, size=None)
ensemble_program = ensemble_optimizer.compile([prog_fast, prog_strong, prog_rs])


# --- Step 5: Evaluate ---

dspy.configure(lm=fast_lm)  # LM for evaluation context

evaluator = Evaluate(
    devset=devset,
    metric=correct_intent,
    num_threads=4,
    display_progress=True,
)

score_fast = evaluator(prog_fast)
print(f"Fast-model program accuracy: {score_fast:.1f}%")

score_strong = evaluator(prog_strong)
print(f"Strong-model program accuracy: {score_strong:.1f}%")

score_rs = evaluator(prog_rs)
print(f"Random-search program accuracy: {score_rs:.1f}%")

ensemble_score = evaluator(ensemble_program)
print(f"Ensemble accuracy: {ensemble_score:.1f}%")


# --- Step 6: Use in production ---

result = ensemble_program(
    message="I see an unauthorized charge on my credit card",
    categories=categories,
)
print(f"Intent: {result.intent}")

Key points:

• Each program uses a different optimization strategy and potentially a different LM, creating genuine diversity in how they approach the task
• The fast-model program (gpt-4o-mini + BootstrapFewShot) is cheap and handles easy cases well
• The strong-model program (gpt-4o + MIPROv2) handles edge cases and ambiguous inputs better
• The random-search program explores a wider space of few-shot demonstrations
• Majority voting across all three is more reliable than any single program because errors from different strategies are unlikely to be correlated
• In production, you pay for all three LM calls per input -- use size=2 if you want to reduce cost by sampling a subset each time

Condensed from dspy.ai/api/optimizers/Ensemble/. Verify against upstream for latest.

dspy.Ensemble -- API Reference

Constructor

dspy.Ensemble(
    reduce_fn=None,       # Callable | None
    size=None,            # int | None
    deterministic=False,  # bool (must be False -- not yet implemented)
)

compile()

ensemble_optimizer.compile(programs)

Returns: EnsembledProgram -- a callable module that runs the selected programs and applies reduce_fn.

EnsembledProgram

The object returned by compile(). When called:

1. If size is set, randomly samples size programs from the list 2. Runs each selected program on the same inputs 3. Applies reduce_fn to the list of predictions 4. Returns the reduced result (or the raw list if reduce_fn=None)

# Use like any DSPy module
result = ensemble_program(question="What is DSPy?")
print(result.answer)

Built-in reduce functions

dspy.majority

Picks the most frequent output value across all predictions (majority voting).

ensemble = dspy.Ensemble(reduce_fn=dspy.majority)

Best for categorical outputs: classification labels, short factual answers, yes/no.

Custom reduce function protocol

A reduce function receives a list of dspy.Prediction objects and must return a single dspy.Prediction (or compatible object with the same output fields).

def my_reduce(predictions: list[dspy.Prediction]) -> dspy.Prediction:
    # Aggregate predictions
    # Return a Prediction-like object
    ...

Example: averaging numeric outputs

def average_scores(predictions):
    scores = [float(p.score) for p in predictions]
    avg = sum(scores) / len(scores)
    return predictions[0].__class__(score=str(avg))

Example: weighted voting

from collections import Counter

def weighted_vote(predictions):
    votes = Counter(p.answer for p in predictions)
    winner = votes.most_common(1)[0][0]
    return predictions[0].__class__(answer=winner)

Key behaviors

• Ensemble does not modify the constituent programs -- it only combines their outputs at inference time
• Each program retains its own LM context from when it was optimized
• Programs are run sequentially (not parallelized by default)
• With size=None, all programs run on every call; with size=N, a random subset of N is sampled each time