Dspy Best Of N

Pick the Best Output with dspy.BestOfN

Guide the user through using DSPy's BestOfN module to run a program multiple times and keep the highest-scoring result. This is rejection sampling -- generate N candidates, score each one, return the winner.

What is BestOfN

dspy.BestOfN wraps any DSPy module and calls it up to N times with temperature=1.0 (each attempt uses a different rollout ID to get diverse outputs). A reward function scores every result, and BestOfN returns the single best prediction.

If any attempt hits a score threshold you set, execution stops early -- no need to burn through all N attempts when you already have a great result.

Your module ──> Run N times ──> Score each with reward_fn ──> Return best

When to use BestOfN

• You have a cheap, fast metric that can score outputs automatically (test suite passes, regex match, word count check, etc.)
• Quality variance is high -- the same prompt sometimes produces great output and sometimes doesn't
• You'd rather spend tokens than engineering time -- BestOfN is the simplest way to boost quality without optimization
• You need a quick quality boost before investing in full prompt optimization with MIPROv2 or BootstrapFewShot

Do not use BestOfN when:

• You have no way to automatically score outputs (you need a metric)
• Latency matters more than quality (N calls take N times longer, unless you can parallelize)
• Cost is a hard constraint and N is large

Basic usage

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# 1. Define your module
qa = dspy.ChainOfThought("question -> answer")

# 2. Define a reward function
def short_answer(args, pred):
    """Prefer concise single-word answers."""
    return 1.0 if len(pred.answer.split()) == 1 else 0.0

# 3. Wrap with BestOfN
best_qa = dspy.BestOfN(
    module=qa,
    N=3,
    reward_fn=short_answer,
    threshold=1.0,
)

# 4. Call it like any module
result = best_qa(question="What is the capital of Belgium?")
print(result.answer)

Constructor parameters

dspy.BestOfN(
    module,       # Any dspy.Module to run repeatedly
    N,            # Number of attempts (int)
    reward_fn,    # Scoring function: (args_dict, prediction) -> float
    threshold,    # Early-stop threshold: stop as soon as a score >= threshold
    fail_count=None,  # Max failures before raising an error (defaults to N)
)

The reward function

The reward function is the core of BestOfN. It receives two arguments:

def reward_fn(args: dict, prediction: dspy.Prediction) -> float:
    # args: the keyword arguments you passed to the BestOfN call
    # prediction: the output from one attempt of the wrapped module
    # Return: a scalar score (higher = better)
    ...

Key differences from a dspy.Evaluate metric:

• Signature: (args_dict, prediction) not (example, prediction, trace)
• No gold labels: args contains only the inputs you passed, not expected outputs
• No trace parameter: BestOfN doesn't use traces

Reward function examples

Binary pass/fail:

def passes_tests(args, pred):
    """Score 1.0 if generated code passes all tests, 0.0 otherwise."""
    try:
        exec(pred.code)
        return 1.0
    except Exception:
        return 0.0

Graded score:

def quality_score(args, pred):
    """Score summaries on length and keyword coverage."""
    score = 0.0
    # Prefer summaries under 100 words
    if len(pred.summary.split()) <= 100:
        score += 0.5
    # Reward covering key topics
    keywords = ["revenue", "growth", "forecast"]
    covered = sum(1 for kw in keywords if kw in pred.summary.lower())
    score += 0.5 * (covered / len(keywords))
    return score

Using an LM as judge inside the reward:

class JudgeQuality(dspy.Signature):
    """Rate the answer quality from 0.0 to 1.0."""
    question: str = dspy.InputField()
    answer: str = dspy.InputField()
    score: float = dspy.OutputField(desc="Quality score from 0.0 to 1.0")

judge = dspy.Predict(JudgeQuality)

def llm_reward(args, pred):
    result = judge(question=args["question"], answer=pred.answer)
    return result.score

Note: Using an LM as judge inside the reward function costs additional tokens per attempt. Reserve this for cases where programmatic scoring isn't feasible.

Tuning N

Rule of thumb: if your base module succeeds ~50% of the time, N=3 gives you a ~87.5% chance of at least one success. If it succeeds ~20% of the time, you need N=8 for ~83%.

The math: probability of at least one success in N tries = 1 - (1 - p)^N where p is the single-attempt success rate.

How selection works internally

1. BestOfN calls your module with temperature=1.0 and a unique rollout ID for each attempt 2. Each attempt produces a dspy.Prediction 3. The reward function scores the prediction 4. If the score >= threshold, return immediately (early stopping) 5. If the attempt raises an exception, increment the failure counter 6. After all N attempts (or early stopping), return the prediction with the highest score 7. If failures exceed fail_count, raise an exception

The unique rollout IDs ensure the LM produces diverse outputs even with the same input. Temperature is fixed at 1.0 to maximize diversity.

Cost considerations

BestOfN multiplies your token usage by up to N times (fewer if early stopping kicks in). Budget accordingly:

Ways to manage cost:

• Set a tight threshold so good results stop early (often after 1-2 attempts)
• Use a cheap model as the base module and a stronger model only for the reward function
• Start with N=3 and increase only if your metric shows it helps
• Use programmatic reward functions (regex, test execution, length checks) instead of LM-based judges to avoid extra LM calls per attempt

BestOfN vs MultiChainComparison

Both BestOfN and dspy.MultiChainComparison aim to pick the best output from multiple candidates, but they work differently:

Use BestOfN when you can write a reward function. Use MultiChainComparison when you want the LM to judge quality using its own understanding.

Combining BestOfN with optimization

BestOfN works well as a complement to DSPy optimizers. Optimize your module first, then wrap the optimized version with BestOfN for an additional quality boost:

# Optimize the base module
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# Wrap the optimized module with BestOfN
best_qa = dspy.BestOfN(
    module=optimized_qa,
    N=3,
    reward_fn=my_reward,
    threshold=1.0,
)

This stacks two quality improvements: better prompts from the optimizer, and rejection sampling from BestOfN.

Gotchas

• Claude writes the reward function with `(example, prediction, trace=None)` signature. BestOfN reward functions take (args_dict, prediction), not the (example, prediction, trace) signature used by dspy.Evaluate metrics. The args dict contains only the inputs you passed to the call, not labeled examples with gold outputs.
• Claude sets N too high without considering cost. Each attempt is a full LM call at temperature=1.0. N=10 means 10x the token cost. Start with N=3 and increase only if your metric shows improvement — diminishing returns kick in quickly above N=5.
• Claude uses BestOfN when the reward function is as expensive as the module itself. If your reward function calls an LM (e.g., LM-as-judge), each BestOfN attempt costs 2x tokens (one for the module, one for the judge). For N=5, that is 10 LM calls total. Use programmatic reward functions (test execution, regex, length checks) whenever possible.
• Claude forgets to set `threshold` to enable early stopping. Without a meaningful threshold, BestOfN always runs all N attempts even when the first one is perfect. Set threshold to a value that represents "good enough" (e.g., 1.0 for binary pass/fail, 0.9 for graded metrics) to save tokens on easy inputs.
• Claude wraps an already-optimized module but does not evaluate the incremental gain. BestOfN on top of an optimized module costs N times more per call at inference time. Always measure the quality gain from BestOfN separately to confirm the extra cost is justified — if the optimized module already hits 95%+, BestOfN may not add enough to be worth it.

Additional resources

• dspy.BestOfN API docs
• reference.md — constructor parameters, forward() method, key behaviors
• examples.md — code generation with test-based selection, summarization with graded metric

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• MultiChainComparison for LM-based candidate selection -- see /dspy-multi-chain-comparison
• Evaluate for measuring quality with metrics and devsets -- see /dspy-evaluate
• Improving accuracy for the full optimization workflow -- see /ai-improving-accuracy
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

dspy-best-of-n -- Worked Examples

Example 1: Best-of-3 code generation with test-based selection

Generate a Python function multiple times and pick the version that passes a test suite. This is one of the strongest use cases for BestOfN -- automated tests give you a perfect binary reward signal.

import dspy


class GenerateFunction(dspy.Signature):
    """Write a Python function that solves the given task."""
    task_description: str = dspy.InputField(desc="What the function should do")
    function_name: str = dspy.InputField(desc="Name of the function to implement")
    code: str = dspy.OutputField(desc="Complete Python function implementation")


# --- Reward function: run tests against generated code ---

def passes_tests(args, pred):
    """Return 1.0 if the generated code passes all test cases, 0.0 otherwise."""
    test_cases = args.get("test_cases", [])
    if not test_cases:
        return 0.0

    # Execute the generated code in a sandboxed namespace
    namespace = {}
    try:
        exec(pred.code, namespace)
    except Exception:
        return 0.0

    # Run each test case
    fn = namespace.get(args["function_name"])
    if fn is None:
        return 0.0

    passed = 0
    for test_input, expected_output in test_cases:
        try:
            result = fn(*test_input) if isinstance(test_input, tuple) else fn(test_input)
            if result == expected_output:
                passed += 1
        except Exception:
            pass

    return passed / len(test_cases)


# --- Setup ---

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

generator = dspy.ChainOfThought(GenerateFunction)

best_generator = dspy.BestOfN(
    module=generator,
    N=3,
    reward_fn=passes_tests,
    threshold=1.0,  # Stop as soon as all tests pass
)

# --- Usage ---

result = best_generator(
    task_description="Write a function that takes a list of integers and returns a new list with duplicates removed, preserving the original order.",
    function_name="remove_duplicates",
    test_cases=[
        ([1, 2, 3, 2, 1], [1, 2, 3]),
        ([1, 1, 1], [1]),
        ([], []),
        ([5, 3, 5, 3, 5], [5, 3]),
    ],
)

print(result.code)

Key points:

• The reward function executes the generated code and runs test cases against it -- a fully automated, deterministic check
• threshold=1.0 means BestOfN stops as soon as it gets code that passes all tests, potentially saving 1-2 LM calls
• The test_cases are passed through args so the reward function can access them without hardcoding
• N=3 is enough here because code generation either works or doesn't -- you don't need many samples when the pass rate per attempt is reasonable

Example 2: Best-of-5 summarization with quality metric

Generate multiple summaries and pick the one that best covers key information while staying concise. This demonstrates a graded (non-binary) reward function.

import dspy


class Summarize(dspy.Signature):
    """Produce a concise summary of the given text."""
    text: str = dspy.InputField(desc="The text to summarize")
    audience: str = dspy.InputField(desc="Who the summary is for")
    summary: str = dspy.OutputField(desc="A concise summary of the text")


# --- Reward function: multi-criteria quality score ---

def summary_quality(args, pred):
    """Score a summary on length, keyword coverage, and structure."""
    summary = pred.summary
    text = args["text"]
    words = summary.split()
    text_words = text.split()

    score = 0.0

    # 1. Length ratio: summary should be 10-25% of original length
    ratio = len(words) / max(len(text_words), 1)
    if 0.10 <= ratio <= 0.25:
        score += 0.4  # ideal compression
    elif 0.05 <= ratio <= 0.35:
        score += 0.2  # acceptable compression
    # else: too long or too short, no points

    # 2. Keyword coverage: extract frequent meaningful words from the source
    # (simple heuristic -- in production you'd use something smarter)
    stopwords = {"the", "a", "an", "is", "are", "was", "were", "in", "on", "at",
                 "to", "for", "of", "and", "or", "but", "with", "that", "this",
                 "it", "as", "by", "from", "be", "has", "had", "have", "not"}
    source_words = [w.lower().strip(".,!?;:") for w in text_words if len(w) > 3]
    source_words = [w for w in source_words if w not in stopwords]

    # Find the top keywords by frequency
    from collections import Counter
    word_counts = Counter(source_words)
    top_keywords = [w for w, _ in word_counts.most_common(10)]

    if top_keywords:
        summary_lower = summary.lower()
        covered = sum(1 for kw in top_keywords if kw in summary_lower)
        score += 0.4 * (covered / len(top_keywords))

    # 3. Structure: prefer summaries that start with a capital letter and end with
    # punctuation (basic well-formedness)
    if summary and summary[0].isupper() and summary.rstrip().endswith((".", "!", "?")):
        score += 0.2

    return score


# --- Setup ---

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

summarizer = dspy.ChainOfThought(Summarize)

best_summarizer = dspy.BestOfN(
    module=summarizer,
    N=5,
    reward_fn=summary_quality,
    threshold=0.9,  # Early stop if we get an excellent summary
)

# --- Usage ---

article = """
Artificial intelligence is transforming the healthcare industry in several
significant ways. Machine learning algorithms can now analyze medical images
with accuracy that rivals or exceeds human radiologists. Natural language
processing enables automated extraction of clinical information from
unstructured doctor notes and medical records. Predictive models help
hospitals forecast patient admissions, optimize staffing levels, and identify
patients at risk of readmission. Drug discovery has been accelerated by AI
systems that can screen millions of molecular compounds in days rather than
years. Remote patient monitoring powered by AI analyzes data from wearable
devices to detect early signs of deterioration. Despite these advances,
challenges remain around data privacy, algorithmic bias, regulatory approval,
and the need to maintain physician trust and oversight.
"""

result = best_summarizer(text=article, audience="hospital executives")
print(result.summary)

Key points:

• The reward function uses a weighted multi-criteria score (length + keyword coverage + structure) rather than a single binary check
• N=5 gives more candidates for subjective tasks like summarization where quality varies more across attempts
• threshold=0.9 allows early stopping without requiring a perfect score -- useful when the metric has soft criteria
• No gold labels are needed -- the reward function evaluates quality using only the input text and the generated summary
• For production use, you could replace the keyword heuristic with a stronger signal like an LM-based judge (at the cost of extra tokens per attempt)

Condensed from dspy.ai/api/modules/BestOfN/. Verify against upstream for latest.

dspy.BestOfN — API Reference

Constructor

dspy.BestOfN(
    module,          # dspy.Module (required)
    N,               # int (required)
    reward_fn,       # Callable[[dict, Prediction], float] (required)
    threshold,       # float (required)
    fail_count=None, # int | None
)

Inheritance

BestOfN extends dspy.Module.

Methods

forward()

best_of_n.forward(**kwargs) -> Prediction

Executes the wrapped module up to N times with temperature=1.0 and unique rollout IDs for each attempt. Returns the prediction with the highest reward score, or the first prediction that meets the threshold.

Behavior: 1. Deep-copies the module for each rollout 2. Calls the module with temperature=1.0 and a unique rollout ID 3. Scores the result with reward_fn(kwargs, prediction) 4. If score >= threshold, returns immediately (early stopping) 5. If the attempt raises an exception, increments failure counter 6. After all N attempts, returns the highest-scoring prediction 7. If failures exceed fail_count, raises an exception

__call__()

best_of_n(**kwargs) -> Prediction

Entry point with callback support and usage tracking. Delegates to forward().

Module management methods

Key behaviors

• Module instances are deep-copied for each rollout to maintain isolation between attempts
• Temperature is fixed at 1.0 to maximize output diversity
• Unique rollout IDs ensure the LM produces different outputs even with identical inputs
• Execution traces from the best attempt are preserved
• The reward function signature is (args_dict, prediction) -> float, NOT (example, prediction, trace) -> float