Dspy Better Together

BetterTogether: Joint Prompt + Weight Optimization

Guide the user through using dspy.BetterTogether to get the best possible quality by combining prompt optimization and model fine-tuning in alternating rounds. Each round builds on the improvements from the previous one, creating compounding gains that beat either approach alone.

What it is

BetterTogether is a DSPy optimizer that alternates between prompt optimization (instructions, few-shot examples) and weight optimization (fine-tuning). Instead of running these independently, it chains them so each phase builds on the previous one's improvements:

1. Prompt optimization discovers effective task decompositions and reasoning strategies 2. Weight optimization specializes the model to execute those discovered patterns efficiently 3. Repeated rounds compound the gains -- each phase benefits from the prior improvements

Research shows this consistently outperforms either approach alone, with 5-78% gains over individual techniques (arXiv 2407.10930v2). A Databricks case study on IE Bench showed GEPA alone +2.1 points, fine-tuning alone +1.9 points, but combined they achieved +4.8 points over baseline.

When to use

• You have 500+ labeled examples and a reliable metric
• You've already tried prompt optimization (MIPROv2) and fine-tuning (BootstrapFinetune) separately and want more
• You want the absolute best quality and have the compute budget for multiple optimization rounds
• Fine-tuning alone didn't close the gap to your quality target
• You need a production-grade model and can afford longer optimization time

When NOT to use

• You have fewer than 500 examples -- use MIPROv2 or BootstrapFewShot instead (see /ai-improving-accuracy)
• You haven't tried prompt optimization yet -- start with /ai-improving-accuracy
• Your baseline is below 50% -- fix your task definition or data first
• You're still iterating on what the task is -- BetterTogether is expensive to re-run
• You don't have access to a fine-tunable model (OpenAI gpt-4o-mini/gpt-4o, or local models)

Prerequisites

Before starting, confirm:

• [ ] Data: 500+ labeled examples (1000+ recommended), split 80/10/10 (train/dev/test)
• [ ] Baseline: Measured accuracy from prompt optimization (MIPROv2) and/or fine-tuning (BootstrapFinetune)
• [ ] Metric: Automated metric that scores predictions
• [ ] Fine-tunable model: OpenAI fine-tuning API, Databricks, or local models with GPU
• [ ] Budget: Multiple optimization rounds cost 2-3x more than a single optimizer run

Basic usage

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# Define your program
class Classify(dspy.Signature):
    """Classify the support ticket into a category."""
    text: str = dspy.InputField()
    category: str = dspy.OutputField()

program = dspy.ChainOfThought(Classify)

# IMPORTANT: All predictors must have explicit LMs assigned
program.set_lm(lm)

# Define your metric
def metric(example, prediction, trace=None):
    return prediction.category.strip().lower() == example.category.strip().lower()

# Prepare data
trainset = [dspy.Example(text=x["text"], category=x["category"]).with_inputs("text") for x in data]
valset = trainset[800:900]
trainset = trainset[:800]

# Run BetterTogether with defaults
optimizer = dspy.BetterTogether(metric=metric)
compiled = optimizer.compile(program, trainset=trainset, valset=valset)

By default, BetterTogether uses:

• `p`: BootstrapFewShotWithRandomSearch for prompt optimization
• `w`: BootstrapFinetune for weight optimization
• Strategy: "p -> w -> p" (prompts, then weights, then prompts again)

How it combines prompt and weight tuning

BetterTogether executes a strategy string that defines the order of optimization phases:

"p -> w -> p"
 |    |    |
 |    |    +-- Re-optimize prompts for the fine-tuned model
 |    +------- Fine-tune weights using the optimized prompts
 +------------ Optimize prompts first (instructions + few-shot)

At each step:

1. Shuffle the trainset (prevents overfitting to data order) 2. Run the designated optimizer on the current best program 3. Evaluate the result on the validation set 4. Record the candidate program and score 5. Move to the next step in the strategy

After all steps, BetterTogether returns the best-scoring candidate across all phases (ties broken by earlier position).

Why alternating works

• Prompt optimization finds the right "recipe" -- effective instructions, good examples, useful reasoning patterns
• Weight optimization bakes those patterns into the model so it executes them reliably and cheaply
• Re-optimizing prompts after fine-tuning discovers new strategies that the specialized model can now handle

Custom optimizers

Pass your own optimizers as keyword arguments. The keys become identifiers in the strategy string:

from dspy.teleprompt import GEPA, BootstrapFinetune

optimizer = dspy.BetterTogether(
    metric=metric,
    p=GEPA(metric=metric, auto="medium"),
    w=BootstrapFinetune(metric=metric),
)

program.set_lm(lm)
compiled = optimizer.compile(
    program,
    trainset=trainset,
    valset=valset,
    strategy="p -> w -> p",
)

You can use any DSPy Teleprompter as an optimizer. Common choices:

Key parameters

Constructor: BetterTogether(metric, **optimizers)

Compile: optimizer.compile(student, *, trainset, ...)

Return value

The compiled program has two extra attributes:

• candidate_programs: List of dicts with 'program', 'score', 'strategy' keys, sorted by score descending
• flag_compilation_error_occurred: Boolean indicating if any step failed

Strategy patterns

Computational cost

BetterTogether runs multiple optimization rounds, so it costs more than individual optimizers:

Fine-tuning is the expensive part. Each fine-tuning round involves:

• Bootstrapping traces from training data
• Uploading traces to the fine-tuning provider
• Waiting for fine-tuning to complete (minutes to hours depending on provider)
• Evaluating the fine-tuned model

Reducing cost

• Start with "p -> w" to see if two rounds are enough
• Use a smaller valset (but keep at least 50-100 examples)
• Use optimizer_compile_args to limit individual optimizer budgets

BetterTogether vs individual optimizers

Rule of thumb: Try MIPROv2 first. If you're still short of your quality target, try BootstrapFinetune. If you need more, use BetterTogether.

Important requirements

1. Explicit LM assignment: All predictors in your program must have LMs assigned via set_lm(). Global dspy.configure(lm=...) is not enough for BetterTogether.

program = dspy.ChainOfThought(MySignature)
program.set_lm(lm)  # Required

2. Fine-tunable model: The weight optimizer needs a model that supports fine-tuning (OpenAI, Databricks, or local models with GPU).

3. Validation data: Provide either an explicit valset or set valset_ratio > 0. Without validation data, BetterTogether returns the latest program instead of the best one.

4. Strategy keys must match: Keys in the strategy string must match the keyword argument names from the constructor.

Inspecting results

After compilation, examine all candidate programs:

compiled = optimizer.compile(program, trainset=trainset, valset=valset)

# See all candidates ranked by score
for candidate in compiled.candidate_programs:
    print(f"Strategy step: {candidate['strategy']}, Score: {candidate['score']:.1f}%")

# Check if any errors occurred
if compiled.flag_compilation_error_occurred:
    print("Warning: one or more optimization steps failed")

Error handling

BetterTogether has built-in resilience. If any optimization step fails:

• It logs the error and continues to the next step
• Returns the best program found before the failure
• Sets flag_compilation_error_occurred = True on the result

Always check this flag in production workflows.

Gotchas

• Claude forgets `set_lm()` and relies on global `dspy.configure()`. BetterTogether requires every predictor to have an explicit LM assignment via program.set_lm(lm). Without it, the weight optimizer cannot identify which model to fine-tune and raises an error. Always call set_lm() on the program before compile().
• Claude jumps straight to BetterTogether without trying simpler optimizers first. BetterTogether costs 2-3x more than a single optimizer and takes hours. If you have not tried MIPROv2 or BootstrapFinetune individually first, start there — BetterTogether is only worth it when individual optimizers have plateaued.
• Claude omits the validation set. Without a valset (or valset_ratio > 0), BetterTogether returns the latest program instead of the best-scoring one across all phases. Always provide a valset or leave valset_ratio=0.1 so the optimizer can select the best candidate.
• Claude uses the same trainset for both training and validation. If valset overlaps with trainset, the optimizer selects based on inflated scores. Use a held-out split or let BetterTogether auto-split via valset_ratio.
• Claude does not check `flag_compilation_error_occurred` after compile. If a fine-tuning step fails silently (API timeout, quota exceeded), BetterTogether returns the best program found before the failure. Always check compiled.flag_compilation_error_occurred and inspect compiled.candidate_programs to verify which steps completed.

Additional resources

• dspy.BetterTogether API docs
• reference.md — constructor parameters, compile() method, key behaviors
• examples.md — combined optimization workflow, two-phase strategy with custom optimizers

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• For the full fine-tuning workflow, see /ai-fine-tuning
• For prompt optimization alone, see /ai-improving-accuracy
• For evaluation and metrics, see /dspy-evaluate
• For data preparation, see /dspy-data
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

BetterTogether Examples

Worked examples showing how to use dspy.BetterTogether for joint prompt and weight optimization.

Example 1: Combined prompt + weight optimization

Full workflow for a classification task, comparing BetterTogether against prompt-only and fine-tune-only approaches.

Setup and data

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

class ClassifyIntent(dspy.Signature):
    """Classify the user message into an intent category."""
    message: str = dspy.InputField()
    intent: str = dspy.OutputField(
        desc="one of: purchase, refund, support, feedback, account, other"
    )

program = dspy.ChainOfThought(ClassifyIntent)

# IMPORTANT: Assign LM explicitly for BetterTogether
program.set_lm(lm)

# Load labeled data (1000+ examples)
import json
with open("intents.json") as f:
    data = json.load(f)

examples = [
    dspy.Example(message=x["message"], intent=x["intent"]).with_inputs("message")
    for x in data
]

# Split 80/10/10
trainset = examples[:800]
devset = examples[800:900]
testset = examples[900:]

def metric(example, prediction, trace=None):
    return prediction.intent.strip().lower() == example.intent.strip().lower()

evaluator = Evaluate(devset=devset, metric=metric, num_threads=4, display_progress=True)

Step 1: Measure baselines individually

# Baseline (no optimization)
baseline_score = evaluator(program)
print(f"Baseline: {baseline_score:.1f}%")

# Prompt optimization only
prompt_opt = dspy.MIPROv2(metric=metric, auto="medium")
prompt_optimized = prompt_opt.compile(program, trainset=trainset)
prompt_score = evaluator(prompt_optimized)
print(f"Prompt-only (MIPROv2): {prompt_score:.1f}%")

# Fine-tuning only
ft_opt = dspy.BootstrapFinetune(metric=metric, num_threads=24)
finetuned = ft_opt.compile(program, trainset=trainset)
ft_score = evaluator(finetuned)
print(f"Fine-tune-only: {ft_score:.1f}%")

Step 2: Run BetterTogether

optimizer = dspy.BetterTogether(metric=metric)

compiled = optimizer.compile(
    program,
    trainset=trainset,
    valset=devset,
    strategy="p -> w -> p",
)

bt_score = evaluator(compiled)
print(f"BetterTogether: {bt_score:.1f}%")

Step 3: Compare results

test_eval = Evaluate(devset=testset, metric=metric, num_threads=4, display_progress=True)

print("Test set results:")
print(f"  Baseline:         {test_eval(program):.1f}%")
print(f"  Prompt-only:      {test_eval(prompt_optimized):.1f}%")
print(f"  Fine-tune-only:   {test_eval(finetuned):.1f}%")
print(f"  BetterTogether:   {test_eval(compiled):.1f}%")

# Inspect per-step scores
for candidate in compiled.candidate_programs:
    print(f"  Step '{candidate['strategy']}': {candidate['score']:.1f}%")

Expected results

BetterTogether gets +6 pts beyond the best individual approach because the prompt and weight optimization rounds compound on each other.

Save for production

compiled.save("intent_classifier_bt.json")

# Load later
from my_module import build_program
production = build_program()
production.load("intent_classifier_bt.json")
result = production(message="I want my money back")
print(result.intent)  # "refund"

---

Example 2: Two-phase optimization strategy with custom optimizers

Use GEPA for instruction tuning and BootstrapFinetune for weight optimization in a simpler two-phase strategy. This is cheaper than the default three-phase strategy and works well when you want faster iteration.

Setup

import dspy
from dspy.evaluate import Evaluate
from dspy.teleprompt import GEPA, BootstrapFinetune

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

class SummarizeReview(dspy.Signature):
    """Summarize the product review into a short, factual summary."""
    review: str = dspy.InputField()
    summary: str = dspy.OutputField(desc="1-2 sentence factual summary")
    sentiment: str = dspy.OutputField(desc="one of: positive, negative, mixed")

program = dspy.ChainOfThought(SummarizeReview)
program.set_lm(lm)

# Load data
import json
with open("reviews.json") as f:
    data = json.load(f)

examples = [
    dspy.Example(
        review=x["review"],
        summary=x["summary"],
        sentiment=x["sentiment"],
    ).with_inputs("review")
    for x in data
]

trainset = examples[:800]
valset = examples[800:900]
testset = examples[900:]

# Composite metric: sentiment accuracy + summary quality
class AssessSummary(dspy.Signature):
    """Assess if the summary accurately captures the review."""
    review: str = dspy.InputField()
    gold_summary: str = dspy.InputField()
    predicted_summary: str = dspy.InputField()
    is_accurate: bool = dspy.OutputField()

def metric(example, prediction, trace=None):
    # Sentiment must be exact match
    sentiment_correct = prediction.sentiment.strip().lower() == example.sentiment.strip().lower()

    # Summary quality via LM judge
    judge = dspy.Predict(AssessSummary)
    assessment = judge(
        review=example.review,
        gold_summary=example.summary,
        predicted_summary=prediction.summary,
    )
    summary_good = float(assessment.is_accurate)

    return 0.5 * float(sentiment_correct) + 0.5 * summary_good

evaluator = Evaluate(devset=valset, metric=metric, num_threads=4, display_progress=True)

Run two-phase BetterTogether

# Use GEPA for instruction tuning (good with composite metrics)
# and BootstrapFinetune for weight optimization
optimizer = dspy.BetterTogether(
    metric=metric,
    p=GEPA(metric=metric, auto="medium"),
    w=BootstrapFinetune(metric=metric),
)

compiled = optimizer.compile(
    program,
    trainset=trainset,
    valset=valset,
    strategy="p -> w",  # Two phases only -- cheaper and faster
)

score = evaluator(compiled)
print(f"BetterTogether (p -> w): {score:.1f}%")

Pass custom arguments to individual optimizers

Use optimizer_compile_args to control each optimizer's behavior independently:

optimizer = dspy.BetterTogether(
    metric=metric,
    p=GEPA(metric=metric, auto="medium"),
    w=BootstrapFinetune(metric=metric),
)

compiled = optimizer.compile(
    program,
    trainset=trainset,
    valset=valset,
    strategy="p -> w",
    optimizer_compile_args={
        "p": {"num_threads": 8},
        "w": {"num_threads": 24},
    },
)

Compare two-phase vs three-phase

# Two-phase: cheaper, faster
two_phase = optimizer.compile(
    program,
    trainset=trainset,
    valset=valset,
    strategy="p -> w",
)
two_phase_score = evaluator(two_phase)

# Three-phase: potentially better quality
three_phase = optimizer.compile(
    program,
    trainset=trainset,
    valset=valset,
    strategy="p -> w -> p",
)
three_phase_score = evaluator(three_phase)

print(f"Two-phase (p -> w):     {two_phase_score:.1f}%")
print(f"Three-phase (p -> w -> p): {three_phase_score:.1f}%")

Expected results

The third phase (re-optimizing prompts) typically adds 2-5 percentage points. Whether the extra cost is worth it depends on your quality requirements.

When to choose two-phase

• You're iterating quickly and want faster feedback
• The quality gap between two-phase and three-phase is small for your task
• You want to save on compute costs
• You plan to run BetterTogether multiple times with different configurations

Condensed from dspy.ai/api/optimizers/BetterTogether/. Verify against upstream for latest.

dspy.BetterTogether — API Reference

Constructor

dspy.BetterTogether(
    metric,                  # Callable (required)
    **optimizers,            # Teleprompter instances as keyword args
)

Inheritance

BetterTogether extends Teleprompter.

Methods

compile()

optimizer.compile(
    student,                              # Module (required)
    *,
    trainset,                             # list[Example] (required)
    teacher=None,                         # Module | list[Module] | None
    valset=None,                          # list[Example] | None
    num_threads=None,                     # int | None
    max_errors=None,                      # int | None
    provide_traceback=None,               # bool | None
    seed=None,                            # int | None
    valset_ratio=0.1,                     # float
    shuffle_trainset_between_steps=True,  # bool
    strategy="p -> w -> p",              # str
    optimizer_compile_args=None,          # dict[str, dict[str, Any]] | None
) -> Module

Returns: Optimized Module with two extra attributes:

• candidate_programs: List of dicts with "program", "score", "strategy" keys, sorted by score descending
• flag_compilation_error_occurred: True if any optimization step failed

Raises:

• ValueError if trainset is empty, valset_ratio outside [0, 1), strategy keys do not match optimizer names, or optimizer_compile_args has invalid keys
• TypeError if optimizer_compile_args contains "student" key

get_params()

optimizer.get_params() -> dict[str, Any]

Returns all configuration parameters as a dictionary.

Key behaviors

• Strategy execution: Applies optimizers sequentially per the strategy string. At each step, trainset is optionally shuffled and the result is evaluated on the validation set.
• Program selection: With validation data, returns the best-scoring program across all phases. Without validation, returns the latest program. Earlier programs win ties.
• Error resilience: If any optimization step fails, logs the error, continues to the next step, and sets flag_compilation_error_occurred = True.
• Model lifecycle: Automatically launches, kills, and relaunches models between steps (critical for local providers, no-ops for API LMs).
• Default optimizers: When no **optimizers are provided, uses BootstrapFewShotWithRandomSearch (key p) and BootstrapFinetune (key w).