Dspy Simba

Small-Step Optimization with dspy.SIMBA

Guide the user through using dspy.SIMBA (Stochastic Introspective Mini-Batch Ascent) to optimize DSPy programs through incremental, targeted improvements rather than large sweeping changes.

What is dspy.SIMBA

dspy.SIMBA is a DSPy optimizer that improves programs by analyzing mini-batches of examples, identifying where the program struggles most, and making small targeted fixes -- either adding demonstrations or generating self-reflective rules. Instead of rewriting the entire prompt at once, SIMBA takes conservative steps, focusing on the examples with the highest output variability.

Key properties:

• Mini-batch driven -- samples small batches from the training set each iteration, rather than evaluating the entire dataset
• Variability-focused -- identifies the hardest examples by measuring output variability (gap between best and worst scores)
• Two improvement strategies -- adds few-shot demonstrations or generates introspective rules based on failure analysis
• Maintains a program pool -- keeps multiple candidate programs and probabilistically selects from the best performers
• Incremental by design -- each step makes a small, targeted change rather than overhauling the entire program

When to use SIMBA

Use dspy.SIMBA when:

• You want conservative, incremental optimization that avoids regressions
• Your program already works reasonably well and you want to push accuracy higher
• You have a moderate dataset (50-500 examples) and want efficient optimization
• You need stability -- production systems where large prompt changes are risky
• You want to understand which examples are hardest for your program

Do not use SIMBA when:

• You are starting from scratch with no working program -- use dspy.BootstrapFewShot first
• You want maximum prompt optimization in one shot -- use dspy.MIPROv2 instead
• You need to fine-tune model weights -- use dspy.BootstrapFinetune
• Your dataset is very small (fewer than 30 examples) -- mini-batch sampling needs enough data

Basic usage

Three things are needed: a DSPy program, a metric function, and a training set.

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))  # or "anthropic/claude-sonnet-4-5-20250929", etc.

# 1. Define your program
classify = dspy.ChainOfThought("text -> label")

# 2. Define a metric
def metric(example, prediction, trace=None):
    return prediction.label.lower() == example.label.lower()

# 3. Build training data
trainset = [
    dspy.Example(text="Great product!", label="positive").with_inputs("text"),
    dspy.Example(text="Terrible service.", label="negative").with_inputs("text"),
    # ... more examples
]

# 4. Optimize with SIMBA
optimizer = dspy.SIMBA(metric=metric)
optimized = optimizer.compile(classify, trainset=trainset)

# 5. Use the optimized program
result = optimized(text="This exceeded my expectations!")
print(result.label)

# 6. Save for later
optimized.save("optimized_classifier.json")

How small-step optimization works

SIMBA's optimization loop proceeds through repeated small steps:

Step 1: Trajectory sampling

SIMBA runs the current program pool on a mini-batch of examples. Each program runs with distinct LM configurations to produce diverse outputs, scored by your metric.

Step 2: Bucket analysis

Examples are grouped and sorted by output variability -- the gap between the best and worst scores across runs. High-variability examples are where the program is inconsistent and has the most room for improvement.

Step 3: Strategy application

For each high-variability example, SIMBA applies one of two strategies:

• Demonstration injection -- takes a successful output for a hard example and adds it as a few-shot demonstration, teaching the program by example
• Introspective rules -- uses the LM to analyze why certain examples fail, then generates natural-language rules (instructions) that address the failure patterns

Step 4: Candidate evaluation

New candidate programs (with the added demos or rules) are evaluated on a fresh mini-batch. This prevents overfitting to the examples used for rule generation.

Step 5: Pool registration

The best-performing candidates are added to the program pool. Future iterations select source programs using softmax sampling weighted by average scores -- favoring better programs while still exploring alternatives.

This cycle repeats for max_steps iterations, with each step making a small, targeted improvement.

Constructor parameters

dspy.SIMBA(
    metric,                         # Scoring function (required)
    bsize=32,                       # Mini-batch size
    num_candidates=6,               # New candidates per iteration
    max_steps=8,                    # Number of optimization iterations
    max_demos=4,                    # Max demonstrations per predictor
    prompt_model=None,              # LM for generating rules (defaults to global LM)
    teacher_settings=None,          # Teacher model configuration dict
    demo_input_field_maxlen=100000, # Char limit for demo input fields
    num_threads=None,               # Parallel execution threads
    temperature_for_sampling=0.2,   # Temperature for trajectory sampling
    temperature_for_candidates=0.2, # Temperature for source program selection
)

Choosing parameter values

`bsize` (mini-batch size):

`max_steps`:

`num_candidates`:

Key methods

compile()

Runs the optimization loop and returns the best program found.

optimized = optimizer.compile(program, trainset=trainset, seed=0)

The seed parameter (default 0) controls random sampling for reproducible results.

The returned program includes two additional attributes:

• `candidate_programs` -- list of scored alternative programs discovered during optimization. Useful for ensemble strategies or analyzing what SIMBA tried.
• `trial_logs` -- per-batch metrics from each optimization step. Useful for understanding how performance evolved.

get_params()

Returns the optimizer's configuration as a dictionary. Useful for logging and experiment tracking.

params = optimizer.get_params()
print(params)
# {'bsize': 32, 'num_candidates': 6, 'max_steps': 8, ...}

Inspecting optimization results

After optimization, examine what SIMBA found:

optimizer = dspy.SIMBA(metric=metric)
optimized = optimizer.compile(program, trainset=trainset)

# Check the candidate pool
for i, (prog, score) in enumerate(optimized.candidate_programs):
    print(f"Candidate {i}: score={score:.3f}")

# Review trial logs to see improvement over time
for step, log in enumerate(optimized.trial_logs):
    print(f"Step {step}: {log}")

Comparison with other optimizers

Optimization workflow

A common approach is to layer optimizers:

1. Start with `BootstrapFewShot` to get a working baseline with good demonstrations 2. Then run SIMBA to incrementally improve by targeting the hardest examples 3. Optionally run `MIPROv2` if you need maximum quality and can tolerate larger changes

# Step 1: Bootstrap baseline
bootstrap = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
baseline = bootstrap.compile(program, trainset=trainset)

# Step 2: Incrementally improve with SIMBA
simba = dspy.SIMBA(metric=metric, max_steps=8)
improved = simba.compile(baseline, trainset=trainset)

Typical improvement trajectory

Expect incremental gains per step rather than a single large jump:

The exact numbers depend on your task, data, and LM. SIMBA shines on the incremental step — it finds the examples your program is inconsistent on and fixes those specifically.

Gotchas

1. Claude uses binary 0/1 metrics with SIMBA. SIMBA measures output variability (gap between best and worst scores) to find hard examples. With binary metrics, the variability is either 0 or 1 -- SIMBA cannot distinguish "almost right" from "completely wrong." Return floats between 0.0 and 1.0 so SIMBA can rank examples by difficulty meaningfully. 2. Claude runs SIMBA on an unoptimized program. SIMBA makes small incremental improvements -- it is not designed for large jumps from a blank slate. Run BootstrapFewShot first to establish a baseline with good demonstrations, then run SIMBA on the bootstrapped program to push accuracy higher. 3. Claude sets `max_demos` too high. Each demo added by SIMBA increases prompt length. With max_demos=10 and multi-paragraph examples, prompts can exceed context limits or degrade quality from demo overload. Keep max_demos at 4-6 (the default is 4). 4. Claude uses the same LM for `prompt_model` and the main program. SIMBA's introspective rules are generated by analyzing failures and writing natural-language instructions. If your main LM is small (e.g., gpt-4o-mini), the rule quality suffers. Set prompt_model to a stronger model for rule generation while keeping the cheaper model for the main program. 5. Claude ignores `candidate_programs` and `trial_logs` on the result. After compile(), the returned program has candidate_programs (list of scored alternatives) and trial_logs (per-step metrics). Inspecting these reveals whether optimization plateaued, which strategies worked, and whether alternative candidates might be better for specific inputs.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Quick-start optimization with few-shot examples -- see /ai-improving-accuracy
• Evaluating your program before and after optimization -- see /dspy-evaluate
• Building the program to optimize -- see /dspy-chain-of-thought or /dspy-modules
• Preparing training data -- see /dspy-data
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• dspy.SIMBA API docs
• For API details, see reference.md
• For worked examples, see examples.md

dspy.SIMBA Examples

Example 1: Incremental optimization of a classification pipeline

A support ticket classifier that already works at ~70% accuracy. SIMBA targets the hardest tickets -- ambiguous ones where the model is inconsistent -- and incrementally pushes accuracy higher.

import dspy
from dspy.evaluate import Evaluate

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))  # or "anthropic/claude-sonnet-4-5-20250929", etc.


# Define the classification signature
class ClassifyTicket(dspy.Signature):
    """Classify a support ticket into the correct department."""
    ticket_text: str = dspy.InputField(desc="The customer support ticket")
    department: str = dspy.OutputField(
        desc="One of: billing, technical, account, shipping, general"
    )


# Build the program
class TicketRouter(dspy.Module):
    def __init__(self):
        self.classify = dspy.ChainOfThought(ClassifyTicket)

    def forward(self, ticket_text):
        return self.classify(ticket_text=ticket_text)


# Prepare training and dev data
raw_data = [
    ("I was charged twice for my subscription", "billing"),
    ("The app crashes when I open settings", "technical"),
    ("I need to update my email address", "account"),
    ("My package hasn't arrived in 2 weeks", "shipping"),
    ("How do I export my data?", "technical"),
    ("Can I get a refund for last month?", "billing"),
    ("I forgot my password and can't reset it", "account"),
    ("The tracking number shows delivered but I didn't get it", "shipping"),
    ("Your API returns 500 errors intermittently", "technical"),
    ("I want to cancel and get a prorated refund", "billing"),
    ("How do I add a team member to my account?", "account"),
    ("The delivery was left at the wrong address", "shipping"),
    ("Integration with Slack stopped working", "technical"),
    ("I see an unknown charge on my invoice", "billing"),
    ("Can I merge two accounts?", "account"),
    ("My order status hasn't updated in 5 days", "shipping"),
    ("How do I contact support?", "general"),
    ("What are your business hours?", "general"),
    ("The checkout page won't load on mobile", "technical"),
    ("I need a copy of my receipt from January", "billing"),
    ("Where is your return policy?", "general"),
    ("Dashboard loading times are very slow", "technical"),
    ("I want to downgrade my plan", "billing"),
    ("How do I enable two-factor authentication?", "account"),
    ("Package arrived damaged", "shipping"),
    ("Do you offer student discounts?", "general"),
    ("OAuth login fails with Google accounts", "technical"),
    ("I was billed after cancelling", "billing"),
    ("Can I transfer my subscription to someone else?", "account"),
    ("Shipment is stuck in customs", "shipping"),
    ("File upload feature is broken", "technical"),
    ("I need an itemized invoice for tax purposes", "billing"),
    ("How do I delete my account permanently?", "account"),
    ("Wrong item was shipped to me", "shipping"),
    ("What payment methods do you accept?", "general"),
    ("The search feature returns no results", "technical"),
    ("Autopay didn't process this month", "billing"),
    ("I can't change my username", "account"),
    ("Estimated delivery date keeps changing", "shipping"),
    ("Is there a desktop app?", "general"),
]

examples = [
    dspy.Example(ticket_text=text, department=dept).with_inputs("ticket_text")
    for text, dept in raw_data
]

trainset = examples[:30]
devset = examples[30:]


# Define the metric
def correct_department(example, prediction, trace=None):
    return prediction.department.lower().strip() == example.department.lower().strip()


# Evaluate baseline
program = TicketRouter()
evaluator = Evaluate(devset=devset, metric=correct_department, num_threads=4)
baseline_score = evaluator(program)
print(f"Baseline accuracy: {baseline_score:.1f}%")


# Step 1: Bootstrap a starting point
bootstrap = dspy.BootstrapFewShot(
    metric=correct_department,
    max_bootstrapped_demos=3,
)
bootstrapped = bootstrap.compile(program, trainset=trainset)

bootstrap_score = evaluator(bootstrapped)
print(f"After BootstrapFewShot: {bootstrap_score:.1f}%")


# Step 2: Incrementally improve with SIMBA
optimizer = dspy.SIMBA(
    metric=correct_department,
    bsize=16,           # smaller batches since dataset is small
    num_candidates=4,   # moderate exploration
    max_steps=6,        # enough iterations to find improvements
    max_demos=4,        # keep prompts manageable
)

optimized = optimizer.compile(bootstrapped, trainset=trainset)


# Evaluate the optimized program
final_score = evaluator(optimized)
print(f"After SIMBA: {final_score:.1f}%")


# Inspect what SIMBA found
print(f"\nCandidate programs found: {len(optimized.candidate_programs)}")
for i, (prog, score) in enumerate(optimized.candidate_programs[:5]):
    print(f"  Candidate {i}: avg_score={score:.3f}")


# Save the best program
optimized.save("optimized_ticket_router.json")

What this demonstrates:

• Two-phase optimization -- BootstrapFewShot establishes a baseline, then SIMBA incrementally improves it
• Smaller `bsize` for a small dataset -- 16 instead of the default 32 to keep mini-batches meaningful
• Reduced `num_candidates` and `max_steps` to match the dataset size and budget
• Evaluation at each stage to track improvement from baseline to bootstrapped to SIMBA-optimized
• Inspecting candidate programs to understand what alternatives SIMBA explored

Example 2: Conservative tuning for production stability

A production Q&A system that must improve without regressing on already-correct answers. SIMBA's small-step approach is ideal here: each iteration makes a minimal change, and you can validate against a held-out set after each step.

import dspy
from dspy.evaluate import Evaluate

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))  # or "anthropic/claude-sonnet-4-5-20250929", etc.


# A multi-step Q&A pipeline already in production
class ProductionQA(dspy.Module):
    def __init__(self):
        self.analyze = dspy.ChainOfThought(
            "question -> key_concepts"
        )
        self.answer = dspy.ChainOfThought(
            "question, key_concepts -> answer"
        )

    def forward(self, question):
        analysis = self.analyze(question=question)
        return self.answer(
            question=question,
            key_concepts=analysis.key_concepts,
        )


# Production dataset with known-good answers
qa_pairs = [
    ("What causes tides?", "gravitational pull of the moon and sun"),
    ("Why is the sky blue?", "rayleigh scattering of sunlight"),
    ("How do vaccines work?", "stimulate immune response with weakened or inactive pathogens"),
    ("What is photosynthesis?", "process where plants convert sunlight into energy"),
    ("Why do seasons change?", "earth's axial tilt relative to its orbit around the sun"),
    ("How does GPS work?", "triangulation using signals from multiple satellites"),
    ("What causes earthquakes?", "movement of tectonic plates"),
    ("How do antibiotics work?", "kill bacteria or stop them from reproducing"),
    ("What is inflation?", "general increase in prices and decrease in purchasing power"),
    ("How does WiFi work?", "radio waves transmitting data between devices and a router"),
    ("What causes thunder?", "rapid expansion of air heated by lightning"),
    ("How do magnets work?", "alignment of magnetic domains creating a magnetic field"),
    ("What is DNA?", "molecule carrying genetic instructions for development and function"),
    ("Why do we dream?", "brain processes memories and emotions during sleep"),
    ("How does a car engine work?", "internal combustion converts fuel into mechanical energy"),
    ("What causes rainbows?", "refraction and reflection of light in water droplets"),
    ("How do airplanes fly?", "lift generated by air pressure difference over wings"),
    ("What is machine learning?", "algorithms that improve through experience with data"),
    ("Why do leaves change color?", "chlorophyll breaks down revealing other pigments"),
    ("How does the internet work?", "network of networks using standardized protocols to route data"),
]

examples = [
    dspy.Example(question=q, answer=a).with_inputs("question")
    for q, a in qa_pairs
]

trainset = examples[:14]
devset = examples[14:]


# Graduated metric -- partial credit for close answers
def answer_quality(example, prediction, trace=None):
    pred = prediction.answer.lower().strip()
    gold = example.answer.lower().strip()

    # Exact match
    if gold in pred:
        return 1.0

    # Check for key term overlap
    gold_terms = set(gold.split())
    pred_terms = set(pred.split())
    overlap = gold_terms & pred_terms
    if not gold_terms:
        return 0.0

    overlap_ratio = len(overlap) / len(gold_terms)

    # Penalize very long answers (want conciseness)
    length_penalty = 1.0
    if len(pred.split()) > 50:
        length_penalty = 0.8

    return overlap_ratio * length_penalty


# Load the existing production program (or start fresh)
program = ProductionQA()

# Evaluate current production performance
evaluator = Evaluate(
    devset=devset,
    metric=answer_quality,
    num_threads=4,
    display_progress=True,
)
production_score = evaluator(program)
print(f"Current production score: {production_score:.2f}")


# Conservative SIMBA optimization
# - Low num_candidates to limit the size of changes
# - Default temperature to keep outputs stable
# - max_demos=3 to avoid bloating the production prompt
optimizer = dspy.SIMBA(
    metric=answer_quality,
    bsize=12,            # small batches from limited data
    num_candidates=4,    # conservative -- fewer candidates per step
    max_steps=6,         # moderate iteration count
    max_demos=3,         # keep prompts lean for production
)

optimized = optimizer.compile(program, trainset=trainset)

# Validate on held-out set
optimized_score = evaluator(optimized)
print(f"Optimized score: {optimized_score:.2f}")
print(f"Improvement: {optimized_score - production_score:+.2f}")


# Safety check: verify no regression on individual examples
print("\nPer-example comparison:")
regressions = 0
improvements = 0

for ex in devset:
    old_pred = program(question=ex.question)
    new_pred = optimized(question=ex.question)

    old_score = answer_quality(ex, old_pred)
    new_score = answer_quality(ex, new_pred)

    if new_score < old_score - 0.1:
        regressions += 1
        print(f"  REGRESSION: '{ex.question}' ({old_score:.2f} -> {new_score:.2f})")
    elif new_score > old_score + 0.1:
        improvements += 1
        print(f"  IMPROVED:   '{ex.question}' ({old_score:.2f} -> {new_score:.2f})")

print(f"\nImprovements: {improvements}, Regressions: {regressions}")


# Only deploy if no regressions (or regressions are acceptable)
if regressions == 0:
    optimized.save("production_qa_optimized.json")
    print("Safe to deploy -- saved optimized program.")
else:
    print(f"Found {regressions} regression(s) -- review before deploying.")
    # Still save for analysis
    optimized.save("production_qa_candidate.json")

What this demonstrates:

• Production safety workflow -- evaluates on a held-out dev set and checks for regressions before saving
• Conservative parameters -- low num_candidates (4) and max_demos (3) to minimize prompt changes
• Graduated metric -- partial credit for keyword overlap plus a length penalty for conciseness
• Per-example regression analysis -- compares old and new predictions individually to catch quality drops
• Conditional deployment -- only saves as the production model if no regressions are found
• Multi-step pipeline -- SIMBA optimizes both the analyze and answer predictors within ProductionQA

SIMBA API Reference

Condensed from dspy.ai/api/optimizers/SIMBA. Verify against upstream for latest.

Constructor

dspy.SIMBA(
    *,
    metric,                          # Callable -- required
    bsize=32,                        # mini-batch size
    num_candidates=6,                # candidates per iteration
    max_steps=8,                     # optimization iterations
    max_demos=4,                     # max demos per predictor
    prompt_model=None,               # dspy.LM for rule generation
    teacher_settings=None,           # teacher model config dict
    demo_input_field_maxlen=100000,  # char limit for demo inputs
    num_threads=None,                # parallel threads
    temperature_for_sampling=0.2,    # trajectory sampling temperature
    temperature_for_candidates=0.2,  # source program selection temperature
)

Key Methods

compile()

optimized = optimizer.compile(student, *, trainset, seed=0)

Returns an optimized dspy.Module with:

• candidate_programs -- list of (program, score) tuples
• trial_logs -- per-step metrics

get_params()

optimizer.get_params() -> dict[str, Any]

Returns optimizer configuration as a dictionary.