Ai Switching Models

Switch Models Without Breaking Things

Guide the user through switching AI models or providers safely. The key insight: optimized prompts don't transfer between models (arxiv 2402.10949v2 — "The Unreasonable Effectiveness of Eccentric Automatic Prompts"). DSPy solves this by separating your task definition (signatures + modules) from model-specific prompts (compiled by optimizers).

Why switching models breaks things

Hand-tuned prompts are model-specific. A prompt engineered for GPT-4o will perform differently on Claude, Llama, or even GPT-4o-mini. Research shows optimized prompts for one model can actually hurt performance on another.

DSPy makes switching safe because:

• Signatures define what the task is (inputs, outputs, types) — model-independent
• Modules define how to solve it (chain of thought, ReAct, etc.) — model-independent
• Compiled prompts (few-shot examples, instructions) are model-specific — but re-generated automatically by optimizers

The workflow: keep your program the same, swap the model, re-optimize. Done.

Step 1: Understand the situation

Ask the user: 1. What model are you using now, and what do you want to switch to? (e.g., GPT-4o to Claude, cloud to local) 2. Why are you switching? (cost, vendor diversification, performance regression, privacy) 3. Do you have evaluation metrics and test data? (needed to measure if the switch works — if not, start with /ai-improving-accuracy)

Common scenarios:

• Cost reduction — "GPT-4o is too expensive, can we use something cheaper?"
• Vendor diversification — "We can't depend on one provider"
• Performance regression — "The provider updated their model and our outputs got worse"
• Data privacy / compliance — "We need to run models on our own infrastructure"

Step 2: Configure any provider

DSPy uses LiteLLM under the hood, so you can use any supported provider with a simple string:

import dspy

# OpenAI
lm = dspy.LM("openai/gpt-4o")
lm = dspy.LM("openai/gpt-4o-mini")

# Anthropic
lm = dspy.LM("anthropic/claude-sonnet-4-5-20250929")
lm = dspy.LM("anthropic/claude-haiku-4-5-20251001")

# Azure OpenAI
lm = dspy.LM("azure/my-gpt4-deployment")

# Google
lm = dspy.LM("gemini/gemini-2.0-flash")

# Together AI (open-source models)
lm = dspy.LM("together_ai/meta-llama/Llama-3-70b-chat-hf")

# Local models (via Ollama)
lm = dspy.LM("ollama_chat/llama3.1", api_base="http://localhost:11434")

# Any OpenAI-compatible server (vLLM, TGI, etc.)
lm = dspy.LM("openai/my-model", api_base="http://localhost:8000/v1", api_key="none")

dspy.configure(lm=lm)

Environment variables

Set API keys as environment variables — don't hardcode them:

# .env file
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
TOGETHER_API_KEY=...
AZURE_API_KEY=...
AZURE_API_BASE=https://your-resource.openai.azure.com/

See LiteLLM provider docs for the full list of 100+ supported providers.

Step 3: Benchmark your current model

Before changing anything, measure your baseline. You need a metric and test data.

from dspy.evaluate import Evaluate

# Your existing program and metric
program = MyProgram()
program.load("current_optimized.json")  # load your production prompts

evaluator = Evaluate(
    devset=devset,
    metric=metric,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

# Benchmark with your current model
current_lm = dspy.LM("openai/gpt-4o")
dspy.configure(lm=current_lm)
baseline_score = evaluator(program)
print(f"Current model baseline: {baseline_score:.1f}%")

If you don't have a metric or test data yet, use /ai-improving-accuracy to set them up first.

Step 4: Try the new model (quick test)

Swap the model and run your evaluation without re-optimizing. This demonstrates the problem — your old prompts don't transfer.

# Try the new model with your OLD optimized prompts
new_lm = dspy.LM("anthropic/claude-sonnet-4-5-20250929")
dspy.configure(lm=new_lm)

naive_score = evaluator(program)
print(f"Old model (optimized):  {baseline_score:.1f}%")
print(f"New model (old prompts): {naive_score:.1f}%")
print(f"Drop: {baseline_score - naive_score:.1f}%")

You'll typically see a quality drop — this is expected. The optimized prompts were tuned for the old model.

Step 5: Re-optimize for the new model

Now re-optimize your program for the new model. Use the same signatures and modules — only the compiled prompts change.

# Configure the new model
new_lm = dspy.LM("anthropic/claude-sonnet-4-5-20250929")
dspy.configure(lm=new_lm)

# Start from a fresh (unoptimized) program
fresh_program = MyProgram()

# Re-optimize for the new model
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized_for_new = optimizer.compile(fresh_program, trainset=trainset)

# Evaluate
reoptimized_score = evaluator(optimized_for_new)
print(f"Old model (optimized):      {baseline_score:.1f}%")
print(f"New model (old prompts):     {naive_score:.1f}%")
print(f"New model (re-optimized):    {reoptimized_score:.1f}%")

The re-optimized score should recover most or all of the quality. If it doesn't, either:

• The new model genuinely can't handle this task as well
• Try a heavier optimization (auto="heavy")
• Try BootstrapFewShot first for a quick sanity check

Quick re-optimization (fast test)

For a quick check before committing to a full MIPROv2 run:

optimizer = dspy.BootstrapFewShot(
    metric=metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,
)
quick_optimized = optimizer.compile(fresh_program, trainset=trainset)
quick_score = evaluator(quick_optimized)

Step 6: Compare models systematically

Loop over candidate models, optimize each, and build a comparison table:

candidates = [
    ("openai/gpt-4o", "GPT-4o"),
    ("openai/gpt-4o-mini", "GPT-4o-mini"),
    ("anthropic/claude-sonnet-4-5-20250929", "Claude Sonnet"),
    ("together_ai/meta-llama/Llama-3-70b-chat-hf", "Llama 3 70B"),
]

results = []
for model_id, label in candidates:
    lm = dspy.LM(model_id)
    dspy.configure(lm=lm)

    # Optimize for this model
    fresh = MyProgram()
    optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
    optimized = optimizer.compile(fresh, trainset=trainset)

    # Evaluate
    score = evaluator(optimized)

    # Save the optimized program
    optimized.save(f"optimized_{label.lower().replace(' ', '_')}.json")

    results.append({"model": label, "score": score})
    print(f"{label}: {score:.1f}%")

# Print comparison table
print("\n--- Model Comparison ---")
print(f"{'Model':<25} {'Score':>8}")
print("-" * 35)
for r in sorted(results, key=lambda x: x["score"], reverse=True):
    print(f"{r['model']:<25} {r['score']:>7.1f}%")

For a more thorough comparison with MIPROv2 and cost/latency tracking, see examples.md.

Step 7: Mix models in one pipeline

You don't have to use one model for everything. Assign different models to different steps — cheap for simple tasks, expensive for hard ones.

Using dspy.context (temporary, per-call)

cheap_lm = dspy.LM("openai/gpt-4o-mini")
expensive_lm = dspy.LM("openai/gpt-4o")

dspy.configure(lm=expensive_lm)  # default

class MyPipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict(ClassifySignature)
        self.generate = dspy.ChainOfThought(GenerateSignature)

    def forward(self, text):
        # Cheap model for simple classification
        with dspy.context(lm=cheap_lm):
            category = self.classify(text=text)

        # Expensive model for complex generation
        return self.generate(text=text, category=category.label)

Using set_lm (permanent, per-module)

pipeline = MyPipeline()
pipeline.classify.set_lm(cheap_lm)
pipeline.generate.set_lm(expensive_lm)

See /ai-cutting-costs for more cost optimization patterns with per-module LM assignment.

Step 8: Save and deploy

Save a separate optimized program for each model you might use in production:

# Save per-model optimized programs
optimized_gpt4o.save("optimized_gpt4o.json")
optimized_claude.save("optimized_claude.json")
optimized_llama.save("optimized_llama.json")

# In production — load the right one
import os

model_name = os.environ.get("AI_MODEL", "openai/gpt-4o")
lm = dspy.LM(model_name)
dspy.configure(lm=lm)

program = MyProgram()
program.load(f"optimized_{model_name.split('/')[-1]}.json")

Common scenarios

GPT-4o to GPT-4o-mini (cost reduction)

1. Benchmark GPT-4o baseline (Step 2) 2. Try GPT-4o-mini with old prompts — see the drop (Step 3) 3. Re-optimize for GPT-4o-mini with MIPROv2 (Step 4) 4. Compare scores — if quality is close enough, ship it

OpenAI to Anthropic (vendor diversification)

1. Set up Anthropic API key in environment 2. Change model string: "openai/gpt-4o" to "anthropic/claude-sonnet-4-5-20250929" 3. Re-optimize — different models need different prompts 4. Keep both optimized programs, switch via environment variable

Cloud to local (data privacy)

1. Set up local model server (Ollama, vLLM, or TGI) 2. Point DSPy at it: dspy.LM("ollama_chat/llama3.1", api_base="http://localhost:11434") 3. Re-optimize — local models especially need re-optimization 4. Expect some quality trade-off vs large cloud models; use heavier optimization

Model version update broke things

When a provider updates their model (e.g., GPT-4o version bump): 1. Run your evaluation to confirm the regression 2. Re-optimize against the updated model 3. Save the new optimized program 4. This is why having evaluation + optimization in your workflow matters — version updates become routine, not emergencies

When NOT to switch models

• You have not set up evaluation yet. Without a metric and test set, you cannot tell if the new model is better or worse. Set up evaluation first with /ai-improving-accuracy.
• You are debugging prompt quality, not the model. If your outputs are bad on your current model, switching models will not fix a poorly defined signature or missing examples. Optimize your current setup first.
• You only need a faster response, not a different model. If latency is the issue, consider caching (dspy.cache), shorter signatures, or dspy.Predict instead of dspy.ChainOfThought before switching to a weaker model.
• Your task is simple enough that any model works. If zero-shot dspy.Predict already scores 95%+, the model choice barely matters. Focus effort elsewhere.

Gotchas

• Reusing optimized prompts across models without re-optimization. Claude defaults to loading a saved program and swapping only the LM config. The compiled few-shot demos and instructions are tuned for the original model and typically degrade on a different one. Always re-optimize from a fresh (uncompiled) program after switching models.
• Comparing models using unoptimized or single-model prompts. Running candidates with zero-shot prompts or with prompts optimized for one model gives misleading rankings. Optimize each candidate independently before comparing scores, or the comparison measures prompt fit, not model capability.
• Forgetting to pin the judge model during model shootouts. When using an LLM-as-judge metric, the judge model changes if you call dspy.configure(lm=candidate_lm) without isolating the judge. Use dspy.context(lm=judge_lm) inside your metric function so the judge stays constant across all candidates.
• Using `dspy.context` when `set_lm` is needed (and vice versa). dspy.context(lm=...) is temporary and scoped to a with block -- good for per-call overrides. module.set_lm(lm) is permanent and persists through optimization -- use it when a module should always use a specific model. Mixing them up causes silent evaluation bugs.
• Expecting local models to match cloud model quality without heavier optimization. Smaller local models (7B-13B) typically need more bootstrapped demos and heavier optimization (auto="heavy") to approach cloud model quality. Start with BootstrapFewShot with max_bootstrapped_demos=8 and move to MIPROv2 if scores are still low.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Set up metrics and evaluation before switching -- see /ai-improving-accuracy
• Per-module model assignment for cost optimization -- see /ai-cutting-costs
• Multi-step pipelines with mixed models -- see /ai-building-pipelines
• Distill from expensive model to cheap one -- see /ai-fine-tuning
• Understand DSPy optimizers for re-optimization -- see /dspy-optimizers
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• For worked examples (cost migration, vendor switch, model shootout), see examples.md

Examples: Switching Models

Example 1: Cost migration — GPT-4o to GPT-4o-mini

A support ticket classifier running on GPT-4o costs too much. Let's switch to GPT-4o-mini and see if quality holds.

Setup

import dspy
from dspy.evaluate import Evaluate

# The task: classify support tickets
class ClassifyTicket(dspy.Signature):
    """Classify a support ticket into a category."""
    ticket_text: str = dspy.InputField()
    category: str = dspy.OutputField(desc="one of: billing, technical, account, feature_request, other")

class TicketClassifier(dspy.Module):
    def __init__(self):
        self.classify = dspy.ChainOfThought(ClassifyTicket)

    def forward(self, ticket_text):
        return self.classify(ticket_text=ticket_text)

# Metric
def metric(example, prediction, trace=None):
    return prediction.category.strip().lower() == example.category.strip().lower()

# Test data (50+ examples for reliable evaluation)
devset = [
    dspy.Example(ticket_text="I was charged twice for my subscription", category="billing").with_inputs("ticket_text"),
    dspy.Example(ticket_text="The API returns 500 errors", category="technical").with_inputs("ticket_text"),
    # ... more examples
]
trainset = devset[:40]  # for optimization
testset = devset[40:]   # held out for final eval

evaluator = Evaluate(devset=testset, metric=metric, num_threads=4, display_progress=True)

Step 1: Benchmark GPT-4o (the expensive model)

gpt4o = dspy.LM("openai/gpt-4o")
dspy.configure(lm=gpt4o)

# Optimize for GPT-4o
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized_gpt4o = optimizer.compile(TicketClassifier(), trainset=trainset)

baseline = evaluator(optimized_gpt4o)
print(f"GPT-4o (optimized): {baseline:.1f}%")
# GPT-4o (optimized): 92.0%

Step 2: Try GPT-4o-mini with GPT-4o's prompts (see the drop)

gpt4o_mini = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=gpt4o_mini)

# Use GPT-4o's optimized prompts on GPT-4o-mini — no re-optimization
naive_score = evaluator(optimized_gpt4o)
print(f"GPT-4o-mini (GPT-4o's prompts): {naive_score:.1f}%")
# GPT-4o-mini (GPT-4o's prompts): 78.0%  <-- quality dropped!

The optimized prompts from GPT-4o don't work well on GPT-4o-mini. This is the core finding from the research — prompts are model-specific.

Step 3: Re-optimize for GPT-4o-mini

dspy.configure(lm=gpt4o_mini)

# Fresh program, optimize specifically for GPT-4o-mini
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized_mini = optimizer.compile(TicketClassifier(), trainset=trainset)

reoptimized_score = evaluator(optimized_mini)
print(f"\n--- Results ---")
print(f"GPT-4o (optimized):              {baseline:.1f}%")
print(f"GPT-4o-mini (old prompts):       {naive_score:.1f}%")
print(f"GPT-4o-mini (re-optimized):      {reoptimized_score:.1f}%")
# GPT-4o (optimized):              92.0%
# GPT-4o-mini (old prompts):       78.0%   <-- 14% drop without re-optimization
# GPT-4o-mini (re-optimized):      89.0%   <-- recovered most of the quality

Step 4: Ship it

# 89% vs 92% — close enough for a 33x cost reduction
optimized_mini.save("ticket_classifier_gpt4o_mini.json")

# In production
classifier = TicketClassifier()
classifier.load("ticket_classifier_gpt4o_mini.json")
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
result = classifier(ticket_text="I can't log in to my account")

Key takeaway: Without re-optimization, switching to GPT-4o-mini dropped quality by 14%. With re-optimization, the drop was only 3% — an acceptable trade-off for 33x cost savings.

---

Example 2: Vendor switch — OpenAI to Anthropic

The team wants to reduce dependency on OpenAI. Let's migrate a Q&A system to Anthropic.

Setup

import dspy
from dspy.evaluate import Evaluate

class AnswerQuestion(dspy.Signature):
    """Answer the question based on the given context."""
    context: str = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

class QASystem(dspy.Module):
    def __init__(self):
        self.answer = dspy.ChainOfThought(AnswerQuestion)

    def forward(self, context, question):
        return self.answer(context=context, question=question)

# F1 metric for text overlap
def metric(example, prediction, trace=None):
    gold_tokens = set(example.answer.lower().split())
    pred_tokens = set(prediction.answer.lower().split())
    if not gold_tokens or not pred_tokens:
        return float(gold_tokens == pred_tokens)
    precision = len(gold_tokens & pred_tokens) / len(pred_tokens)
    recall = len(gold_tokens & pred_tokens) / len(gold_tokens)
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

evaluator = Evaluate(devset=testset, metric=metric, num_threads=4, display_progress=True)

Step 1: Benchmark OpenAI baseline

openai_lm = dspy.LM("openai/gpt-4o")
dspy.configure(lm=openai_lm)

optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized_openai = optimizer.compile(QASystem(), trainset=trainset)

openai_score = evaluator(optimized_openai)
print(f"OpenAI GPT-4o (optimized): {openai_score:.1f}%")
# OpenAI GPT-4o (optimized): 85.2%

Step 2: Try Anthropic with OpenAI's prompts

claude_lm = dspy.LM("anthropic/claude-sonnet-4-5-20250929")
dspy.configure(lm=claude_lm)

# OpenAI's optimized prompts on Claude
naive_score = evaluator(optimized_openai)
print(f"Claude (OpenAI's prompts): {naive_score:.1f}%")
# Claude (OpenAI's prompts): 76.8%

Step 3: Re-optimize for Anthropic

dspy.configure(lm=claude_lm)

optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized_claude = optimizer.compile(QASystem(), trainset=trainset)

claude_score = evaluator(optimized_claude)
print(f"\n--- Results ---")
print(f"OpenAI GPT-4o (optimized):     {openai_score:.1f}%")
print(f"Claude (OpenAI's prompts):      {naive_score:.1f}%")
print(f"Claude (re-optimized):          {claude_score:.1f}%")
# OpenAI GPT-4o (optimized):     85.2%
# Claude (OpenAI's prompts):      76.8%  <-- 8.4% drop
# Claude (re-optimized):          86.1%  <-- actually beat the original!

Step 4: Deploy with model selection

# Save both optimized programs
optimized_openai.save("qa_openai_gpt4o.json")
optimized_claude.save("qa_anthropic_claude.json")

# In production — select model via environment variable
import os

model_configs = {
    "openai": ("openai/gpt-4o", "qa_openai_gpt4o.json"),
    "anthropic": ("anthropic/claude-sonnet-4-5-20250929", "qa_anthropic_claude.json"),
}

provider = os.environ.get("AI_PROVIDER", "anthropic")
model_id, program_path = model_configs[provider]

lm = dspy.LM(model_id)
dspy.configure(lm=lm)

qa = QASystem()
qa.load(program_path)

Key takeaway: Swapping providers without re-optimization lost 8.4%. After re-optimization, Claude actually scored higher. The DSPy program (signatures + modules) didn't change at all — only the compiled prompts did.

---

Example 3: Model shootout — compare 4 models

Choosing a model for a new feature? Run a systematic comparison.

Setup

import dspy
from dspy.evaluate import Evaluate

class Summarize(dspy.Signature):
    """Summarize the article in 2-3 sentences."""
    article: str = dspy.InputField()
    summary: str = dspy.OutputField()

class Summarizer(dspy.Module):
    def __init__(self):
        self.summarize = dspy.ChainOfThought(Summarize)

    def forward(self, article):
        return self.summarize(article=article)

# AI-as-judge metric
class AssessSummary(dspy.Signature):
    """Assess if the summary captures the key points of the article."""
    article: str = dspy.InputField()
    gold_summary: str = dspy.InputField()
    predicted_summary: str = dspy.InputField()
    is_good: bool = dspy.OutputField()

# Use a strong model as the judge (separate from candidates)
judge_lm = dspy.LM("openai/gpt-4o")

def metric(example, prediction, trace=None):
    with dspy.context(lm=judge_lm):
        judge = dspy.Predict(AssessSummary)
        result = judge(
            article=example.article,
            gold_summary=example.summary,
            predicted_summary=prediction.summary,
        )
    return result.is_good

evaluator = Evaluate(devset=testset, metric=metric, num_threads=4, display_progress=True)

Run the shootout

candidates = [
    ("openai/gpt-4o", "GPT-4o"),
    ("openai/gpt-4o-mini", "GPT-4o-mini"),
    ("anthropic/claude-sonnet-4-5-20250929", "Claude Sonnet"),
    ("together_ai/meta-llama/Llama-3-70b-chat-hf", "Llama 3 70B"),
]

results = []

for model_id, label in candidates:
    print(f"\n{'='*40}")
    print(f"Testing: {label}")
    print(f"{'='*40}")

    lm = dspy.LM(model_id)
    dspy.configure(lm=lm)

    # Quick optimization (BootstrapFewShot for speed)
    fresh = Summarizer()
    optimizer = dspy.BootstrapFewShot(
        metric=metric,
        max_bootstrapped_demos=4,
        max_labeled_demos=4,
    )
    optimized = optimizer.compile(fresh, trainset=trainset)

    # Evaluate
    score = evaluator(optimized)

    # Save
    optimized.save(f"summarizer_{label.lower().replace(' ', '_')}.json")

    results.append({
        "model": label,
        "model_id": model_id,
        "score": score,
    })

Compare results

print("\n" + "=" * 50)
print("MODEL COMPARISON")
print("=" * 50)
print(f"{'Model':<25} {'Score':>8}")
print("-" * 35)
for r in sorted(results, key=lambda x: x["score"], reverse=True):
    print(f"{r['model']:<25} {r['score']:>7.1f}%")

# Example output:
# MODEL COMPARISON
# ==================================================
# Model                      Score
# -----------------------------------
# Claude Sonnet               88.0%
# GPT-4o                      86.0%
# Llama 3 70B                 82.0%
# GPT-4o-mini                 79.0%

Deeper comparison with MIPROv2

If BootstrapFewShot results are close, run MIPROv2 on the top contenders for a more accurate comparison:

top_models = [r for r in sorted(results, key=lambda x: x["score"], reverse=True)[:2]]

for r in top_models:
    lm = dspy.LM(r["model_id"])
    dspy.configure(lm=lm)

    fresh = Summarizer()
    optimizer = dspy.MIPROv2(metric=metric, auto="medium")
    optimized = optimizer.compile(fresh, trainset=trainset)

    score = evaluator(optimized)
    optimized.save(f"summarizer_{r['model'].lower().replace(' ', '_')}_mipro.json")
    print(f"{r['model']} (MIPROv2): {score:.1f}%")

Key takeaway: Always optimize per-model before comparing. Comparing models with unoptimized (or another model's) prompts gives misleading results. The ranking can change after optimization.