Dspy Lm

Configure Language Models with dspy.LM

dspy.LM is DSPy's unified interface for calling language models. It wraps LiteLLM so any provider -- OpenAI, Anthropic, Google, Together AI, Ollama, vLLM, and 100+ others -- works through one consistent API. You configure a model once, then every DSPy module uses it automatically.

Basic setup

import dspy

# Create an LM instance with a provider/model string
lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.

# Set it as the default for all DSPy modules
dspy.configure(lm=lm)

# Now any module uses this LM automatically
classify = dspy.ChainOfThought("text -> label")
result = classify(text="DSPy is great")
print(result.label)

The pattern is always: dspy.LM("provider/model") then dspy.configure(lm=lm).

Provider strings

DSPy uses the LiteLLM "provider/model-name" format. Here are the most common providers:

See LiteLLM provider docs for the full list.

Constructor parameters

lm = dspy.LM(
    model="openai/gpt-4o",           # Required: "provider/model-name"
    model_type="chat",                # "chat" (default), "text", or "responses"
    temperature=0.7,                  # Sampling temperature (default: provider default)
    max_tokens=1000,                  # Max output tokens (default: provider default)
    cache=True,                       # Enable built-in caching (default: True)
    num_retries=3,                    # Retry on transient failures (default: 3)
    use_developer_role=False,         # Use developer/system role (default: False)
    # Plus any extra kwargs passed to LiteLLM
)

Key parameters

• `model` (required) -- The provider/model string. This is the only required argument.
• `temperature` -- Controls randomness. Lower = more deterministic. Set to 0.0 for reproducible outputs. Reasoning models (o1, o3) require temperature=1.0 or None.
• `max_tokens` -- Maximum tokens in the response. Reasoning models require max_tokens >= 16000 or None.
• `cache` -- When True (the default), DSPy caches LM responses to reduce costs and speed up repeated calls. Set to False to disable.
• `num_retries` -- Number of retries with exponential backoff on transient failures.
• `model_type` -- Usually leave as "chat". Use "text" for completion-only models. Use "responses" for OpenAI responses API.

Per-module LM assignment

You do not have to use the same model for every step. Assign different LMs to different modules with set_lm():

expensive_lm = dspy.LM("openai/gpt-4o")
cheap_lm = dspy.LM("openai/gpt-4o-mini")

# Set a default
dspy.configure(lm=cheap_lm)

class MyPipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict("text -> category")
        self.generate = dspy.ChainOfThought("text, category -> summary")

    def forward(self, text):
        category = self.classify(text=text)
        return self.generate(text=text, category=category.category)

pipeline = MyPipeline()

# Route: cheap model for classification, expensive for generation
pipeline.classify.set_lm(cheap_lm)
pipeline.generate.set_lm(expensive_lm)

Temporary LM override with dspy.context

Use dspy.context to temporarily switch LMs for a block of code:

with dspy.context(lm=expensive_lm):
    # Everything inside uses expensive_lm
    result = pipeline(text="important document")

# Back to the default LM outside the block

Direct LM calls

You can call an LM instance directly for one-off prompts outside of DSPy modules:

lm = dspy.LM("openai/gpt-4o-mini")

# Pass a string prompt
response = lm("What is the capital of France?")
print(response)  # returns a list of strings

# Pass a messages list (chat format)
response = lm(messages=[
    {"role": "user", "content": "What is the capital of France?"}
])
print(response)  # returns a list of strings

Direct calls are useful for quick tests, but for structured tasks use DSPy modules and signatures -- they give you type checking, optimization, and caching.

Environment variables

Set API keys as environment variables. Never hardcode them.

# OpenAI
export OPENAI_API_KEY=sk-...

# Anthropic
export ANTHROPIC_API_KEY=sk-ant-...

# Together AI
export TOGETHER_API_KEY=...

# Google
export GEMINI_API_KEY=...

# Groq
export GROQ_API_KEY=...

# Azure OpenAI
export AZURE_API_KEY=...
export AZURE_API_BASE=https://your-resource.openai.azure.com/

DSPy (via LiteLLM) reads these automatically. You can also pass api_key directly to dspy.LM() if needed, but environment variables are preferred.

Caching

DSPy caches LM responses by default. This means:

• Repeated identical calls are free -- same prompt, same parameters, same model returns a cached result instantly with no API call.
• Development is faster -- re-running your script doesn't re-call the LM for already-seen inputs.
• Optimization is cheaper -- optimizers that re-evaluate examples benefit from cached results.

Controlling caching

# Caching enabled (default)
lm = dspy.LM("openai/gpt-4o-mini", cache=True)

# Disable caching for this LM
lm = dspy.LM("openai/gpt-4o-mini", cache=False)

# Configure cache settings globally
dspy.configure_cache(
    enable=True,          # Toggle caching on/off
)

Cache is stored locally. If you need different responses for the same prompt (e.g., generating diverse examples), disable caching or use different temperature values.

Useful methods

Inspecting history

lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)

classify = dspy.Predict("text -> label")
classify(text="Hello world")

# See what was sent to the LM
dspy.inspect_history(n=1)

Connecting to local models

Ollama

# Start Ollama: ollama serve
# Pull a model: ollama pull llama3.1
lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
    api_key="",
    temperature=0.7,
    num_ctx=8192,  # set context window explicitly — Ollama defaults to 4096
)
dspy.configure(lm=lm)

For full Ollama setup (model selection, GPU tuning, context window gotchas, optimization tips), see /dspy-ollama.

vLLM or any OpenAI-compatible server

# Start vLLM: vllm serve meta-llama/Llama-3.1-8B-Instruct
lm = dspy.LM(
    "openai/meta-llama/Llama-3.1-8B-Instruct",
    api_base="http://localhost:8000/v1",
    api_key="none",
)
dspy.configure(lm=lm)

For any server that exposes an OpenAI-compatible /v1/chat/completions endpoint, use the "openai/model-name" provider string with api_base pointing to your server.

For full vLLM setup (tensor parallelism, GPU sizing, quantization, production deployment), see /dspy-vllm.

Gotchas

1. Claude omits the provider prefix from the model string. Claude writes dspy.LM("gpt-4o-mini") instead of dspy.LM("openai/gpt-4o-mini"). While some models auto-detect the provider, the explicit "provider/model" format is required for reliable routing through LiteLLM. Always include the provider prefix. 2. Claude sets `temperature=0` for reasoning models. OpenAI reasoning models (o1, o3, o4, gpt-5 families) require temperature=1.0 or None. Setting temperature=0 raises an error. Similarly, max_tokens must be >= 16000 or None for these models. 3. Claude calls `dspy.configure(lm=lm)` inside `forward()`. Configuration should happen once at the top of your script, not per-call. Calling dspy.configure inside forward() resets global state on every invocation and breaks caching. Use set_lm() or dspy.context() for per-module or temporary overrides instead. 4. Claude forgets `api_base` for local models. Ollama and vLLM require api_base pointing to the local server (http://localhost:11434 for Ollama, http://localhost:8000/v1 for vLLM). Without it, DSPy tries to reach the cloud API and fails with an authentication error. 5. Claude hardcodes API keys in source code. API keys should be set as environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.), never passed directly to dspy.LM(). DSPy reads them automatically via LiteLLM.

Additional resources

• dspy.LM API docs
• LiteLLM provider docs
• For API details, see reference.md
• For worked examples, see examples.md

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• `/dspy-signatures` -- Define what your LM should do (inputs, outputs, types)
• `/dspy-modules` -- Wrap signatures with inference strategies (Predict, ChainOfThought, ReAct)
• `/ai-switching-models` -- Safely migrate between providers with re-optimization
• `/ai-cutting-costs` -- Reduce LM costs with per-module assignment and cheaper models
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Examples: Configuring Language Models

Example 1: Multi-provider setup

Switch between OpenAI and Anthropic, showing how the provider string format works and how to verify your configuration.

import dspy

# --- OpenAI setup ---
openai_lm = dspy.LM("openai/gpt-4o-mini", temperature=0.0)
dspy.configure(lm=openai_lm)

# Test it
classify = dspy.ChainOfThought("text -> sentiment: str")
result = classify(text="I love this product!")
print(f"OpenAI says: {result.sentiment}")

# --- Switch to Anthropic ---
anthropic_lm = dspy.LM("anthropic/claude-sonnet-4-5-20250929", temperature=0.0)
dspy.configure(lm=anthropic_lm)

# Same module, different provider -- no code changes needed
result = classify(text="I love this product!")
print(f"Anthropic says: {result.sentiment}")

# --- Try Google ---
google_lm = dspy.LM("gemini/gemini-2.0-flash", temperature=0.0)
dspy.configure(lm=google_lm)

result = classify(text="I love this product!")
print(f"Google says: {result.sentiment}")

# --- Together AI (open-source) ---
together_lm = dspy.LM("together_ai/meta-llama/Llama-3-70b-chat-hf", temperature=0.0)
dspy.configure(lm=together_lm)

result = classify(text="I love this product!")
print(f"Llama 3 says: {result.sentiment}")

What to notice

• The provider string format is always "provider/model-name".
• Your DSPy modules and signatures stay exactly the same across providers.
• Set the API key for each provider as an environment variable (OPENAI_API_KEY, ANTHROPIC_API_KEY, TOGETHER_API_KEY, etc.).
• temperature=0.0 ensures deterministic outputs for testing.

Example 2: Cost optimization with per-module LM routing

Use an expensive model for tasks that need it and a cheap model for everything else. This pipeline classifies support tickets (easy -- use cheap model) and then generates a detailed response (hard -- use expensive model).

import dspy
from typing import Literal

# --- Define two LMs at different price points ---
cheap_lm = dspy.LM("openai/gpt-4o-mini", temperature=0.0)    # ~$0.15/1M input tokens
expensive_lm = dspy.LM("openai/gpt-4o", temperature=0.0)      # ~$2.50/1M input tokens

# Set the cheap model as the default
dspy.configure(lm=cheap_lm)

# --- Signatures ---
class ClassifyTicket(dspy.Signature):
    """Classify a support ticket into a category."""
    ticket_text: str = dspy.InputField()
    urgency: Literal["low", "medium", "high", "critical"] = dspy.OutputField()
    category: Literal["billing", "technical", "account", "feature_request"] = dspy.OutputField()

class DraftResponse(dspy.Signature):
    """Draft a helpful, empathetic response to a support ticket."""
    ticket_text: str = dspy.InputField()
    urgency: str = dspy.InputField()
    category: str = dspy.InputField()
    response: str = dspy.OutputField(desc="A helpful reply to the customer, 2-4 sentences")

# --- Pipeline with mixed models ---
class SupportPipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict(ClassifyTicket)
        self.draft = dspy.ChainOfThought(DraftResponse)

    def forward(self, ticket_text):
        classification = self.classify(ticket_text=ticket_text)
        return self.draft(
            ticket_text=ticket_text,
            urgency=classification.urgency,
            category=classification.category,
        )

pipeline = SupportPipeline()

# Route: cheap model classifies, expensive model drafts responses
pipeline.classify.set_lm(cheap_lm)
pipeline.draft.set_lm(expensive_lm)

# --- Use it ---
result = pipeline(ticket_text="I've been charged twice for my subscription and I need a refund ASAP")
print(f"Response: {result.response}")

# --- Alternative: use dspy.context for temporary overrides ---
# Everything uses cheap_lm by default, but you can override per-call:
with dspy.context(lm=expensive_lm):
    important_result = pipeline(ticket_text="Our production system is down")

What to notice

• set_lm() is permanent for that module instance -- every call to pipeline.classify uses cheap_lm.
• dspy.context(lm=...) is temporary -- it only applies inside the with block.
• Classification is a simple routing task where a cheap model is sufficient. Response drafting benefits from a more capable model.
• You can mix and match: use set_lm() for the common case and dspy.context() for exceptions.

Example 3: Local model setup with Ollama and vLLM

Run models on your own hardware for data privacy, zero API costs, or offline use.

Option A: Ollama (easiest local setup)

# Install Ollama: https://ollama.ai
# Pull a model
ollama pull llama3.1
ollama pull mistral

# Ollama serves on port 11434 by default
ollama serve

import dspy

# Connect to Ollama
lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
    temperature=0.7,
    max_tokens=1000,
)
dspy.configure(lm=lm)

# Use it like any other LM
classify = dspy.ChainOfThought("text -> category: str")
result = classify(text="The API keeps returning 500 errors")
print(result.category)

# Switch to a different local model
mistral_lm = dspy.LM(
    "ollama_chat/mistral",
    api_base="http://localhost:11434",
    temperature=0.7,
)
dspy.configure(lm=mistral_lm)

Option B: vLLM (high-performance serving)

# Install vLLM
pip install vllm

# Serve a model with OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8b-chat-hf \
    --port 8000

import dspy

# Connect to vLLM using the OpenAI-compatible provider
lm = dspy.LM(
    "openai/meta-llama/Llama-3-8b-chat-hf",
    api_base="http://localhost:8000/v1",
    api_key="none",  # vLLM doesn't need a real key
    temperature=0.7,
    max_tokens=1000,
)
dspy.configure(lm=lm)

# Works exactly like a cloud model
summarize = dspy.ChainOfThought("document -> summary")
result = summarize(document="DSPy is a framework for programming language models...")
print(result.summary)

Option C: Mix local and cloud models

Use a local model for cheap/private tasks and a cloud model for quality-critical tasks:

import dspy

# Local model for classification (free, private)
local_lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
)

# Cloud model for generation (better quality)
cloud_lm = dspy.LM("openai/gpt-4o")

# Default to local
dspy.configure(lm=local_lm)

class HybridPipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict("text -> category: str")
        self.generate = dspy.ChainOfThought("text, category -> response: str")

    def forward(self, text):
        classification = self.classify(text=text)
        return self.generate(text=text, category=classification.category)

pipeline = HybridPipeline()
pipeline.classify.set_lm(local_lm)    # Runs locally -- free, data stays on your machine
pipeline.generate.set_lm(cloud_lm)    # Cloud model for quality generation

result = pipeline(text="How do I reset my password?")
print(result.response)

What to notice

• Ollama is the easiest way to run models locally. Use "ollama_chat/model-name" with api_base.
• vLLM gives higher throughput for production. Use "openai/model-name" with api_base since vLLM exposes an OpenAI-compatible API.
• Any server that implements the OpenAI /v1/chat/completions endpoint works with the "openai/..." provider string.
• Local models benefit the most from DSPy optimization -- run /ai-improving-accuracy to get better results from smaller models.
• You can freely mix local and cloud models in the same pipeline using set_lm().

Condensed from dspy.ai/api/models/LM/. Verify against upstream for latest.

dspy.LM — API Reference

Constructor

dspy.LM(
    model,
    model_type="chat",
    temperature=None,
    max_tokens=None,
    cache=True,
    callbacks=None,
    num_retries=3,
    provider=None,
    finetuning_model=None,
    launch_kwargs=None,
    train_kwargs=None,
    use_developer_role=False,
    **kwargs,
)

Key methods

Calling the LM

Both prompt (string) and messages (list of dicts with role/content) formats are supported. Returns a list of strings.

Configuration and state

Fine-tuning

Global configuration

# Set the default LM for all modules
dspy.configure(lm=lm)

# Temporary LM override
with dspy.context(lm=other_lm):
    result = module(...)

# Configure caching
dspy.configure_cache(enable=True)

# View recent LM history
dspy.inspect_history(n=1)

Provider string format

The model string follows LiteLLM's "provider/model-name" convention:

Key behaviors

• Caching: Enabled by default. Same prompt + same params + same model returns cached result with no API call. Use rollout_id parameter to bypass cache for stochastic sampling without disabling future caching.
• History: Auto-tracked unless settings.disable_history is set. Accessible via inspect_history().
• API keys: Read from environment variables automatically via LiteLLM. Never serialized in dump_state().