Dspy Ollama

Ollama — Run DSPy with Local Models

Guide the user through running DSPy with local models via Ollama. No API keys, no cloud costs, full privacy.

What is Ollama

Ollama is a local LLM runner (166k+ GitHub stars) that wraps llama.cpp. It downloads, manages, and serves models locally with a simple CLI. DSPy connects to it through LiteLLM's ollama_chat/ provider.

Setup

Install Ollama

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download from ollama.com

Start the server and pull a model

# Start the Ollama server (runs in background)
ollama serve

# Pull a model (one-time download)
ollama pull llama3.1

# Quick test
ollama run llama3.1 "What is DSPy?"

Connect DSPy to Ollama

import dspy

lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
    api_key="",  # required but ignored
    temperature=0.7,
    num_ctx=8192,  # IMPORTANT: set context window explicitly
)
dspy.configure(lm=lm)

# Test it
classify = dspy.Predict("text -> sentiment")
result = classify(text="DSPy makes AI development easier")
print(result.sentiment)

Note: dspy.OllamaLocal is deprecated. Use dspy.LM("ollama_chat/...") instead.

Model selection guide

Quick recommendations

Prototyping (fast iteration, good quality):
  → llama3.1:8b or qwen2.5:7b

Best quality on consumer hardware (16GB+ RAM):
  → llama3.1:8b or phi4:14b

Complex reasoning:
  → deepseek-r1:14b or qwen3:14b

Coding tasks:
  → qwen2.5-coder:7b or codellama:13b

Minimal resources (8GB RAM):
  → llama3.2:3b or gemma2:2b or qwen2.5:3b

Context window gotcha (critical)

Ollama defaults to 4096 tokens regardless of the model's actual capacity. This is the #1 source of issues when running DSPy with Ollama. DSPy prompts with few-shot demos can easily exceed 4096 tokens.

Always set num_ctx explicitly:

# BAD — defaults to 4096 tokens, will silently truncate
lm = dspy.LM("ollama_chat/llama3.1", api_base="http://localhost:11434", api_key="")

# GOOD — set context window to match model capability
lm = dspy.LM(
    "ollama_chat/llama3.1",
    api_base="http://localhost:11434",
    api_key="",
    num_ctx=8192,  # 8K is a safe default for most tasks
)

Larger context = more VRAM. If you get OOM errors, reduce num_ctx:

Performance tuning

GPU acceleration

Ollama automatically uses GPU if available. Check with:

ollama ps  # shows which models are loaded and GPU usage

Control GPU usage with environment variables:

# Use all GPU layers (default if GPU detected)
export OLLAMA_NUM_GPU=999

# CPU only (useful for testing or shared machines)
export OLLAMA_NUM_GPU=0

# Partial offload (when model doesn't fully fit in VRAM)
export OLLAMA_NUM_GPU=20

Concurrent requests

# Allow multiple parallel requests (default: 1)
export OLLAMA_NUM_PARALLEL=4

# Keep multiple models loaded (for multi-model pipelines)
export OLLAMA_MAX_LOADED_MODELS=2

Apple Silicon optimization

Ollama runs natively on Apple Silicon using Metal. Performance tips:

• M1/M2 (8GB): 8B models work well with num_ctx=4096
• M1/M2 Pro (16GB): 8B models with num_ctx=8192, or 14B with num_ctx=4096
• M1/M2 Max (32GB+): 70B quantized models with num_ctx=4096
• M3/M4 Max (64GB+): 70B models with num_ctx=8192

Per-module model assignment

Use a big model for hard tasks and a small model for simple ones:

import dspy

big = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434",
              api_key="", num_ctx=8192)
small = dspy.LM("ollama_chat/llama3.2:3b", api_base="http://localhost:11434",
                api_key="", num_ctx=4096)

dspy.configure(lm=small)  # default: cheap model

class Pipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict("text -> category")
        self.analyze = dspy.ChainOfThought("text, category -> analysis")

    def forward(self, text):
        cat = self.classify(text=text)
        return self.analyze(text=text, category=cat.category)

pipeline = Pipeline()
pipeline.classify.set_lm(small)   # simple task → small model
pipeline.analyze.set_lm(big)      # complex task → big model

Running DSPy optimization with Ollama

Optimization works with local models but is significantly slower than cloud APIs. Tips:

import dspy

lm = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434",
             api_key="", num_ctx=8192)
dspy.configure(lm=lm)

# Tip 1: Start with BootstrapFewShot (fastest optimizer)
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized = optimizer.compile(program, trainset=trainset)

# Tip 2: For MIPROv2, use auto="light" (fewest trials)
optimizer = dspy.MIPROv2(metric=metric, auto="light")
optimized = optimizer.compile(program, trainset=trainset)

# Tip 3: Use a bigger model as teacher, smaller as student
teacher_lm = dspy.LM("ollama_chat/llama3.1:70b", api_base="http://localhost:11434",
                     api_key="", num_ctx=8192)
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
with dspy.context(lm=teacher_lm):
    optimized = optimizer.compile(program, trainset=trainset)
# Deploy optimized program with the smaller model

Expect hours, not minutes for optimization with local models. A MIPROv2 auto="medium" run that takes 5 minutes with GPT-4o-mini might take 2-4 hours with a local 8B model.

Ollama vs vLLM

Recommended workflow: Develop with Ollama locally, deploy with vLLM in production. The DSPy code is identical — only the LM config line changes:

# Development (Ollama)
lm = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434", api_key="")

# Production (vLLM)
lm = dspy.LM("openai/meta-llama/Llama-3.1-8B-Instruct", api_base="http://gpu-server:8000/v1", api_key="none")

Gotchas

1. Context window defaults to 4096 — always set num_ctx explicitly. DSPy optimized prompts with few-shot demos easily exceed 4096 tokens. 2. `api_key=""` is required — even though Ollama doesn't use it, LiteLLM requires the parameter. 3. First request is slow — Ollama loads the model into memory on the first call. Subsequent calls are fast. 4. OOM errors — reduce num_ctx or switch to a smaller model. Check VRAM with ollama ps. 5. `dspy.OllamaLocal` is deprecated — use dspy.LM("ollama_chat/...") instead.

Cross-references

• LM configuration basics (providers, parameters, caching) — /dspy-lm
• Production serving with vLLM — /dspy-vllm
• Reducing costs (model routing, caching) — /ai-cutting-costs
• Switching models without breaking things — /ai-switching-models
• For worked examples, see examples.md

Ollama Examples

Set up Ollama for a DSPy project

# Install and start
brew install ollama  # or: curl -fsSL https://ollama.com/install.sh | sh
ollama serve &

# Pull a model
ollama pull llama3.1:8b

# Verify it's running
ollama list

import dspy

# Configure DSPy with Ollama
lm = dspy.LM(
    "ollama_chat/llama3.1:8b",
    api_base="http://localhost:11434",
    api_key="",
    num_ctx=8192,
)
dspy.configure(lm=lm)

# Test with a simple task
classify = dspy.Predict("email -> category: str, priority: str")
result = classify(email="Server is down, customers can't log in!")
print(f"Category: {result.category}, Priority: {result.priority}")

Optimize a classifier with a local model

import dspy

lm = dspy.LM(
    "ollama_chat/llama3.1:8b",
    api_base="http://localhost:11434",
    api_key="",
    num_ctx=8192,
)
dspy.configure(lm=lm)

# Training data
trainset = [
    dspy.Example(text="Server is down!", label="critical").with_inputs("text"),
    dspy.Example(text="Can't reset password", label="account").with_inputs("text"),
    dspy.Example(text="How much is the pro plan?", label="billing").with_inputs("text"),
    dspy.Example(text="Feature request: dark mode", label="feature").with_inputs("text"),
    dspy.Example(text="API returns 500 errors", label="critical").with_inputs("text"),
    dspy.Example(text="Invoice is wrong", label="billing").with_inputs("text"),
    dspy.Example(text="Want to cancel subscription", label="account").with_inputs("text"),
    dspy.Example(text="App crashes on iOS 17", label="critical").with_inputs("text"),
]

def metric(example, prediction, trace=None):
    return prediction.label.strip().lower() == example.label.strip().lower()

# Use BootstrapFewShot — fastest optimizer for local models
program = dspy.Predict("text -> label")
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=3)
optimized = optimizer.compile(program, trainset=trainset)

# Test
result = optimized(text="Database migration failed")
print(f"Label: {result.label}")

# Save for later
optimized.save("optimized_classifier.json")

Multi-model pipeline (big + small)

import dspy

# Small model for simple tasks
small = dspy.LM(
    "ollama_chat/llama3.2:3b",
    api_base="http://localhost:11434",
    api_key="",
    num_ctx=4096,
)

# Big model for complex reasoning
big = dspy.LM(
    "ollama_chat/llama3.1:8b",
    api_base="http://localhost:11434",
    api_key="",
    num_ctx=8192,
)

dspy.configure(lm=small)

class SupportPipeline(dspy.Module):
    def __init__(self):
        self.classify = dspy.Predict("ticket -> category, priority")
        self.draft = dspy.ChainOfThought("ticket, category, priority -> response")

    def forward(self, ticket):
        triage = self.classify(ticket=ticket)
        return self.draft(
            ticket=ticket,
            category=triage.category,
            priority=triage.priority,
        )

pipeline = SupportPipeline()
pipeline.classify.set_lm(small)  # fast classification
pipeline.draft.set_lm(big)       # better response quality

result = pipeline(ticket="I was charged twice for my subscription last month")
print(f"Category: {result.category}")
print(f"Priority: {result.priority}")
print(f"Response: {result.response}")