Dspy Weave

W&B Weave — Cloud Observability & Experiment Tracking for DSPy

Guide the user through setting up W&B Weave for tracing DSPy calls, tracking optimization experiments, and collaborating with team dashboards.

What is W&B Weave

Weave is Weights & Biases' LLM observability and experiment tracking product. It provides cloud-hosted dashboards for tracing function calls, comparing optimization runs, and sharing results across teams.

• Cloud-hosted: Dashboards at wandb.ai
• Manual instrumentation: Uses @weave.op() decorator (not auto-instrument like Langtrace)
• Team collaboration: shared projects, comments, and run comparisons

Key difference from Langtrace/Phoenix

Weave uses a manual decorator (@weave.op()) — you choose which functions to trace. Langtrace and Phoenix auto-instrument all DSPy calls. This gives Weave more control over what gets tracked but requires more setup.

When to use Weave

Use Weave when:

• Your team already uses W&B for ML experiments
• You want cloud-hosted dashboards with team collaboration
• You want to track and compare optimization runs side-by-side
• You need fine-grained control over which functions are traced

Do NOT use Weave when:

• You want auto-instrumentation with zero code changes — see /dspy-langtrace
• You want a free, local-only trace viewer — see /dspy-phoenix
• You need the full ML lifecycle (model registry, deployment) — see /dspy-mlflow
• You're a solo developer who doesn't need team features — Langtrace or Phoenix is simpler

Setup

Install

pip install weave

Initialize

import weave

weave.init("my-dspy-project")  # Creates project at wandb.ai

# You'll be prompted to log in on first run
# Or set WANDB_API_KEY environment variable

Environment variable configuration

export WANDB_API_KEY="your-key"          # From wandb.ai/settings
export WANDB_ENTITY="your-team"          # Optional: team name
export WANDB_PROJECT="my-dspy-project"   # Optional: project name

Tracing with @weave.op()

The @weave.op() decorator traces a function's inputs, outputs, latency, and cost:

import weave
import dspy

weave.init("my-dspy-project")
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))  # or "anthropic/claude-sonnet-4-5-20250929", etc.

class QABot(dspy.Module):
    def __init__(self):
        self.retrieve = dspy.Retrieve(k=3)
        self.answer = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        context = self.retrieve(question).passages
        return self.answer(context=context, question=question)

bot = QABot()

@weave.op()
def handle_question(question: str) -> str:
    """Traced by Weave — inputs, outputs, and latency logged."""
    result = bot(question=question)
    return result.answer

# Every call is tracked at wandb.ai
answer = handle_question("How do refunds work?")

Tracing multiple functions

Decorate each function you want to trace:

@weave.op()
def retrieve_context(question: str) -> list[str]:
    return dspy.Retrieve(k=3)(question).passages

@weave.op()
def generate_answer(context: list[str], question: str) -> str:
    cot = dspy.ChainOfThought("context, question -> answer")
    return cot(context=context, question=question).answer

@weave.op()
def handle_question(question: str) -> str:
    context = retrieve_context(question)
    return generate_answer(context, question)

# Weave shows the call tree: handle_question -> retrieve_context, generate_answer

Tracking optimization experiments

Weave excels at comparing optimization runs. Wrap your optimization in @weave.op():

import weave
import dspy
from dspy.evaluate import Evaluate

weave.init("optimization-experiments")
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))  # or "anthropic/claude-sonnet-4-5-20250929", etc.

@weave.op()
def run_optimization(optimizer_name: str, model: str, auto_setting: str):
    """Run and track an optimization experiment."""
    lm = dspy.LM(model)
    dspy.configure(lm=lm)

    program = dspy.ChainOfThought("question -> answer")

    if optimizer_name == "miprov2":
        optimizer = dspy.MIPROv2(metric=metric, auto=auto_setting)
    elif optimizer_name == "bootstrap":
        optimizer = dspy.BootstrapFewShot(metric=metric)

    optimized = optimizer.compile(program, trainset=trainset)

    evaluator = Evaluate(devset=devset, metric=metric, num_threads=4)
    score = evaluator(optimized)

    # Save the artifact
    path = f"experiments/{optimizer_name}_{model}_{auto_setting}.json"
    optimized.save(path)

    return {
        "score": score,
        "optimizer": optimizer_name,
        "model": model,
        "auto": auto_setting,
        "artifact_path": path,
    }

# Run experiments — each is tracked in Weave
run_optimization("miprov2", "openai/gpt-4o-mini", "light")
run_optimization("miprov2", "openai/gpt-4o-mini", "medium")
run_optimization("bootstrap", "openai/gpt-4o-mini", "n/a")

Comparing runs in the W&B dashboard

1. Go to wandb.ai and open your project 2. Click on the "Traces" tab to see all tracked calls 3. Compare inputs and outputs across runs 4. Sort by score to find the best experiment 5. Share the dashboard URL with your team

Weave vs Langtrace vs Phoenix

Decision guide

Want DSPy observability?
|
+- Team already uses W&B? -> Weave
+- Want auto-instrumentation (no decorators)? -> Langtrace (/dspy-langtrace)
+- Want local-only + built-in evals? -> Phoenix (/dspy-phoenix)
+- Need full ML lifecycle (registry, deploy)? -> MLflow (/dspy-mlflow)

Verifying the setup

After initializing Weave and adding @weave.op() decorators, run one traced call and confirm it appears in the dashboard:

# Quick smoke test
@weave.op()
def smoke_test(x: str) -> str:
    return x.upper()

result = smoke_test("hello")
print(f"Check your project at https://wandb.ai — look for the smoke_test call")

If the call does not appear: check WANDB_API_KEY is set, confirm weave.init() was called before the decorated function, and verify network access to wandb.ai.

Gotchas

• Claude puts `@weave.op()` on the DSPy module class instead of the calling function. Weave decorators trace regular functions, not DSPy module classes. Decorate the function that calls the module, not the module itself. @weave.op() goes on handle_question(), not on QABot.
• Claude calls `weave.init()` inside a function instead of at module level. weave.init() must run once at startup, before any @weave.op() decorated functions are called. Placing it inside a request handler creates a new project per call and fragments your traces.
• Claude forgets to set `WANDB_API_KEY` in deployment environments. Local development prompts for login interactively, but production (Docker, CI, serverless) needs the environment variable explicitly set. Always include WANDB_API_KEY in environment configuration for non-local setups.
• Claude auto-instruments everything instead of using selective decorators. Unlike Langtrace/Phoenix, Weave traces only what you decorate. Claude sometimes tries to add a global "trace all DSPy calls" setup that does not exist. Each function needs its own @weave.op() decorator.
• Claude nests `@weave.op()` and DSPy decorators incorrectly. If combining with other decorators, @weave.op() should be the outermost decorator so it captures the full function execution including any inner decorator behavior.

Additional resources

• W&B Weave docs
• weave.op() reference
• W&B dashboard
• For API details, see reference.md
• For worked examples, see examples.md

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Langtrace (auto-instrumentation, easiest setup) — /dspy-langtrace
• Arize Phoenix (open-source with evals) — /dspy-phoenix
• MLflow (full ML lifecycle) — /dspy-mlflow
• Aggregate monitoring — /ai-monitoring
• Experiment tracking patterns (JSONL-based, lightweight) — /ai-tracking-experiments
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Weave Examples

Track optimization experiments and compare runs

Setup

import weave
import dspy
from dspy.evaluate import Evaluate

weave.init("dspy-experiments")
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

# Shared data and metric
trainset = [
    dspy.Example(question="What is Python?", answer="A programming language.").with_inputs("question"),
    # ... 50+ examples
]
devset = trainset[:20]

def metric(example, prediction, trace=None):
    judge = dspy.Predict("gold_answer, predicted_answer -> match: bool")
    result = judge(gold_answer=example.answer, predicted_answer=prediction.answer)
    return result.match

Run multiple experiments

@weave.op()
def experiment(name: str, optimizer_type: str, auto: str = "light"):
    """Each call creates a tracked run in Weave."""
    program = dspy.ChainOfThought("question -> answer")

    if optimizer_type == "miprov2":
        optimizer = dspy.MIPROv2(metric=metric, auto=auto)
        optimized = optimizer.compile(program, trainset=trainset)
    elif optimizer_type == "bootstrap":
        optimizer = dspy.BootstrapFewShot(metric=metric)
        optimized = optimizer.compile(program, trainset=trainset)
    elif optimizer_type == "gepa":
        def gepa_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
            score = float(metric(gold, pred))
            return {"score": score, "feedback": "" if score else "Wrong answer."}
        optimizer = dspy.GEPA(metric=gepa_metric, auto=auto)
        optimized = optimizer.compile(program, trainset=trainset)

    evaluator = Evaluate(devset=devset, metric=metric, num_threads=4)
    score = evaluator(optimized)

    optimized.save(f"experiments/{name}.json")
    return {"name": name, "score": score, "optimizer": optimizer_type, "auto": auto}

# Run experiments
experiment("baseline-bootstrap", "bootstrap")
experiment("mipro-light", "miprov2", "light")
experiment("mipro-medium", "miprov2", "medium")
experiment("gepa-light", "gepa", "light")

Compare in W&B dashboard

1. Go to wandb.ai → your project → "Traces" tab 2. You'll see 4 tracked function calls with their inputs and outputs 3. Click each to see:

• Input parameters (optimizer type, auto setting)
• Output (score, artifact path)
• Latency (how long optimization took)
• Cost (token usage)

4. Sort by output score to find the winner 5. Click "Share" to send the dashboard URL to your team

Track production queries with metadata

import weave
import dspy

weave.init("production-qa")
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

program = dspy.ChainOfThought("question -> answer")
program.load("experiments/mipro-medium.json")

@weave.op()
def handle_query(user_id: str, question: str, source: str = "api"):
    """Production query handler — every call tracked in Weave."""
    result = program(question=question)
    return {
        "answer": result.answer,
        "user_id": user_id,
        "source": source,
    }

# Production usage
handle_query("user-42", "What's the return policy?", source="web")
handle_query("user-99", "How do I upgrade?", source="mobile")

# In the Weave dashboard:
# - Filter by source to compare web vs mobile usage
# - Sort by latency to find slow queries
# - Click into a trace to see the full LM prompt/response

W&B Weave API Reference

Condensed from docs.wandb.ai/weave. Verify against upstream for latest.

Setup

pip install weave

import weave
weave.init("project-name")  # creates project at wandb.ai

Set WANDB_API_KEY env var for non-interactive auth.

weave.init

weave.init(project_name: str)

Initializes Weave tracing for a project. Call once at startup, before any @weave.op() calls.

@weave.op()

@weave.op()
def my_function(x: str) -> str:
    ...

Decorator that traces a function's inputs, outputs, latency, and cost. Apply to regular functions, not DSPy module classes.

Key behavior:

• Nested @weave.op() calls create a call tree in the dashboard
• Must be the outermost decorator when combined with others
• Each function needs its own decorator (no global auto-instrumentation)

Environment Variables

Dashboard

View traces at wandb.ai under your project's "Traces" tab. Compare runs side-by-side, sort by score, share with team.