Dspy Ragas

Ragas — Decomposed RAG Evaluation for DSPy

Guide the user through evaluating DSPy RAG pipelines with Ragas, an evaluation framework that decomposes RAG quality into independent metrics for retriever and generator.

What is Ragas

Ragas is an open-source evaluation framework (12.9k+ GitHub stars, Apache 2.0) purpose-built for RAG pipelines. Instead of a single accuracy score, it breaks evaluation into decomposed metrics:

This decomposition tells you where your RAG pipeline fails — retriever or generator — so you know what to fix.

When to use Ragas vs dspy.Evaluate

Best practice: Use dspy.Evaluate with a fast metric (SemanticF1) as your optimization objective, then use Ragas for post-optimization analysis to understand why your pipeline performs the way it does.

Setup

# Core Ragas (evaluation only)
pip install ragas

# With DSPy optimizer support (uses MIPROv2 internally)
pip install "ragas[dspy]"

Ragas requires an LLM for its metrics. By default it uses OpenAI (OPENAI_API_KEY), but you can configure any LLM via LangChain wrappers.

Evaluating a DSPy RAG pipeline with Ragas

Step 1: Collect predictions from your DSPy pipeline

Run your DSPy RAG pipeline on a set of questions and collect the inputs, retrieved contexts, and generated answers:

import dspy

# Your DSPy RAG pipeline
class RAG(dspy.Module):
    def __init__(self, retriever):
        self.retrieve = retriever
        self.generate = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        context = self.retrieve(question).passages
        return dspy.Prediction(
            answer=self.generate(context=context, question=question).answer,
            context=context,
        )

# Collect predictions
results = []
for example in devset:
    pred = rag(question=example.question)
    results.append({
        "user_input": example.question,
        "response": pred.answer,
        "retrieved_contexts": pred.context,
        "reference": example.answer,  # ground truth, if available
    })

Step 2: Build a Ragas EvaluationDataset

from ragas import EvaluationDataset, SingleTurnSample

samples = [
    SingleTurnSample(
        user_input=r["user_input"],
        response=r["response"],
        retrieved_contexts=r["retrieved_contexts"],
        reference=r.get("reference"),  # optional for some metrics
    )
    for r in results
]
dataset = EvaluationDataset(samples=samples)

Step 3: Run evaluation

from ragas import evaluate
from ragas.metrics import (
    Faithfulness,
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
    AnswerCorrectness,
)

# Pick metrics based on what you have
# Without ground truth: Faithfulness + AnswerRelevancy
# With ground truth: add ContextPrecision, ContextRecall, AnswerCorrectness
result = evaluate(
    dataset=dataset,
    metrics=[
        Faithfulness(),
        AnswerRelevancy(),
        ContextPrecision(),
        ContextRecall(),
        AnswerCorrectness(),
    ],
)

print(result)
# {'faithfulness': 0.87, 'answer_relevancy': 0.92, 'context_precision': 0.75,
#  'context_recall': 0.68, 'answer_correctness': 0.81}

Step 4: Interpret results

Faithfulness low (< 0.8)?
  → Generator is hallucinating beyond retrieved context
  → Fix: add assertions, use GroundedRAG pattern (/ai-stopping-hallucinations)

ContextPrecision low (< 0.7)?
  → Retriever returns relevant docs but ranks them poorly
  → Fix: tune k, try hybrid search, re-rank (/dspy-qdrant)

ContextRecall low (< 0.7)?
  → Retriever misses relevant documents entirely
  → Fix: improve chunking, add more docs, try different embeddings (/ai-searching-docs)

AnswerRelevancy low (< 0.8)?
  → Generator answers don't address the question
  → Fix: improve signatures, optimize with MIPROv2 (/dspy-miprov2)

AnswerCorrectness low but Faithfulness high?
  → Generator is faithful to context but context is wrong
  → Focus on retriever improvements

Using a custom LLM with Ragas

By default Ragas uses OpenAI. To use another provider:

from ragas.llms import LangchainLLMWrapper
from langchain_anthropic import ChatAnthropic

evaluator_llm = LangchainLLMWrapper(ChatAnthropic(model="claude-sonnet-4-5-20250929"))

result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), AnswerRelevancy()],
    llm=evaluator_llm,
)

Per-sample scores

Get scores for each sample to find problem areas:

result = evaluate(dataset=dataset, metrics=[Faithfulness(), ContextRecall()])

# Convert to pandas DataFrame
df = result.to_pandas()
print(df[["user_input", "faithfulness", "context_recall"]])

# Find worst-performing samples
worst = df.nsmallest(5, "faithfulness")
for _, row in worst.iterrows():
    print(f"Q: {row['user_input']}")
    print(f"  Faithfulness: {row['faithfulness']:.2f}")

DSPyOptimizer (advanced)

Ragas includes a DSPyOptimizer that uses MIPROv2 internally to optimize Ragas's own metric prompts. This can improve evaluation accuracy for domain-specific data.

pip install "ragas[dspy]"

from ragas.metrics import Faithfulness
from ragas.integrations.dspy import DSPyOptimizer

# Optimize the Faithfulness metric's internal prompts
metric = Faithfulness()
optimizer = DSPyOptimizer(metric=metric)

# Requires a labeled dataset where you know the correct faithfulness scores
optimized_metric = optimizer.optimize(dataset=labeled_eval_dataset)

# Use the optimized metric for more accurate evaluation
result = evaluate(dataset=dataset, metrics=[optimized_metric])

This is advanced — only needed if Ragas's default metrics don't align well with your domain's definition of faithfulness, relevancy, etc.

Ragas in a DSPy development workflow

1. Build RAG pipeline          → /ai-searching-docs or /dspy-retrieval
2. Create devset               → /dspy-data
3. Evaluate with dspy.Evaluate → /dspy-evaluate (SemanticF1 as optimization target)
4. Optimize with MIPROv2       → /dspy-miprov2
5. Deep analysis with Ragas    → this skill (diagnose retriever vs generator)
6. Fix weak components         → /ai-stopping-hallucinations, /dspy-qdrant, /ai-improving-accuracy
7. Re-evaluate with both       → confirm improvements

Gotchas

1. Ragas metrics call an LLM — each metric makes multiple LLM calls per sample. A 100-sample evaluation with 5 metrics = ~500 LLM calls. Budget for the cost. 2. Don't use Ragas as an optimizer objective — it's too slow for inner-loop optimization. Use DSPy's built-in metrics for compile(), then Ragas for analysis. 3. ContextPrecision and ContextRecall need ground truth — if you don't have reference answers, use Faithfulness + AnswerRelevancy (reference-free). 4. Ragas v0.2+ changed the API — if you find old examples using Dataset from datasets, update to EvaluationDataset and SingleTurnSample.

Cross-references

• DSPy's built-in evaluation (SemanticF1, exact match, LM-as-judge) — /dspy-evaluate
• Building RAG pipelines — /ai-searching-docs
• Retrieval modules and vector DBs — /dspy-retrieval, /dspy-qdrant
• Stopping hallucinations (when Faithfulness is low) — /ai-stopping-hallucinations
• Optimizing RAG accuracy — /ai-improving-accuracy, /dspy-miprov2
• For worked examples, see examples.md

Ragas Examples

Evaluate a support bot RAG pipeline

import dspy
from ragas import evaluate, EvaluationDataset, SingleTurnSample
from ragas.metrics import (
    Faithfulness,
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
)

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

# Your support bot RAG pipeline
class SupportBot(dspy.Module):
    def __init__(self, retriever):
        self.retrieve = retriever
        self.answer = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        passages = self.retrieve(question).passages
        result = self.answer(context=passages, question=question)
        return dspy.Prediction(answer=result.answer, context=passages)

# Set up retriever (using FAISS for this example)
embedder = dspy.Embedder("openai/text-embedding-3-small", dimensions=512)
corpus = [
    "Refunds are processed within 5-7 business days.",
    "To reset your password, go to Settings > Security > Reset Password.",
    "Enterprise plans include SSO, SAML, and dedicated support.",
    "Free trial lasts 14 days with full feature access.",
    "Billing is monthly. Annual plans get 20% discount.",
]
search = dspy.retrievers.Embeddings(embedder=embedder, corpus=corpus, k=3)
bot = SupportBot(retriever=search)

# Test questions with ground truth
test_data = [
    {"question": "How long do refunds take?", "answer": "5-7 business days"},
    {"question": "How do I reset my password?", "answer": "Go to Settings > Security > Reset Password"},
    {"question": "What's included in enterprise?", "answer": "SSO, SAML, and dedicated support"},
    {"question": "How long is the free trial?", "answer": "14 days with full features"},
]

# Collect predictions
samples = []
for item in test_data:
    pred = bot(question=item["question"])
    samples.append(SingleTurnSample(
        user_input=item["question"],
        response=pred.answer,
        retrieved_contexts=pred.context,
        reference=item["answer"],
    ))

dataset = EvaluationDataset(samples=samples)

# Run Ragas evaluation
result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), AnswerRelevancy(), ContextPrecision(), ContextRecall()],
)

print("Ragas Evaluation Results:")
print(f"  Faithfulness:      {result['faithfulness']:.2f}")
print(f"  Answer Relevancy:  {result['answer_relevancy']:.2f}")
print(f"  Context Precision: {result['context_precision']:.2f}")
print(f"  Context Recall:    {result['context_recall']:.2f}")

# Interpret: if context_recall is low, the retriever is missing relevant docs
# If faithfulness is low, the generator is hallucinating beyond the context

Diagnose retriever vs generator issues

import pandas as pd
from ragas import evaluate, EvaluationDataset, SingleTurnSample
from ragas.metrics import Faithfulness, ContextRecall, ContextPrecision

# After collecting samples from your pipeline...
result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), ContextRecall(), ContextPrecision()],
)

df = result.to_pandas()

# Find samples where retriever fails (low context recall)
retriever_failures = df[df["context_recall"] < 0.5]
print(f"\nRetriever failures ({len(retriever_failures)} samples):")
for _, row in retriever_failures.iterrows():
    print(f"  Q: {row['user_input']}")
    print(f"     Context Recall: {row['context_recall']:.2f}")

# Find samples where generator fails (low faithfulness despite good retrieval)
generator_failures = df[(df["faithfulness"] < 0.5) & (df["context_recall"] >= 0.7)]
print(f"\nGenerator failures ({len(generator_failures)} samples):")
for _, row in generator_failures.iterrows():
    print(f"  Q: {row['user_input']}")
    print(f"     Faithfulness: {row['faithfulness']:.2f}")
    print(f"     Context Recall: {row['context_recall']:.2f}")

# Summary diagnosis
avg_ctx_recall = df["context_recall"].mean()
avg_faith = df["faithfulness"].mean()

if avg_ctx_recall < 0.7:
    print("\n→ RETRIEVER is the bottleneck. Improve chunking, embeddings, or k.")
    print("  Try: /ai-searching-docs or /dspy-qdrant")
elif avg_faith < 0.8:
    print("\n→ GENERATOR is the bottleneck. Improve grounding or optimize prompts.")
    print("  Try: /ai-stopping-hallucinations or /dspy-miprov2")
else:
    print("\n→ Pipeline looks healthy. Focus on edge cases.")

Compare before and after optimization

import dspy
from dspy.evaluate import Evaluate
from ragas import evaluate as ragas_evaluate, EvaluationDataset, SingleTurnSample
from ragas.metrics import Faithfulness, AnswerRelevancy, ContextRecall

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

# --- Baseline ---
baseline_rag = RAG(retriever=search)

# DSPy metric (fast, used for optimization)
from dspy.evaluate import SemanticF1
dspy_metric = SemanticF1()
evaluator = Evaluate(devset=devset, metric=dspy_metric, num_threads=4)
baseline_dspy_score = evaluator(baseline_rag)

# Ragas metrics (slow, used for analysis)
baseline_samples = collect_ragas_samples(baseline_rag, devset)
baseline_ragas = ragas_evaluate(
    dataset=EvaluationDataset(samples=baseline_samples),
    metrics=[Faithfulness(), AnswerRelevancy(), ContextRecall()],
)

# --- Optimize ---
optimizer = dspy.MIPROv2(metric=dspy_metric, auto="medium")
optimized_rag = optimizer.compile(baseline_rag, trainset=trainset)

# --- Optimized ---
optimized_dspy_score = evaluator(optimized_rag)

optimized_samples = collect_ragas_samples(optimized_rag, devset)
optimized_ragas = ragas_evaluate(
    dataset=EvaluationDataset(samples=optimized_samples),
    metrics=[Faithfulness(), AnswerRelevancy(), ContextRecall()],
)

# --- Compare ---
print("DSPy SemanticF1:")
print(f"  Baseline:  {baseline_dspy_score:.1f}%")
print(f"  Optimized: {optimized_dspy_score:.1f}%")

print("\nRagas Decomposed:")
for metric_name in ["faithfulness", "answer_relevancy", "context_recall"]:
    before = baseline_ragas[metric_name]
    after = optimized_ragas[metric_name]
    print(f"  {metric_name}: {before:.2f} → {after:.2f} ({after - before:+.2f})")