Ai Rag

RAG & Search Engineering — Complete Reference

Build production-grade retrieval systems with hybrid search, grounded generation, and measurable quality.

This skill covers:

• RAG: Chunking, contextual retrieval, grounding, adaptive/self-correcting systems
• Search: BM25, vector search, hybrid fusion, ranking pipelines
• Evaluation: recall@k, nDCG, MRR, groundedness metrics

Modern Best Practices (Jan 2026):

• Separate retrieval quality from answer quality; evaluate both (RAG: https://arxiv.org/abs/2005.11401).
• Default to hybrid retrieval (sparse + dense) with reranking when precision matters (DPR: https://arxiv.org/abs/2004.04906).
• Use a failure taxonomy to debug systematically (Seven Failure Points in RAG: https://arxiv.org/abs/2401.05856).
• Treat freshness/invalidation as first-class; staleness is a correctness bug, not a UX issue.
• Add grounding gates: answerability checks, citation coverage checks, and refusal-on-missing-context defaults.
• Threat-model RAG: retrieved text is untrusted input (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

Default posture: deterministic pipeline, bounded context, explicit failure handling, and telemetry for every stage.

Scope note: For prompt structure and output contracts used in the generation phase, see ai-prompt-engineering.

Quick Reference

Decision Tree: RAG Architecture Selection

Building RAG system: [Architecture Path]
    ├─ Document type?
    │   ├─ Page/section-structured? → Structure-aware chunking (pages/sections + metadata)
    │   ├─ Technical docs/code? → Structure-aware + code-aware chunking (symbols, headers)
    │   └─ Simple content? → Fixed-size token chunking with overlap (baseline)
    │
    ├─ Retrieval accuracy low?
    │   ├─ Query ambiguity? → Query rewriting + multi-query expansion + filters
    │   ├─ Noisy results? → Add reranker + better metadata filters
    │   └─ Mixed queries? → Hybrid retrieval (sparse + dense) + reranking
    │
    ├─ Dataset size?
    │   ├─ <100k chunks? → Flat index (exact search)
    │   ├─ 100k-10M? → HNSW (low latency)
    │   └─ >10M? → IVF/ScaNN/DiskANN (scalable)
    │
    └─ Production quality?
        └─ Add: ACLs, freshness/invalidation, eval gates, and telemetry (end-to-end)

Core Concepts (Vendor-Agnostic)

• Pipeline stages: ingest → chunk → embed → index → retrieve → rerank → pack context → generate → verify.
• Two evaluation planes: retrieval relevance (did we fetch the right evidence?) vs generation fidelity (did we use it correctly?).
• Freshness model: staleness budget, invalidation triggers, and rebuild strategy (incremental vs full).
• Trust boundaries: retrieved content is untrusted; apply the same rigor as user input (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

Implementation Practices (Tooling Examples)

• Use a retrieval API contract: query, filters, top_k, trace_id, and returned evidence IDs.
• Instrument each stage with tracing/metrics (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).
• Add caches deliberately: embeddings cache, retrieval cache (query+filters), and response cache (with invalidation).

Do / Avoid

Do

• Do keep retrieval deterministic: fixed top_k, stable ranking, explicit filters.
• Do enforce document-level ACLs at retrieval time (not only at generation time).
• Do include citations with stable IDs and verify citation coverage in tests.

Avoid

• Avoid shipping RAG without a test set and regression gate.
• Avoid "stuff everything" context packing; it increases cost and can reduce accuracy.
• Avoid mixing corpora without metadata and tenant isolation.

When to Use This Skill

Use this skill when the user asks:

• "Help me design a RAG pipeline."
• "How should I chunk this document?"
• "Optimize retrieval for my use case."
• "My RAG system is hallucinating — fix it."
• "Choose the right vector database / index type."
• "Create a RAG evaluation framework."
• "Debug why retrieval gives irrelevant results."

Tool/Model Recommendation Protocol

When users ask for vendor/model/framework recommendations, validate claims against current primary sources.

Triggers

• "What's the best vector database for [use case]?"
• "What should I use for [chunking/embedding/reranking]?"
• "What's the latest in RAG development?"
• "Current best practices for [retrieval/grounding/evaluation]?"
• "Is [Pinecone/Qdrant/Chroma] still relevant in 2026?"
• "[Vector DB A] vs [Vector DB B]?"
• "Best embedding model for [use case]?"
• "What RAG framework should I use?"

Required Checks

1. Read data/sources.json and start from sources with "add_as_web_search": true. 2. Verify 1-2 primary docs per recommendation (release notes, benchmarks, docs). 3. If browsing isn't available, state assumptions and give a verification checklist.

What to Report

After checking, provide:

• Current landscape: What vector DBs/embeddings are popular NOW (not 6 months ago)
• Emerging trends: Techniques gaining traction (late interaction, agentic RAG, graph RAG)
• Deprecated/declining: Approaches or tools losing relevance
• Recommendation: Based on fresh data, not just static knowledge

Example Topics (verify with current sources)

• Vector databases (Pinecone, Qdrant, Weaviate, Milvus, pgvector, LanceDB)
• Embedding models (OpenAI, Cohere, Voyage AI, Jina, Sentence Transformers)
• Reranking (Cohere Rerank, Jina Reranker, FlashRank, RankGPT)
• RAG frameworks (LlamaIndex, LangChain, Haystack, txtai)
• Advanced RAG (contextual retrieval, agentic RAG, graph RAG, CRAG)
• Evaluation (RAGAS, TruLens, DeepEval, BEIR)

Related Skills

For adjacent topics, reference these skills:

• [ai-llm](../ai-llm/SKILL.md) - Prompting, fine-tuning, instruction datasets
• [ai-agents](../ai-agents/SKILL.md) - Agentic RAG workflows and tool routing
• [ai-llm-inference](../ai-llm-inference/SKILL.md) - Serving performance, quantization, batching
• [ai-mlops](../ai-mlops/SKILL.md) - Deployment, monitoring, security, privacy, and governance
• [ai-prompt-engineering](../ai-prompt-engineering/SKILL.md) - Prompt patterns for RAG generation phase

Templates

System Design (Start Here)

• RAG System Design

Chunking & Ingestion

• Basic Chunking
• Code Chunking
• Long Document Chunking

Embedding & Indexing

• Index Configuration
• Metadata Schema

Retrieval & Reranking

• Retrieval Pipeline
• Hybrid Search
• Reranking
• Ranking Pipeline
• Reranker

Context Packaging & Grounding

• Context Packing
• Grounding

Evaluation

• RAG Evaluation
• RAG Test Set
• Search Evaluation
• Search Test Set

Search Configuration

• BM25 Configuration
• HNSW Configuration
• IVF Configuration
• Hybrid Configuration

Query Rewriting

• Query Rewrite

Navigation

Resources

• references/advanced-rag-patterns.md
• references/agentic-rag-patterns.md
• references/bm25-tuning.md
• references/chunking-patterns.md
• references/chunking-strategies.md
• references/rag-evaluation-guide.md
• references/rag-troubleshooting.md
• references/contextual-retrieval-guide.md
• references/distributed-search-slos.md
• references/grounding-checklists.md
• references/hybrid-fusion-patterns.md
• references/index-selection-guide.md
• references/multilingual-domain-patterns.md
• references/pipeline-architecture.md
• references/query-rewriting-patterns.md
• references/ranking-pipeline-guide.md
• references/retrieval-patterns.md
• references/search-debugging.md
• references/search-evaluation-guide.md
• references/user-feedback-learning.md
• references/vector-search-patterns.md
• references/graph-rag-patterns.md
• references/embedding-model-guide.md
• references/rag-caching-patterns.md

Templates

• assets/context/template-context-packing.md
• assets/context/template-grounding.md
• assets/design/rag-system-design.md
• assets/chunking/template-basic-chunking.md
• assets/chunking/template-code-chunking.md
• assets/chunking/template-long-doc-chunking.md
• assets/retrieval/template-retrieval-pipeline.md
• assets/retrieval/template-hybrid-search.md
• assets/retrieval/template-reranking.md
• assets/eval/template-rag-eval.md
• assets/eval/template-rag-testset.jsonl
• assets/eval/template-search-eval.md
• assets/eval/template-search-testset.jsonl
• assets/indexing/template-index-config.md
• assets/indexing/template-metadata-schema.md
• assets/query/template-query-rewrite.md
• assets/ranking/template-ranking-pipeline.md
• assets/ranking/template-reranker.md
• assets/search/template-bm25-config.md
• assets/search/template-hnsw-config.md
• assets/search/template-ivf-config.md
• assets/search/template-hybrid-config.md

Data

• data/sources.json — Curated external references

Use this skill whenever the user needs retrieval-augmented system design or debugging, not prompt work or deployment.

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Basic Chunking Template (Sliding Window)

A generic sliding-window chunking template suitable for blogs, docs, articles, and general unstructured text.

---

1. Parameters

chunk_size: 600 # tokens chunk_overlap: 100 # tokens min_chunk_size: 150 split_on_headings: true

---

2. Chunking Workflow

1. Clean text 2. Normalize whitespace 3. Split by top-level headings if available 4. Apply sliding window chunking 5. Add metadata 6. Filter out empty/small chunks

---

3. Output Format

{ "id": "<chunk_id>", "text": "<chunk_text>", "source": "<document_id>", "start_pos": <token_index>, "end_pos": <token_index>, "metadata": { "section": "<section_title>", "timestamp": "<optional>", "tags": [] } }

---

4. Validation Checklist

• [ ] No broken sentences
• [ ] No empty chunks
• [ ] IDs stable
• [ ] Chunk count reasonable

Code Chunking Template (Function/Class Aware)

Chunk code by logical boundaries rather than fixed token sizes.

---

1. Parameters

chunk_type: "code" min_block_size: 30 max_block_size: 200 language: "<python/js/java/etc>"

---

2. Chunking Rules

• Split by:
• Function definitions
• Class definitions
• Top-level blocks
• Maintain indentation context
• Add file path metadata
• Avoid breaking functions across chunks

---

3. Output Format

{ "id": "<chunk_id>", "text": "<code_block>", "source_file": "<path/to/file>", "function": "<function_name>", "language": "<lang>" }

---

4. Checklist

• [ ] Syntax-preserving
• [ ] Blocks not split mid-function
• [ ] Language metadata applied

Long Document Chunking Template (Hierarchical)

Use for manuals, books, legal docs, and structured multi-section content.

---

1. Parameters

max_section_length: 1200 # tokens subsection_overlap: 150 preserve_headings: true

---

2. Workflow

1. Split by major section headings 2. Inside each section:

• Apply sliding-window chunking
• Maintain parent/child structure

3. Add hierarchical metadata:

• section
• subsection
• page number

---

3. Output Format

{ "id": "<chunk_id>", "text": "<chunk>", "source": "<document_id>", "metadata": { "section": "<heading>", "subsection": "<subheading>", "page": <page_number> } }

Context Packing Template

A deterministic method for assembling the final context block fed into the LLM.

---

1. Strategy

context_window: <max_tokens> max_chunks: 5 ordering: "relevance" dedupe: true preserve_section_titles: true

---

2. Context Assembly Steps

1. Sort retrieved chunks by score 2. Remove duplicates 3. Add section titles 4. Concatenate with separators 5. Stop when reaching token budget

---

3. Template Format

<START_CONTEXT> [1] <section_title> <chunk_text> [2] <section_title> <chunk_text> ... <END_CONTEXT>

---

4. Checklist

• [ ] Fits within model token limit
• [ ] Ordered by relevance
• [ ] Section titles preserved
• [ ] Clean concatenation (no markup noise)

Grounding Template

Ensures that generation stays strictly tied to retrieved context.

---

System

You must use only the evidence provided in the context block. If the answer is not contained in the context, output: "Not found in the documents."

---

Input

Context: <CONTEXT_BLOCK>

Query: <USER_QUERY>

---

Output Requirements

• No external facts
• No speculation
• Use citations referencing chunk index
• Format:

{ "answer": "<text>", "sources": ["<chunk-id-1>", "<chunk-id-2>"] }

---

Checklist

• [ ] All claims traceable
• [ ] Refusal used when answer not in context
• [ ] JSON-safe output

RAG System Design Template

Purpose: Document architecture decisions, define evaluation criteria, plan for production operation.

---

Template Contract

Goals

• Deliver correct, grounded answers with measurable retrieval and generation quality.
• Meet latency/cost SLOs with a predictable, observable pipeline.
• Ensure security, privacy, and governance for the corpus and queries.

Inputs

• Use case, query distribution, and acceptance criteria.
• Corpus characteristics (types, size, update frequency, sensitivity, ACLs).
• Platform constraints (latency, QPS, budget, residency, retention).

Decisions

• Whether to use RAG vs alternatives, and the retrieval architecture (sparse/dense/hybrid, reranking).
• Chunking/parsing strategy and metadata schema.
• Freshness/invalidation strategy and failure fallbacks.

Risks

• Staleness, incorrect ACL enforcement, and prompt injection via retrieved text.
• Poor retrieval recall/precision leading to hallucinations or "missing evidence."
• Cost blowups from oversized contexts or unbounded retries.

Metrics

• Retrieval: recall@k, nDCG/MRR, empty-result rate, latency.
• Answer: groundedness/faithfulness, citation coverage, refusal correctness, hallucination rate.
• Ops: cost per request, error rate, cache hit rates, rebuild time.

1. Problem Definition

Use Case

• Domain: _______________
• Query types: [ ] Factual [ ] Analytical [ ] Conversational [ ] Multi-hop
• Expected QPS: ___
• Latency budget (total): ___ms
• Latency budget (retrieval): ___ms

Data Characteristics

• Corpus size: ___ documents / ___ chunks
• Update frequency: [ ] Real-time [ ] Hourly [ ] Daily [ ] Weekly [ ] Static
• Document types: [ ] Text [ ] Tables [ ] Code [ ] PDFs [ ] Images [ ] Mixed
• Average doc length: ___ tokens
• Languages: _______________
• Sensitive data: [ ] PII [ ] Confidential [ ] Public only

---

2. Decision: Is RAG the Right Approach?

RAG Decision Tree

All data fits in context window?
  -> YES: Consider direct prompting (simpler, faster)
  -> NO: Continue...

Data changes frequently?
  -> YES: RAG preferred (vs fine-tuning)
  -> NO: Consider fine-tuning if data is stable

Need citations/traceability?
  -> YES: RAG required
  -> NO: Fine-tuning acceptable

Latency budget <100ms?
  -> YES: Pre-compute or fine-tune
  -> NO: RAG acceptable

Query requires reasoning over multiple docs?
  -> YES: RAG with multi-hop retrieval
  -> NO: Standard RAG or hybrid

Decision

• [ ] RAG is appropriate for this use case
• [ ] Alternative considered: _______________
• [ ] Rationale: _______________

---

3. Architecture Decisions

Chunking Strategy

Selected approach: _______________

• Chunk size: ___ tokens
• Overlap: ___ tokens
• Rationale: _______________

Embedding Model

Selected model: _______________

• Rationale: _______________

Retrieval Method

Selected method: _______________

• Hybrid weights (if applicable): Dense=___, Sparse=___
• Top-K: ___

Reranking

• [ ] Reranking enabled
• Reranker model: _______________
• Rerank top-K: ___
• Final top-K after rerank: ___

Vector Store

Selected store: _______________

• Index type: [ ] HNSW [ ] IVF [ ] Flat
• Distance metric: [ ] Cosine [ ] L2 [ ] Inner Product

---

4. Contextual Retrieval (Recommended)

Chunk Context Augmentation

• [ ] Add lightweight context to chunks before embedding (document title/section summary)
• Context generation method: _______________
• Context length: ___ tokens per chunk

Implementation

chunk_format:
  context: "{document_summary}. {section_context}"
  content: "{chunk_text}"
  metadata: {source, page, section, timestamp}

---

5. Evaluation Plan

Retrieval Metrics

Answer Metrics

Test Set Requirements

• [ ] Minimum 100 test queries
• [ ] Ground truth documents identified
• [ ] Adversarial queries included (edge cases)
• [ ] Multi-hop queries included (if applicable)

---

6. Production Considerations

Index Freshness

Selected strategy: _______________

• Staleness tolerance: ___ hours
• Invalidation approach: _______________

Failure Modes & Fallbacks

Monitoring

---

7. Security & Privacy

• [ ] PII detection in chunks
• [ ] Access control per document/chunk
• [ ] Audit logging for queries
• [ ] Data retention policy defined
• [ ] Encryption at rest and in transit

---

8. Cost Estimation

---

9. Implementation Checklist

Phase 1: Data Pipeline

• [ ] Document ingestion implemented
• [ ] Chunking strategy implemented
• [ ] Embedding pipeline working
• [ ] Vector store provisioned and indexed

Phase 2: Retrieval

• [ ] Query embedding working
• [ ] Vector search working
• [ ] Hybrid search configured (if applicable)
• [ ] Reranking integrated (if applicable)

Phase 3: Generation

• [ ] Context injection working
• [ ] Citation extraction implemented
• [ ] Grounding validation active

Phase 4: Evaluation

• [ ] Test set created
• [ ] Retrieval metrics baseline established
• [ ] Answer quality metrics baseline established
• [ ] Regression tests automated

Phase 5: Production

• [ ] Monitoring configured
• [ ] Alerts set up
• [ ] Index refresh automated
• [ ] Failover tested

---

10. Sign-Off

RAG Evaluation Template

A reproducible structure for evaluating retrieval quality and grounded generation.

---

1. Evaluation Tasks

• Closed-book QA
• Grounded QA
• Multi-hop reasoning
• Summarization with citations
• Fact extraction

---

2. Metrics

Retrieval Metrics

• Recall@k
• Precision@k
• nDCG

Generation Metrics

• Correctness
• Groundedness
• Hallucination rate
• Citation validity

---

3. Evaluation Protocol

1. Build 20–200 sample queries 2. Define gold answers 3. Run:

• BM25
• Vector-only
• Hybrid
• Reranked

4. Evaluate grounded answers 5. Score with rubric

---

4. Output Template

{ "query": "<query>", "retrieved_chunks": ["<id1>", "<id2>", ...], "generated_answer": "<answer>", "gold_answer": "<gold>", "metrics": { "recall@k": <value>, "groundedness": <value>, "hallucination_rate": <value> } }

---

5. Checklist

• [ ] All retrieval stages evaluated
• [ ] Grounding validated
• [ ] Scores stored with version

Search Evaluation Template

A reproducible evaluation structure for BM25, vector, or hybrid search systems.

---

1. Evaluation Settings

evaluation: metrics:

• "ndcg@10"
• "recall@10"
• "precision@5"

slices:

• "query_length"
• "domain"

---

2. Test Steps

1. Load testset 2. Run BM25 3. Run vector search 4. Run hybrid search 5. Compute metrics 6. Compare variants 7. Log version

---

3. Output Format

{ "query": "<query>", "gold_docs": ["<id1>", "<id2>"], "retrieved_docs": ["<idX>", "<idY>"], "metrics": { "ndcg@10": <float>, "recall@10": <float>, "precision@5": <float> } }

---

4. Evaluation Checklist

• [ ] Testset covers keyword + semantic queries
• [ ] Baselines compared
• [ ] Reranker evaluated
• [ ] Version logged

Vector Index Configuration Template

Defines index parameters for ANN (HNSW, IVF, ScaNN, DiskANN) vector search.

---

1. Index Type

index_type: "<flat | hnsw | ivf | scann | diskann>"

Recommended:

• Flat → small datasets
• HNSW → general purpose
• IVF → large datasets
• ScaNN → high-dimensional
• DiskANN → extremely large corpora

---

2. Embedding Settings

embedding_model: "<model_name>" embedding_dim: <dimension> normalize_vectors: true/false

---

3. HNSW Parameters

hnsw: M: 32 ef_construction: 200 ef_search: 128

---

4. IVF Parameters

ivf: nlist: 4096 nprobe: 16

---

5. ScaNN Parameters

scann: training_sample_size: 50000 leaves: 2048 reordering_candidates: 100

---

6. Metadata Indexing

metadata_fields: "section" "tags" "timestamp"

---

7. Validation Checklist

• [ ] Index type matches data size
• [ ] ef_search/nprobe tuned
• [ ] Embedding model consistent
• [ ] Metadata searchable

Vector Index Configuration Template

Defines index parameters for ANN (HNSW, IVF, ScaNN, DiskANN) vector search.

---

1. Index Type

index_type: "<flat | hnsw | ivf | scann | diskann>"

Recommended:

• Flat → small datasets
• HNSW → general purpose
• IVF → large datasets
• ScaNN → high-dimensional
• DiskANN → extremely large corpora

---

2. Embedding Settings

embedding_model: "<model_name>" embedding_dim: <dimension> normalize_vectors: true/false

---

3. HNSW Parameters

hnsw: M: 32 ef_construction: 200 ef_search: 128

---

4. IVF Parameters

ivf: nlist: 4096 nprobe: 16

---

5. ScaNN Parameters

scann: training_sample_size: 50000 leaves: 2048 reordering_candidates: 100

---

6. Metadata Indexing

metadata_fields: "section" "tags" "timestamp"

---

7. Validation Checklist

• [ ] Index type matches data size
• [ ] ef_search/nprobe tuned
• [ ] Embedding model consistent
• [ ] Metadata searchable

Query Rewriting Template

A template for LLM-assisted query rewriting to improve retrieval performance.

---

Task

Rewrite the query to maximize retrieval relevance. Add synonyms and alternative phrasings without changing intent.

---

Input

<USER_QUERY>

---

Output

{ "rewritten_query": "<expanded_query>", "keywords": ["<keyword1>", "<keyword2>"] }

---

Rules

• Do not add assumptions
• Maintain original meaning
• Keep stable, deterministic structure
• Do not exceed 20 tokens unless necessary

---

Checklist

• [ ] No hallucinated facts
• [ ] Meaning preserved
• [ ] Improves recall@k on evaluation set

Ranking Pipeline Template

A complete architecture for a multi-stage ranking pipeline.

---

1. Pipeline Structure

query → candidate_generation → filtering → scoring → reranking → final_output

---

2. Candidate Generation Config

candidate_generation: methods:

• "bm25"
• "vector"

top_k: 200

---

3. Filtering Rules

filtering: required_metadata:

• "language"
• "visibility"

allowed_types:

• "article"
• "faq"

---

4. Scoring Strategy

scoring: method: "rrf" k: 60

---

5. Reranking Stage

reranking: enabled: true model: "<cross_encoder_or_llm>" top_k_candidates: 50 final_top_n: 10

---

6. Output Format

{ "results": [ { "doc_id": "<id>", "score": <score>, "snippet": "<text>" } ] }

---

7. Ranking Checklist

• [ ] High recall from candidate generation
• [ ] Filtering correct
• [ ] Reranker improves relevance
• [ ] Latency acceptable

Reranker Configuration Template

Config for applying cross-encoder or LLM reranking to candidate documents.

---

1. Reranker Settings

reranker: model: "<model_id>" top_k_candidates: 50 final_top_n: 10 batch_size: 8

---

2. Scoring Logic

Pairs each query with each candidate document:

score = reranker.predict(query, doc_text)

Then sorts descending by score.

---

3. Output Schema

{ "reranked": [ { "doc_id": "<id>", "score": <score> } ] }

---

4. Checklist

• [ ] Reranker latency measured
• [ ] Improves nDCG@10
• [ ] Scores stable across evaluation set

Hybrid Search Template (BM25 + Vector Search)

A unified configuration for combining lexical and semantic retrieval.

---

1. Hybrid Mode

hybrid: enabled: true methods:

• bm25
• vector

---

2. BM25 Stage

bm25: top_k: 20 field_boosts: title: 3 body: 1

---

3. Vector Stage

vector: top_k: 20 index_type: "hnsw" ef_search: 128

---

4. Fusion

fusion: type: "weighted_sum" alpha: 0.4 beta: 0.6

or use:

fusion: type: "rrf" k: 60

---

5. Checklist

• [ ] BM25 tuned
• [ ] Vector search tuned
• [ ] Fusion validated
• [ ] Recall@k increased vs single method

Reranking Template

Use a cross-encoder or LLM to refine retrieved candidates.

---

1. Reranker Config

reranker: model: "<cross_encoder_model>" top_k_candidates: 50 final_top_n: 5

---

2. Reranking Workflow

1. Input: K candidates from ANN/BM25 2. Score each candidate with cross-encoder/LLM 3. Sort by descending score 4. Select top N

---

3. Output Schema

{ "reranked": [ { "chunk_id": "<id>", "score": <score>, "text": "<text>", "metadata": { ... } } ] }

---

4. Reranking Checklist

• [ ] Latency acceptable
• [ ] Improves nDCG@10
• [ ] No regressions on keyword queries

Retrieval Pipeline Template

Defines the full retrieval → rerank → context assembly pipeline.

---

1. Pipeline Overview

query → preprocess → embed → search(K) → rerank(optional) → select_top_N → pack_context

---

2. Query Preprocessing

• Normalize whitespace
• Strip HTML/Markdown
• (Optional) query rewriting

---

3. Retrieval Step

retrieval: top_k: 20 embedding_model: "<model_name>" index: "<index_type>"

---

4. Optional Reranking

reranking: enabled: true model: "<cross_encoder_or_llm>" top_k: 50

---

5. Context Selection

context: max_chunks: 5 ordering: "by_score"

---

6. Output Schema

{ "query": "<query>", "results": [ { "chunk_id": "<id>", "score": <score>, "text": "<text>", "metadata": { ... } } ] }

---

7. Checklist

• [ ] Same embedding model for indexing & querying
• [ ] K tuned for recall@k
• [ ] Reranking improves precision

BM25 Configuration Template

A production-ready configuration for lexical BM25 retrieval.

---

1. Preprocessing Rules

preprocessing: lowercase: true strip_html: true normalize_unicode: true remove_punctuation: false stopwords: "auto" # auto | none | custom_list

---

2. BM25 Parameters

bm25: k1: 1.4 b: 0.65

These values are baseline defaults — tune based on evaluation.

---

3. Field Weights

field_boosts: title: 3.0 subtitle: 1.5 body: 1.0 tags: 2.0

---

4. Indexing Settings

index: store_positions: true store_offsets: false store_docvectors: false

---

5. Query Configuration

query: expand_synonyms: true use_spellcheck: false

---

6. Quality Checklist

• [ ] k1 tuned
• [ ] b tuned
• [ ] Field boosts validated
• [ ] Tokenizer consistent across indexing and query

HNSW Index Configuration Template

HNSW is the default ANN index for dense vector search.

---

1. Index Parameters

hnsw: M: 32 # graph connectivity ef_construction: 200 # build-time accuracy ef_search: 128 # query-time accuracy

---

2. Embedding Settings

embedding: model: "<embedding_model>" dim: <dimension> normalize_vectors: true

---

3. Metadata

metadata_fields: "section" "tags" "timestamp"

---

4. Latency vs Recall Tuning

• Increase ef_search → higher recall, slower
• Decrease ef_search → lower recall, faster

---

5. Validation Checklist

• [ ] M tuned
• [ ] ef_search tuned
• [ ] Same embedding model for indexing & querying
• [ ] Metadata filters tested

Hybrid Search Configuration Template

Combine lexical and vector retrieval results for improved relevance.

---

1. Hybrid Mode

hybrid: enabled: true components:

• "bm25"
• "vector"

---

2. BM25 Component

bm25: top_k: 20 k1: 1.4 b: 0.65 field_boosts: title: 3 body: 1

---

3. Vector Component

vector: top_k: 20 index_type: "hnsw" ef_search: 128

---

4. Fusion Method

Weighted sum:

fusion: method: "weighted" alpha: 0.4 beta: 0.6

or RRF:

fusion: method: "rrf" k: 60

---

5. Validation Checklist

• [ ] Hybrid improves recall
• [ ] Fusion tuned
• [ ] Keyword queries preserved
• [ ] Long-tail semantic queries improved

IVF Index Configuration Template

Use IVF for large corpora (millions–hundreds of millions of vectors).

---

1. IVF Parameters

ivf: nlist: 4096 nprobe: 16 metric: "cosine" # cosine | l2

---

2. Embedding Settings

embedding: model: "<model_name>" dim: <dimension> normalize_vectors: true

---

3. Training

training: sample_size: 100000

---

4. Validation Checklist

• [ ] nlist selected based on data size
• [ ] nprobe tuned for recall
• [ ] Embedding model consistent
• [ ] Clustering stable

Advanced RAG Patterns

Modern RAG paradigms beyond static retrieval: structured data, graph RAG, multimodal retrieval, online evaluation, and production telemetry.

---

Structured, Graph, and Multimodal RAG

Use when: Data has strong structure (tables/graphs), relationships, or images that text-only chunks miss.

Graph/Knowledge RAG

• Build entity/relation graph
• Store text evidence per edge
• Use graph traversal (k-hop, path ranking) before generation
• Fall back to text RAG when graph thin
• Example use cases: Knowledge bases, entity-heavy documents, technical manuals with component relationships

Table/Structured Data RAG

• Normalize tables
• Add row/column headers to text
• Use hybrid retrieval (lexical + dense) with schema-aware metadata
• Cite cell coordinates
• Example use cases: Financial reports, scientific data, product specifications

Multimodal RAG

• Encode images with vision encoder + connector
• Store image embeddings alongside text
• Retrieve both text and images
• Include low-res thumbnails/alt-text for grounding
• Gate unsafe images
• Example use cases: Product catalogs, medical imaging, technical diagrams

Two-Stage Pipelines

• Stage 1: Structured/graph retrieval first
• Stage 2: Augment with unstructured context
• Use reranker that can handle structured hints
• Benefit: Best of both worlds - precision from structure, coverage from text

Freshness/Governance

• Keep graph/table versioning
• Backfill after schema changes
• Per-tenant/index isolation when required
• Track lineage and provenance

Implementation Checklist

• [ ] Graph/table indexing path defined with versioning and rollback
• [ ] Multimodal retrieval tested (vision encoder + connector latency)
• [ ] Hybrid pipeline returns structured + unstructured context
• [ ] Safety filter for images and PII in structured data
• [ ] Evaluated on structured/multimodal benchmarks or slice sets

---

Online Evaluation, Telemetry, and Freshness

Use when: Running RAG in production with changing data.

Online Signals

• Capture click/open, dwell/scroll, abandonment, manual edits
• Link to request IDs
• Scrub PII before logging
• Metrics to track:
• Click-through rate (CTR)
• Time to first action
• Session abandonment rate
• Manual edit frequency

Shadow/Canary Testing

• Route small % of traffic to new retrievers/chunkers/rerankers
• Measure solve rate, cost, latency, groundedness
• Abort on regressions
• Best practice: Start with 1-5% traffic, expand gradually

Freshness Telemetry

• Track ingestion lag per source
• Alert on staleness
• Surface index/version in responses for audit
• SLA targets:
• Real-time systems: <5min lag
• Batch systems: <24hr lag
• Archive systems: <7d lag

Slice Dashboards

Track metrics by dimension:

• Domain/source/task slices
• Multilingual slices
• Structured vs unstructured
• Hallucination/grounding errors tracked separately
• Purpose: Identify performance degradation in specific segments

Eval Set Protection

• Prevent production data (with IDs) from contaminating eval sets
• Log hash of eval items
• Periodic refresh with human review
• Anti-pattern: Using production queries directly as eval set

Production Checklist

• [ ] Online metrics wired (solve rate, grounding, latency, cost)
• [ ] Shadow/canary gates with auto-abort on regression
• [ ] Ingestion lag + index/version surfaced and monitored
• [ ] Sliced dashboards (domain, language, data type)
• [ ] Eval contamination checks in place

---

Context Compression & Budgeting

Use when context window is tight.

Compression Strategies

1. Merge adjacent chunks - Combine semantically related chunks 2. Deduplicate repeated sentences - Remove redundant information 3. Summarize long chunks (LLM distillation) - Use smaller model to compress 4. Prioritize by relevance score - Include highest-scoring chunks first 5. Structure context as sections - Group by topic/source

Token Budget Management

• Calculate token budget: model_context_window - prompt_tokens - max_output_tokens
• Reserve 20-30% buffer for formatting overhead
• Track actual usage vs budget
• Example: Claude 200k context → reserve ~150k for retrieved content

Context Optimization Checklist

• [ ] Fits within model's token budget
• [ ] Includes top-ranked chunks
• [ ] Avoids filler / irrelevant content
• [ ] Document titles preserved
• [ ] Compression tested on eval set (no quality degradation)

---

Modern Paradigm Shift

The era of static RAG is over. Modern RAG systems are:

Adaptive Retrieval

• Query complexity determines retrieval strategy
• Simple queries: direct vector search
• Complex queries: multi-hop, graph traversal, iterative refinement

Self-Correcting Systems

• Monitor retrieval quality in real-time
• Automatic fallback strategies
• Query reformulation on poor results

Wise Retrieval

• Context-aware chunking (Contextual Retrieval)
• Learned ranking (rerankers trained on domain data)
• Personalized retrieval (user history, preferences)

Multimodal & Structured Integration

• Text + images + tables + graphs
• Unified retrieval across modalities
• Structure-aware generation

---

GEAR: Graph-Enhanced Agentic RAG

2026 Update: GEAR combines knowledge graphs with agentic retrieval for enterprise use cases where relationships between entities are critical.

Architecture

GEAR Pipeline:
  1. Query → Entity extraction (NER)
  2. Graph lookup → Find related entities and relationships
  3. Decision: Graph-only, vector-only, or hybrid?
  4. Hybrid retrieval:
     - Graph traversal for structured facts
     - Vector search for unstructured context
  5. Merge results with entity grounding
  6. Generate with structured + unstructured evidence

When GEAR Outperforms Standard RAG

Implementation Components

1. Knowledge Graph Construction

• Entity extraction from documents
• Relationship extraction (LLM or rule-based)
• Graph storage (Neo4j, Amazon Neptune, or embedded)

2. Query Understanding

• Entity recognition in query
• Intent classification (factual vs relational vs aggregation)
• Graph query generation (Cypher, SPARQL, or custom)

3. Hybrid Retrieval

• Graph: Entities + 1-2 hop neighbors + edge evidence
• Vector: Semantically similar chunks
• Fusion: Interleave or prioritize based on query type

GEAR Checklist

• [ ] Entity extraction pipeline defined
• [ ] Graph schema designed for domain
• [ ] Query router distinguishes factual vs relational queries
• [ ] Hybrid fusion tested (graph + vector)
• [ ] Evaluated on relationship-heavy queries

---

Contextual Memory: RAG Alternative for Agentic AI

2026 Trend: For agentic AI systems, contextual memory (also called agentic memory or long-context memory) is emerging as an alternative to traditional RAG.

RAG vs Contextual Memory

When to Use Contextual Memory

Good candidates:

• Long-running agent sessions
• Conversational AI with persistent state
• Workflows where previous actions inform future decisions
• Small-to-medium knowledge bases (<200K tokens)

Stick with RAG when:

• Large corpus (>500K tokens)
• Strict citation requirements
• Multi-user with different access permissions
• Freshness from external sources needed

Hybrid Approach

Many production systems combine both:

Hybrid Memory Architecture:
  1. Session context: Recent interactions in context window
  2. Short-term memory: Important facts from current session (contextual)
  3. Long-term memory: Persistent knowledge base (RAG)
  4. Query routing: Decide which memory tier to query

Implementation Patterns

1. Sliding Window + RAG

• Keep last N turns in context
• RAG for older or external knowledge

2. Memory Compression

• Summarize old context periodically
• Store summaries in vector DB
• Retrieve relevant summaries

3. Memory Types

• Episodic: What happened (events, actions)
• Semantic: What is true (facts, knowledge)
• Procedural: How to do things (workflows, rules)

---

Related Resources

• Agentic RAG Patterns - Loop-based RAG architectures

• Contextual Retrieval Guide - Anthropic's 2024 technique
• Retrieval Patterns - Hybrid search and reranking
• Grounding Checklists - Hallucination prevention
• RAG Troubleshooting - Debugging production issues

Agentic RAG Patterns

Loop-based RAG architectures with autonomous reasoning, self-correction, and adaptive retrieval strategies.

Status: Production standard as of January 2026. Linear "naive RAG" pipelines are obsolete for complex use cases.

References:

• Agentic RAG Survey (https://arxiv.org/html/2501.09136v1)
• Building Agentic RAG with LangGraph (https://rahulkolekar.com/building-agentic-rag-systems-with-langgraph/)
• Top Agentic RAG Frameworks 2026 (https://research.aimultiple.com/agentic-rag/)

---

Why Agentic RAG?

Traditional RAG is a pipeline: Retrieve → Augment → Generate.

Agentic RAG is a loop: The LLM acts as a reasoning engine with autonomy to:

• Decide when to retrieve (not always)
• Reformulate queries on poor results
• Perform multi-hop retrieval for complex questions
• Self-correct and verify answers
• Route to different retrieval strategies

---

Architecture Comparison

Traditional RAG (Linear Pipeline):
  Query → Retrieve → Pack Context → Generate → Response
  └─ Single pass, no feedback, brittle

Agentic RAG (Reasoning Loop):
  Query → [Agent Decision Loop]
           ├─ Need retrieval? → Retrieve → Evaluate relevance
           │                    └─ Poor results? → Reformulate → Retry
           ├─ Need decomposition? → Split into sub-queries → Aggregate
           ├─ Have enough context? → Generate → Verify citations
           │                         └─ Verification failed? → Re-retrieve
           └─ Sufficient confidence? → Response

---

Core Patterns

1. Adaptive Retrieval

Problem: Not all queries need retrieval. Simple factual queries waste latency.

Pattern:

Query Analysis:
  ├─ Factual, in training data → Direct answer (no retrieval)
  ├─ Recent/domain-specific → Full RAG pipeline
  ├─ Ambiguous → Clarify before retrieval
  └─ Multi-faceted → Decompose into sub-queries

Implementation:

• Classifier or LLM judge on query complexity
• Track retrieval decision in telemetry
• Measure retrieval skip rate vs answer quality

2. Self-Correcting Retrieval

Problem: First retrieval often returns near-misses or irrelevant results.

Pattern:

Retrieval Loop:
  1. Initial query → Retrieve top-k
  2. Relevance check (reranker or LLM judge)
  3. If relevance < threshold:
     - Analyze why (too broad? wrong terminology?)
     - Reformulate query (expand, narrow, rephrase)
     - Re-retrieve with new query
  4. Max iterations = 3 (prevent infinite loops)

Key Signals for Reformulation:

• All results from same source (too narrow)
• High lexical overlap but low semantic match (wrong terminology)
• Results from wrong time period (add date filters)
• Results in wrong language (add language filter)

3. Multi-Hop Reasoning

Problem: Complex questions require information from multiple sources that must be combined.

Pattern:

Multi-Hop Pipeline:
  1. Decompose: "What's the revenue impact of feature X?" →
     - Sub-Q1: "What is feature X?"
     - Sub-Q2: "When was feature X launched?"
     - Sub-Q3: "What were revenue numbers before/after launch?"
  2. Retrieve for each sub-query
  3. Synthesize: Combine evidence, resolve contradictions
  4. Generate: Answer with citations from multiple hops

When to Use:

• Comparison questions ("X vs Y")
• Causal questions ("Why did X happen?")
• Aggregation questions ("How many...", "What's the total...")
• Timeline questions ("What happened after X?")

4. Verification Loop

Problem: Generated answers may hallucinate or misattribute citations.

Pattern:

Post-Generation Verification:
  1. Generate answer with inline citations [1], [2]
  2. Extract each claim + citation pair
  3. Verify: Does cited chunk support the claim?
     - If yes → Keep
     - If no → Flag for re-generation or removal
  4. Check coverage: All claims have citations?
  5. Fail if verification rate < threshold

Verification Signals:

• Citation points to chunk that doesn't contain claimed fact
• Citation chunk is about different entity/time period
• Claim extrapolates beyond source (opinion presented as fact)

---

Decision Tree: When to Use Agentic RAG

Should you use Agentic RAG?
  │
  ├─ Query complexity?
  │   ├─ Simple factual → Traditional RAG (lower latency)
  │   ├─ Multi-hop/comparative → Agentic (decomposition needed)
  │   └─ Ambiguous → Agentic (clarification + adaptive retrieval)
  │
  ├─ Corpus quality?
  │   ├─ High-quality, well-structured → Traditional may suffice
  │   └─ Noisy, overlapping, inconsistent → Agentic (self-correction)
  │
  ├─ Accuracy requirements?
  │   ├─ Approximate OK → Traditional (faster)
  │   └─ High stakes (legal, medical, financial) → Agentic (verification)
  │
  └─ Latency budget?
      ├─ <2s required → Traditional or hybrid
      └─ 5-10s acceptable → Full agentic loop

---

Implementation Frameworks (2026)

LangGraph Example Structure

# Conceptual structure - not runnable code
graph = StateGraph(AgentState)

graph.add_node("analyze_query", analyze_complexity)
graph.add_node("retrieve", retrieval_node)
graph.add_node("evaluate_relevance", relevance_checker)
graph.add_node("reformulate", query_reformulator)
graph.add_node("generate", generation_node)
graph.add_node("verify", citation_verifier)

# Conditional edges for loops
graph.add_conditional_edges(
    "evaluate_relevance",
    should_reformulate,
    {"reformulate": "reformulate", "generate": "generate"}
)
graph.add_conditional_edges(
    "verify",
    verification_passed,
    {"pass": END, "fail": "retrieve"}  # Loop back on failure
)

---

GEAR: Graph-Enhanced Agentic RAG

Pattern: Combine knowledge graphs with agentic retrieval for enterprise use cases.

GEAR Architecture:
  1. Query → Entity extraction
  2. Graph lookup → Related entities, relationships
  3. Hybrid retrieval:
     - Graph traversal for structured facts
     - Vector search for unstructured context
  4. Agent decides: Graph-only, vector-only, or both?
  5. Synthesis with entity grounding

Best For:

• Enterprise knowledge bases with entity relationships
• Compliance/legal where relationships matter
• Product catalogs with hierarchies
• Technical documentation with cross-references

---

Operational Considerations

Latency Budget

Telemetry Requirements

Track per-request:

• Retrieval decision (skip/execute)
• Number of retrieval iterations
• Query reformulations applied
• Verification pass/fail rate
• Total latency breakdown by stage

Cost Management

Agentic RAG uses more tokens:

• Query analysis: +500-1000 tokens
• Reformulation: +500 tokens per iteration
• Verification: +1000-2000 tokens
• Budget 2-4x traditional RAG token cost

Failure Modes

---

Implementation Checklist

• [ ] Query classifier for adaptive retrieval (skip vs execute)
• [ ] Relevance evaluator (reranker or LLM judge)
• [ ] Query reformulation logic with max iterations
• [ ] Multi-hop decomposition for complex queries
• [ ] Post-generation verification with citation checking
• [ ] Telemetry: retrieval decisions, iterations, latency breakdown
• [ ] Cost tracking: tokens per request by stage
• [ ] Eval set with complexity labels (simple/multi-hop/ambiguous)
• [ ] Latency SLOs per complexity tier

---

Anti-Patterns

• Always retrieve: Wastes latency on simple queries; add skip logic
• No max iterations: Can loop forever; cap at 3 iterations
• Reformulate blindly: Analyze why retrieval failed before changing query
• Skip verification: Hallucinations go undetected; always verify high-stakes answers
• Single eval metric: Measure retrieval quality AND generation quality separately

---

Related Resources

• Retrieval Patterns - Hybrid search, reranking
• RAG Evaluation Guide - Metrics for agentic systems
• RAG Troubleshooting - Debugging retrieval loops
• ai-agents skill - General agent architectures

BM25 Tuning Guide

A practical, repeatable process for tuning BM25 lexical search for maximum relevance.

---

1. When to Use BM25

Use BM25 when:

• Documents are text-heavy
• Users search using keywords
• High precision for exact matches is required
• Queries contain domain-specific vocabulary

---

2. Parameters

k1 (Term Frequency Saturation)

Controls how term frequency influences relevance. Range: 1.2–1.8

b (Length Normalization)

Controls how document length affects scoring. Range: 0.55–0.75

---

3. Optimization Workflow

Step 1 — Preprocessing

• Remove excessive whitespace
• Lowercase (if case-insensitive)
• Strip markup (HTML, markdown)

Checklist

• [ ] Tokenization verified
• [ ] Stopword removal optional (test both ways)

---

Step 2 — Field Weighting

Boost important fields:

• title
• summary
• h1/h2 headings
• metadata fields

Example: title^3 summary^1.5 body^1

---

Step 3 — Query Expansion

Use LLM to:

• Expand synonyms
• Add domain terms
• Add abbreviations

Example Prompt (LLM Tools) Expand this query with synonyms and key domain terms: <query>

---

Step 4 — Parameter Tuning

Systematically search:

• k1 = [1.0, 1.2, 1.4, 1.6, 1.8]
• b = [0.45, 0.55, 0.65, 0.75]

Optimize using:

• nDCG@10
• Recall@10
• Precision@5

---

4. BM25 Quality Checklist

• [ ] Field boosts applied
• [ ] Query expansion improves recall
• [ ] k1 and b tuned
• [ ] Stopword handling validated
• [ ] No tokenization inconsistencies

Chunking Patterns for RAG Systems

Chunking is the primary driver of retrieval quality. These patterns provide repeatable, production-ready chunk strategies for different data types.

---

1. Core Chunking Principles

• Preserve semantic boundaries (sections, headings, paragraphs)
• Overlap chunks to avoid context cutting
• Avoid overly small chunks (too noisy)
• Avoid overly large chunks (retrieval dilution)
• Always include metadata (source, URI, heading, page number)

Checklist

• [ ] Chunk size tuned for doc type
• [ ] Overlap defined
• [ ] Metadata preserved
• [ ] No empty/orphaned chunks

---

2. Standard Sliding-Window Pattern

Use for:

• Blogs
• Articles
• Long-form documentation

Parameters:

• Chunk size: 500–800 tokens
• Overlap: 50–150 tokens

Procedure:

1. Clean text 2. Normalize whitespace 3. Slide window with overlap 4. Attach metadata

---

3. Hierarchical Chunking Pattern

Use for structured docs:

• Manuals
• Section-based docs
• Legal documents
• Research papers

Algorithm:

1. Split by top-level headings 2. Inside each section, chunk using sliding window 3. Metadata includes section → subsection → paragraph

Benefits:

• Section-aware retrieval
• Better context grouping

---

4. Code Chunking Pattern (Special Case)

Code behaves differently from natural language.

Rules:

• Chunk by logical blocks, not fixed sizes
• Use syntax-aware splitting:
• Functions
• Classes
• Modules

Recommended:

• Chunk size: 80–200 tokens
• Overlap: 0–20 tokens

Add metadata:

• Programming language
• File path
• Function name

---

5. Table Chunking Pattern

Tables require structural preservation.

Techniques:

• Convert rows to key-value pairs
• Pair headers with row values
• Use row-wise chunks, not cell-wise

Checklist

• [ ] All rows retain column names
• [ ] Numeric fields not concatenated without separators
• [ ] Provide normalized string + structured version

---

6. PDF/Scanned Document Chunking

Steps:

1. Extract text with OCR (if needed) 2. Remove headers/footers 3. Reconstruct paragraphs using layout metadata 4. Chunk 700–1200 token windows

---

7. Chunk Quality Control Checklist

• [ ] Random sample of chunks inspected
• [ ] No broken sentences or stray tokens
• [ ] Metadata consistently applied
• [ ] Chunk count reasonable (no explosion)

Chunking Strategy Selection (Production)

Chunking is a major quality lever in RAG, but there is no universal best chunk size. Choose a baseline strategy, then validate against a test set.

2026 Update: Semantic chunking is now the enterprise default. Late chunking is gaining traction for relationship-heavy documents.

References:

• RAG paper (https://arxiv.org/abs/2005.11401)
• Failure taxonomy (https://arxiv.org/abs/2401.05856)
• Chunking strategies comprehensive guide
• Semantic boundaries reduce RAG errors 60%

---

1. Baseline Decision Rule (Start Simple)

What are you chunking?
    ├─ Structured documents (PDFs with pages/sections)?
    │   └─ Prefer structure-aware chunking (page/section boundaries + metadata)
    │
    ├─ Technical docs / Markdown / API refs?
    │   └─ Prefer header-aware chunking (H1/H2/H3 boundaries + code fences)
    │
    ├─ Source code?
    │   └─ Prefer syntax-aware chunking (symbols, functions, classes) + file path metadata
    │
    └─ Unstructured text?
        └─ Start with fixed-size token chunks + overlap, then tune

---

2. Strategy Table (Operational Defaults)

---

3. Validation Protocol (REQUIRED)

Build a chunking test set

• 50–200 queries representative of production traffic.
• For each query, record:
• expected sources (doc IDs, sections, pages)
• unacceptable sources (near-miss docs that are commonly retrieved but wrong)

Measure retrieval separately from generation

• Retrieval metrics: recall@k, nDCG/MRR, empty-result rate, latency.
• Generation metrics: citation coverage, groundedness/faithfulness, refusal correctness.

Iterate with one variable at a time

• Change only chunking (hold embedder/index/reranker constant), re-run test set, then decide.

---

4. Anti-Patterns (AVOID)

• Over-chunking (tiny chunks): high recall but poor synthesis and high latency.
• Under-chunking (huge chunks): high cost and more irrelevant context.
• Dropping structure metadata: no stable citations and poor debugging.
• Mixing tenants/corpora without ACL metadata: security and correctness failures.

---

5. Semantic Chunking (2026 Enterprise Default)

What it is: Split documents at semantically meaningful boundaries by comparing sentence embeddings, not arbitrary token counts.

Why it matters: IBM research shows 20-30% reduction in irrelevant retrieval vs fixed-size. Enterprise adoption accelerated in 2025-2026.

How It Works

Semantic Chunking Pipeline:
  1. Split document into sentences
  2. Generate embedding for each sentence
  3. Calculate similarity between adjacent sentences
  4. Identify breakpoints where similarity drops significantly
  5. Group sentences between breakpoints into chunks

Boundary Detection Methods

Cost Considerations

Semantic chunking has hidden costs:

• Embedding computation: Generate embeddings for every sentence to detect boundaries
• Ingestion overhead: 15-40% longer ingestion time vs fixed-size
• API costs: For 1GB dataset, can mean millions of embedding calls

Recommendation: Use for high-value corpora where retrieval quality justifies cost. For large, low-value corpora, fixed-size may be more practical.

Implementation Libraries

---

6. Late Chunking (Emerging 2026)

What it is: Embed the full document first (preserving context), then chunk the embeddings. Opposite of traditional "chunk then embed."

Why it matters: Solves the "relationship problem" where meaning depends on surrounding sections. The "only if" clause on page 3 that modifies the statement on page 1.

Traditional vs Late Chunking

Traditional (Chunk-Then-Embed):
  Document → Split into chunks → Embed each chunk independently
  Problem: Each chunk loses context from the rest of the document

Late Chunking (Embed-Then-Chunk):
  Document → Embed full document (model sees everything) →
  Split embeddings into chunks (preserving contextual understanding)
  Benefit: Each chunk embedding "knows" about the whole document

When to Use Late Chunking

Good candidates:

• Legal contracts (clauses reference each other)
• Technical specifications (definitions on page 1, usage throughout)
• Research papers (methods section context needed for results)
• Policy documents (exceptions and conditions scattered)

Poor candidates:

• Independent FAQ entries
• Product descriptions (self-contained)
• News articles (mostly self-contained paragraphs)

Trade-offs

Implementation

# Conceptual example - Chonkie library
from chonkie import LateChunker

chunker = LateChunker(
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    chunk_size=512,
    # Model processes full document before chunking
)

chunks = chunker.chunk(document_text)
# Each chunk embedding has full document context

Library: Chonkie - Built around the idea that chunking is not one generic operation.

---

7. Chunking Strategy Decision Tree (2026)

Choosing a chunking strategy:
  │
  ├─ Document structure?
  │   ├─ Strong structure (headers, sections) → Structure-aware chunking
  │   ├─ Code → Syntax-aware chunking
  │   └─ Unstructured prose → Continue below
  │
  ├─ Relationship density?
  │   ├─ High (legal, specs, cross-references) → Late chunking or Contextual Retrieval
  │   └─ Low (independent paragraphs) → Continue below
  │
  ├─ Retrieval quality critical?
  │   ├─ Yes (high-stakes domain) → Semantic chunking
  │   └─ No (general use) → Fixed-size with overlap
  │
  └─ Cost constraints?
      ├─ Tight budget → Fixed-size (cheapest)
      └─ Quality over cost → Semantic + reranking

---

8. Implementation Examples

See ../assets/chunking/:

• ../assets/chunking/template-basic-chunking.md
• ../assets/chunking/template-code-chunking.md
• ../assets/chunking/template-long-doc-chunking.md

Chunk Context Augmentation (Contextual Retrieval)

Chunk context augmentation adds a lightweight, generated "header" to each chunk before indexing. This can improve retrieval for entities/time periods that are not explicit inside isolated chunks.

Reference (popularized): Anthropic "Contextual Retrieval" (https://www.anthropic.com/news/contextual-retrieval).

---

When to Use

• Multi-entity corpora where chunks frequently omit the subject (company/product/user).
• Documents with temporal structure (quarters, versions, dates) where chunk-local text is ambiguous.
• Large reports/manuals where headings carry meaning that chunks lose.

Avoid when:

• Your corpus already has strong structure-aware metadata (titles, headings, section paths) and retrieval is good.
• You cannot validate the impact with an evaluation set.

---

Core Idea

Store two representations:

• Raw chunk (for citations and display)
• Augmented chunk = generated context + raw chunk (for embedding / indexing)

The generated context should be:

• short (a few sentences)
• factual (derived only from the parent document/section)
• stable (deterministic prompt + constrained output)

---

Implementation Pattern (Pseudocode)

for each document:
  parse → structured sections (title/headings/page)
  for each chunk:
    metadata_context = {title, heading_path, page, section_id}
    generated_context = LLM(document_context + metadata_context + chunk_text)
    augmented_text = generated_context + "\n\n" + chunk_text
    embed/index augmented_text with metadata pointing to raw chunk

---

Validation Protocol (REQUIRED)

• Hold out a retrieval test set (queries + expected sources).
• Compare baseline vs augmented:
• recall@k / nDCG
• empty-result rate
• latency and index size changes
• If you also use reranking, test:
• baseline retrieval + rerank
• augmented retrieval + rerank

---

Failure Modes (AVOID)

• Hallucinated context that introduces incorrect entities/times.
• Context that includes sensitive data that should not be indexed or cached.
• Overly long context that bloats embeddings and increases latency/cost.

Mitigations:

• Use strict prompts + output length caps.
• Validate context format; reject empty or non-compliant outputs.
• Log and sample augmented chunks for review.

Distributed Search Operations & SLOs

Operational patterns for running search at scale with reliability and performance guarantees.

---

When to Use

Apply these patterns when:

• Search serves production traffic
• Multi-shard/multi-replica deployments
• SLO requirements (latency, availability)
• Need resilience to failures
• Hot/cold tier storage architectures

---

Pattern 1: Topology & Consistency

Shard & Replica Design

Sharding strategy:

# Example: Hash-based sharding
def shard_assignment(doc_id, num_shards=8):
    """
    Assign document to shard based on hash
    """
    return hash(doc_id) % num_shards

# Shard configuration
shard_config = {
    'num_shards': 8,
    'replicas_per_shard': 3,
    'placement': {
        'shard_0': ['node-1', 'node-2', 'node-3'],
        'shard_1': ['node-2', 'node-3', 'node-4'],
        # ... etc
    }
}

Consistency models:

Checklist

• [ ] Shard/replica placement documented
• [ ] Consistency model chosen based on use case
• [ ] Cross-datacenter replication configured (if needed)
• [ ] Quorum reads/writes configured for strong consistency

---

Pattern 2: Resilience & Health Checks

Health Monitoring

class SearchClusterHealth:
    def check_health(self):
        """
        Monitor cluster health
        """
        health = {
            'shards': self.check_shard_health(),
            'replicas': self.check_replica_lag(),
            'query_performance': self.check_query_latency(),
            'ingestion_lag': self.check_ingestion_lag()
        }

        return health

    def check_shard_health(self):
        """
        Verify all shards are reachable
        """
        unhealthy_shards = []
        for shard_id in range(self.num_shards):
            if not self.ping_shard(shard_id):
                unhealthy_shards.append(shard_id)

        return {
            'healthy': len(unhealthy_shards) == 0,
            'unhealthy_shards': unhealthy_shards
        }

    def check_replica_lag(self):
        """
        Monitor replica lag behind primary
        """
        max_lag_threshold = 60  # seconds

        lags = {}
        for shard_id in range(self.num_shards):
            primary_version = self.get_primary_version(shard_id)
            replicas = self.get_replicas(shard_id)

            for replica in replicas:
                lag = primary_version - replica['version']
                if lag > max_lag_threshold:
                    lags[f'shard_{shard_id}_replica_{replica["id"]}'] = lag

        return {
            'within_threshold': len(lags) == 0,
            'lagging_replicas': lags
        }

Automatic Reroute on Failure

def execute_query_with_failover(query, primary_shard, replica_shards):
    """
    Query with automatic failover to replicas
    """
    # Try primary first
    try:
        return query_shard(primary_shard, query, timeout=100)
    except (TimeoutError, ConnectionError):
        # Failover to replicas
        for replica in replica_shards:
            try:
                return query_shard(replica, query, timeout=100)
            except Exception:
                continue

        # All replicas failed
        raise SearchUnavailableError("All replicas failed for shard")

Checklist

• [ ] Health checks run every 10-30 seconds
• [ ] Replica lag thresholds configured (< 60s)
• [ ] Automatic reroute on shard failure
• [ ] Circuit breaker for unhealthy shards

---

Pattern 3: Backpressure & Load Shedding

Queue Depth Monitoring

class BackpressureController:
    def __init__(self, max_queue_depth=1000, latency_threshold=500):
        self.max_queue_depth = max_queue_depth
        self.latency_threshold = latency_threshold  # ms
        self.current_queue_depth = 0

    def should_accept_query(self):
        """
        Reject queries when overloaded
        """
        # Check queue depth
        if self.current_queue_depth > self.max_queue_depth:
            return False, "Queue full"

        # Check latency
        current_latency = self.get_p95_latency()
        if current_latency > self.latency_threshold:
            return False, "Latency SLO breach"

        return True, "OK"

    def execute_with_backpressure(self, query):
        """
        Execute query with backpressure control
        """
        accept, reason = self.should_accept_query()

        if not accept:
            # Shed load
            return {
                'error': 'Service overloaded',
                'reason': reason,
                'retry_after': 5  # seconds
            }

        # Accept query
        self.current_queue_depth += 1
        try:
            result = self.execute_query(query)
            return result
        finally:
            self.current_queue_depth -= 1

Load Shedding Strategies

def load_shedding_strategy(query, qps_limit=1000):
    """
    Prioritize queries during overload
    """
    current_qps = get_current_qps()

    if current_qps < qps_limit:
        # Under limit, accept all
        return True

    # Over limit, prioritize
    priority = classify_query_priority(query)

    if priority == 'critical':
        return True  # Always accept critical queries
    elif priority == 'high':
        return random.random() < 0.5  # Accept 50% of high-priority
    else:
        return False  # Reject low-priority

def classify_query_priority(query):
    """
    Classify query priority based on characteristics
    """
    if query.get('user_type') == 'premium':
        return 'critical'
    elif query.get('source') == 'internal_tool':
        return 'high'
    else:
        return 'normal'

Checklist

• [ ] Queue depth monitoring active
• [ ] Backpressure triggers configured (queue, latency)
• [ ] Load shedding policy defined
• [ ] Rate limiting per client/tenant

---

Pattern 4: Caching Strategy

Multi-Level Caching

class SearchCacheManager:
    def __init__(self):
        # L1: Result cache (short TTL)
        self.result_cache = LRUCache(max_size=10000, ttl=300)  # 5 min

        # L2: Embedding cache (longer TTL)
        self.embedding_cache = LRUCache(max_size=100000, ttl=3600)  # 1 hour

        # L3: Hot query cache (very short TTL)
        self.hot_query_cache = LRUCache(max_size=1000, ttl=60)  # 1 min

    def search_with_cache(self, query, k=10):
        """
        Multi-level cache lookup
        """
        cache_key = f"{query}:{k}"

        # L3: Hot query cache
        cached_result = self.hot_query_cache.get(cache_key)
        if cached_result:
            return cached_result, 'hot_cache_hit'

        # L1: Result cache
        cached_result = self.result_cache.get(cache_key)
        if cached_result:
            return cached_result, 'result_cache_hit'

        # L2: Embedding cache (avoid re-encoding)
        query_embedding = self.embedding_cache.get(query)
        if not query_embedding:
            query_embedding = self.encoder.encode(query)
            self.embedding_cache.put(query, query_embedding)

        # Execute search
        results = self.vector_index.search(query_embedding, k=k)

        # Cache results
        self.result_cache.put(cache_key, results)

        # If query is hot (seen recently), cache in L3
        if self.is_hot_query(query):
            self.hot_query_cache.put(cache_key, results)

        return results, 'cache_miss'

Cache Invalidation

def invalidate_on_index_change(index_version):
    """
    Invalidate caches when index changes
    """
    global current_index_version

    if index_version != current_index_version:
        # Clear all result caches
        result_cache.clear()
        hot_query_cache.clear()

        # Embedding cache can persist (model hasn't changed)

        current_index_version = index_version
        log_event("Cache invalidated due to index change")

Checklist

• [ ] Multi-level cache configured (results, embeddings, hot queries)
• [ ] Cache hit rates monitored (target: >60% for hot queries)
• [ ] Invalidation on index change automated
• [ ] TTLs tuned based on data freshness requirements

---

Pattern 5: Performance Runbook

Metrics to Track

search_slos = {
    'latency': {
        'p50': 100,   # ms
        'p95': 300,   # ms
        'p99': 800    # ms
    },
    'availability': 0.999,  # 99.9%
    'relevance': {
        'ndcg@10': 0.75,
        'mrr': 0.80
    },
    'throughput': {
        'qps': 1000
    }
}

Incident Playbook

class SearchIncidentPlaybook:
    """
    Automated responses to SLO breaches
    """
    def handle_latency_spike(self, current_p95):
        """
        Response to latency SLO breach
        """
        actions = []

        # Action 1: Reduce K (retrieve fewer candidates)
        if current_p95 > 500:
            actions.append("Reduce K from 20 to 10")
            self.config.update({'k': 10})

        # Action 2: Disable reranking temporarily
        if current_p95 > 800:
            actions.append("Disable reranking temporarily")
            self.config.update({'reranking_enabled': False})

        # Action 3: Fall back to cache-only mode
        if current_p95 > 1500:
            actions.append("Fallback to cache-only mode")
            self.config.update({'cache_only_mode': True})

        return actions

    def handle_availability_drop(self, current_availability):
        """
        Response to availability SLO breach
        """
        actions = []

        # Check shard health
        unhealthy = self.check_shard_health()

        if len(unhealthy) > 0:
            actions.append(f"Reroute traffic from shards: {unhealthy}")
            self.reroute_traffic(exclude_shards=unhealthy)

        # Scale up if at capacity
        if self.get_cpu_usage() > 0.8:
            actions.append("Scale up replicas")
            self.scale_replicas(target_count=self.current_replicas + 2)

        return actions

Checklist

• [ ] SLO targets defined (latency, availability, relevance)
• [ ] Metrics tracked per-query with QPS context
• [ ] Incident playbook automated (reduce K, disable rerank, fallback to cache)
• [ ] Runbook tested in staging environment

---

Pattern 6: Upgrades & Rollbacks

Dual-Write During Rebuild

class DualIndexManager:
    """
    Manage dual indexes during upgrades
    """
    def __init__(self):
        self.primary_index = 'index-v1'
        self.shadow_index = 'index-v2'
        self.rollout_percentage = 0  # % of traffic to shadow index

    def write_dual(self, document):
        """
        Write to both indexes during migration
        """
        self.write_to_index(self.primary_index, document)
        self.write_to_index(self.shadow_index, document)

    def read_with_rollout(self, query):
        """
        Gradual rollout to new index
        """
        if random.random() < self.rollout_percentage / 100:
            # Read from shadow index
            result = self.read_from_index(self.shadow_index, query)
            result['index_version'] = self.shadow_index
            return result
        else:
            # Read from primary index
            result = self.read_from_index(self.primary_index, query)
            result['index_version'] = self.primary_index
            return result

    def increase_rollout(self, step=10):
        """
        Gradual rollout in 10% increments
        """
        self.rollout_percentage = min(100, self.rollout_percentage + step)
        log_event(f"Rollout increased to {self.rollout_percentage}%")

    def rollback(self):
        """
        Rollback to primary index
        """
        self.rollout_percentage = 0
        log_event("Rolled back to primary index")

Version Tagging

def tag_query_response(query_id, index_version, model_version):
    """
    Tag responses with versions for debugging
    """
    response_metadata = {
        'query_id': query_id,
        'index_version': index_version,
        'model_version': model_version,
        'timestamp': datetime.now().isoformat()
    }

    return response_metadata

Checklist

• [ ] Dual-write/dual-read during index rebuilds
• [ ] Gradual rollout (10% → 25% → 50% → 100%)
• [ ] Query/response tagged with index/model version
• [ ] Rollback path tested and rehearsed
• [ ] Metrics compared between old/new index

---

Distributed Search SLO Checklist

• [ ] Shard/replica + failover plan documented
• [ ] Consistency model chosen and configured
• [ ] Health checks + replica lag monitoring active
• [ ] Backpressure/load shedding configured
• [ ] Multi-level caching with invalidation on index change
• [ ] Cache hit rates monitored (target: >60%)
• [ ] Relevance + latency metrics tracked with QPS
• [ ] Incident playbook automated (reduce K, disable rerank, fallback)
• [ ] Dual-index upgrades with gradual rollout
• [ ] Rollback path tested and ready

Embedding Model Selection Guide

Operational guide for choosing, deploying, and managing embedding models for RAG and search. Covers model comparison, dimensionality tradeoffs, fine-tuning, batch pipelines, versioning, assessment, and cost analysis. Focus on production decisions, not architecture theory.

Freshness anchor: January 2026 — OpenAI text-embedding-3-*, Cohere embed-v3, Voyage AI voyage-3, Jina v3, MTEB leaderboard current

---

Decision Tree: Choosing an Embedding Model

START
│
├─ Budget constraint?
│   ├─ Zero cost (open source only)
│   │   ├─ English only → all-MiniLM-L6-v2 (fast) or gte-large-en-v1.5 (accurate)
│   │   ├─ Multilingual → multilingual-e5-large or BGE-M3
│   │   └─ Domain-specific → Fine-tune sentence-transformers base
│   │
│   └─ API budget available → Continue
│
├─ Latency requirement?
│   ├─ Real-time (< 50ms per query)
│   │   ├─ API → OpenAI text-embedding-3-small (fast, cheap)
│   │   └─ Self-hosted → all-MiniLM-L6-v2 (CPU viable)
│   │
│   └─ Batch / offline → Any model (latency not critical)
│
├─ Quality requirement?
│   ├─ Best available (enterprise search, legal, medical)
│   │   ├─ API → Voyage AI voyage-3 or Cohere embed-v3 (English)
│   │   ├─ API multilingual → Cohere embed-v3 or OpenAI text-embedding-3-large
│   │   └─ Self-hosted → BGE-large-en-v1.5 or NV-Embed-v2
│   │
│   └─ Good enough (chatbot, FAQ, basic search)
│       └─ OpenAI text-embedding-3-small or all-MiniLM-L6-v2
│
├─ Domain-specific needs?
│   ├─ Code → Voyage Code 3 or CodeBERT fine-tune
│   ├─ Legal → Fine-tune on legal corpus
│   ├─ Medical → PubMedBERT fine-tune or Voyage AI
│   └─ General → Standard models
│
└─ Dimensionality constraint?
    ├─ Storage limited → Matryoshka models (truncate to 256-512)
    └─ No constraint → Full dimensions (1024-3072)

---

Quick Reference: Model Comparison (Q1 2026)

• Prices are per 1M input tokens — always check current pricing
• MTEB scores are approximate averages across retrieval tasks

---

Operational Patterns

Pattern 1: Dimensionality Tradeoffs with Matryoshka

• Use when: Need to balance storage cost vs retrieval quality
• Concept: Matryoshka models produce embeddings where the first N dimensions are independently useful

# OpenAI text-embedding-3 supports dimension reduction natively
from openai import OpenAI
client = OpenAI()

# Full dimensions (best quality)
response = client.embeddings.create(
    model="text-embedding-3-large",
    input="search query",
    dimensions=3072,  # full
)

# Reduced dimensions (cheaper storage, slightly lower quality)
response_small = client.embeddings.create(
    model="text-embedding-3-large",
    input="search query",
    dimensions=256,   # 12x storage savings
)

# For sentence-transformers Matryoshka models:
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5")
embeddings_full = model.encode(texts)           # 768 dims
embeddings_256 = embeddings_full[:, :256]        # truncate to 256
# Normalize after truncation
embeddings_256 = embeddings_256 / np.linalg.norm(embeddings_256, axis=1, keepdims=True)

• Dimension vs quality tradeoff (typical):

Pattern 2: Domain-Specific Fine-Tuning

• Use when: General models underperform on domain-specific retrieval
• Implementation:

from sentence_transformers import SentenceTransformer, losses, InputExample
from torch.utils.data import DataLoader

# Step 1: Prepare training data (query, positive_passage, negative_passage)
train_examples = [
    InputExample(texts=["search query", "relevant passage", "irrelevant passage"]),
    # ...
]

# Step 2: Fine-tune with triplet loss
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32)
train_loss = losses.TripletLoss(model=model)

model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=3,
    warmup_steps=100,
    output_path="./fine-tuned-embeddings",
    show_progress_bar=True,
)

# Step 3: Assess on holdout
from sentence_transformers.evaluation import InformationRetrievalEvaluator
ir_evaluator = InformationRetrievalEvaluator(queries, corpus, relevant_docs)
results = ir_evaluator(model)

• Fine-tuning data requirements:

• Hard negative mining (critical for quality):

def mine_hard_negatives(model, queries, corpus, k=10):
    """Find hard negatives: high similarity but irrelevant."""
    corpus_embeddings = model.encode(list(corpus.values()))
    query_embeddings = model.encode(queries)

    # Top-k most similar but not relevant
    hard_negatives = {}
    for i, query in enumerate(queries):
        scores = cosine_similarity([query_embeddings[i]], corpus_embeddings)[0]
        top_k_indices = np.argsort(scores)[-k:][::-1]
        # Filter out actual positives
        negatives = [idx for idx in top_k_indices if idx not in relevant_docs[query]]
        hard_negatives[query] = negatives

    return hard_negatives

Pattern 3: Batch Embedding Pipeline

• Use when: Embedding large corpus (initial indexing or re-indexing)
• Implementation:

import asyncio
from openai import AsyncOpenAI
from tenacity import retry, wait_exponential, stop_after_attempt

client = AsyncOpenAI()

@retry(wait=wait_exponential(min=1, max=60), stop=stop_after_attempt(5))
async def embed_batch(texts, model="text-embedding-3-small"):
    """Embed a batch of texts with retry logic."""
    response = await client.embeddings.create(model=model, input=texts)
    return [e.embedding for e in response.data]

async def embed_corpus(texts, batch_size=100, max_concurrent=10):
    """Embed entire corpus with batching and rate limiting."""
    semaphore = asyncio.Semaphore(max_concurrent)
    results = [None] * len(texts)

    async def process_batch(start_idx, batch):
        async with semaphore:
            embeddings = await embed_batch(batch)
            for i, emb in enumerate(embeddings):
                results[start_idx + i] = emb

    tasks = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        tasks.append(process_batch(i, batch))

    await asyncio.gather(*tasks)
    return results

# Cost estimation before running
def estimate_embedding_cost(texts, model="text-embedding-3-small"):
    import tiktoken
    enc = tiktoken.encoding_for_model(model)
    total_tokens = sum(len(enc.encode(t)) for t in texts)
    cost_per_1m = {'text-embedding-3-small': 0.02, 'text-embedding-3-large': 0.13}
    estimated_cost = (total_tokens / 1_000_000) * cost_per_1m.get(model, 0.10)
    return {'total_tokens': total_tokens, 'estimated_cost_usd': estimated_cost}

Pattern 4: Embedding Versioning and Migration

• Use when: Upgrading embedding model in production
• Migration strategy:

NEVER do a big-bang migration. Use dual-write pattern:

1. Deploy new model alongside old
2. Write new embeddings to separate index
3. A/B test retrieval quality (shadow mode)
4. If quality >= old model: swap primary index
5. Keep old index for 30 days (rollback safety)
6. Delete old index

class EmbeddingVersionManager:
    """Manage embedding model versions with zero-downtime migration."""

    def __init__(self):
        self.models = {}
        self.active_version = None

    def register_model(self, version, model_name, index_name):
        self.models[version] = {
            'model': model_name,
            'index': index_name,
            'created': datetime.utcnow(),
        }

    def dual_write(self, text, versions):
        """Write embeddings to multiple indices."""
        for version in versions:
            model = self.models[version]
            embedding = encode(text, model['model'])
            write_to_index(model['index'], embedding)

    def migrate(self, from_version, to_version, corpus):
        """Re-embed entire corpus for new model."""
        new_model = self.models[to_version]
        batch_reindex(corpus, new_model['model'], new_model['index'])

Pattern 5: Quality Assessment (MTEB and Custom)

• Use when: Choosing between models or validating fine-tuned model
• Implementation:

# Option A: MTEB benchmark (standardized)
# pip install mteb
from mteb import MTEB
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5")
benchmark = MTEB(tasks=["NFCorpus", "SciFact", "ArguAna"])
results = benchmark.run(model, output_folder="results/bge-large")

# Option B: Custom domain retrieval assessment
def assess_retrieval_quality(model, queries, corpus, relevant_docs, k=10):
    """Measure retrieval quality on your data."""
    query_embeddings = model.encode(queries)
    corpus_embeddings = model.encode(list(corpus.values()))

    metrics = {'recall@5': [], 'recall@10': [], 'mrr': [], 'ndcg@10': []}

    for i, query in enumerate(queries):
        scores = cosine_similarity([query_embeddings[i]], corpus_embeddings)[0]
        top_k = np.argsort(scores)[-k:][::-1]
        relevant = set(relevant_docs[query])

        # Recall@K
        for cutoff in [5, 10]:
            retrieved = set(top_k[:cutoff])
            metrics[f'recall@{cutoff}'].append(len(retrieved & relevant) / len(relevant))

        # MRR
        for rank, idx in enumerate(top_k, 1):
            if idx in relevant:
                metrics['mrr'].append(1 / rank)
                break
        else:
            metrics['mrr'].append(0)

    return {k: np.mean(v) for k, v in metrics.items()}

• Quality targets:

---

Cost Comparison Calculator

def compare_embedding_costs(num_documents, avg_tokens_per_doc, models=None):
    """Compare embedding costs across providers."""
    if models is None:
        models = {
            'openai-small': {'cost_per_1m': 0.02, 'dims': 1536},
            'openai-large': {'cost_per_1m': 0.13, 'dims': 3072},
            'cohere-v3': {'cost_per_1m': 0.10, 'dims': 1024},
            'voyage-3': {'cost_per_1m': 0.06, 'dims': 1024},
            'voyage-3-lite': {'cost_per_1m': 0.02, 'dims': 512},
            'self-hosted-bge': {'cost_per_1m': 0.005, 'dims': 1024},  # GPU amortized
        }

    total_tokens = num_documents * avg_tokens_per_doc
    results = []

    for name, info in models.items():
        embed_cost = (total_tokens / 1_000_000) * info['cost_per_1m']
        storage_gb = (num_documents * info['dims'] * 4) / (1024**3)  # float32
        results.append({
            'model': name,
            'embedding_cost': f"${embed_cost:.2f}",
            'storage_gb': f"{storage_gb:.2f}",
            'dimensions': info['dims'],
        })

    return pd.DataFrame(results)

---

Anti-Patterns

---

Validation Checklist

• [ ] Model assessed on domain-specific data (not just MTEB)
• [ ] Dimensionality chosen based on quality/cost tradeoff
• [ ] Cost estimated for full corpus embedding and ongoing queries
• [ ] Batch pipeline handles retries, rate limits, and progress tracking
• [ ] Embedding versioning strategy defined (dual-write migration)
• [ ] Normalization applied consistently
• [ ] Retrieval metrics tracked (recall@K, MRR, NDCG)
• [ ] Fine-tuning considered if recall@10 < 0.70 on domain data
• [ ] API fallback strategy defined for production
• [ ] Re-embedding schedule planned (model upgrades, corpus growth)

---

Cross-References

• ai-rag/references/rag-caching-patterns.md — caching embeddings to reduce API calls
• ai-rag/references/graph-rag-patterns.md — embeddings for hybrid graph+vector search
• ai-mlops/references/cost-management-finops.md — tracking embedding costs
• ai-mlops/references/experiment-tracking-patterns.md — logging embedding model experiments

Graph RAG Patterns

Operational guide for graph-based retrieval-augmented generation. Covers knowledge graph construction, entity extraction, hybrid graph+vector retrieval, Microsoft GraphRAG patterns, and when graph RAG outperforms flat vector search. Focus on implementation and production decisions.

Freshness anchor: January 2026 — Neo4j 5.x, LangChain 0.3+, LlamaIndex 0.11+, Microsoft GraphRAG 1.x

---

Decision Tree: When to Use Graph RAG

START
│
├─ What kind of questions will users ask?
│   ├─ Factual lookup ("What is X?")
│   │   └─ Standard vector RAG is sufficient
│   │
│   ├─ Relational ("How is X related to Y?")
│   │   └─ Graph RAG strongly recommended
│   │
│   ├─ Multi-hop ("What companies does X's advisor also advise?")
│   │   └─ Graph RAG required — vector search cannot traverse
│   │
│   ├─ Aggregation ("What are the main themes across all documents?")
│   │   └─ Microsoft GraphRAG (community summaries)
│   │
│   └─ Comparison ("How do X and Y differ?")
│       └─ Graph RAG helpful (entity-pair retrieval)
│
├─ Corpus characteristics?
│   ├─ Highly structured (legal, medical, financial)
│   │   └─ Graph RAG — entities and relationships are well-defined
│   │
│   ├─ Loosely structured (blog posts, documentation)
│   │   └─ Vector RAG usually sufficient; graph adds marginal value
│   │
│   └─ Mixed (structured + unstructured)
│       └─ Hybrid graph + vector
│
└─ Maintenance budget?
    ├─ Low → Vector RAG only (graph requires ongoing maintenance)
    ├─ Medium → Graph RAG with automated entity extraction
    └─ High → Full knowledge graph with manual curation + automated updates

---

Quick Reference: Graph RAG vs Vector RAG

---

Operational Patterns

Pattern 1: Knowledge Graph Construction

• Use when: Building a graph from unstructured text
• Pipeline:

Documents → Chunking → Entity Extraction → Relationship Extraction
    → Entity Resolution → Graph Storage → Index Creation

• Implementation:

from langchain_community.graphs import Neo4jGraph
from langchain_openai import ChatOpenAI
from langchain.chains import GraphCypherQAChain

# Step 1: Entity and relationship extraction via LLM
EXTRACTION_PROMPT = """
Extract entities and relationships from the following text.
Return as JSON with format:
{
  "entities": [{"name": "...", "type": "...", "properties": {...}}],
  "relationships": [{"source": "...", "target": "...", "type": "...", "properties": {...}}]
}

Entity types: Person, Organization, Product, Technology, Location, Event
Relationship types: WORKS_AT, FOUNDED, ACQUIRED, PARTNERS_WITH, USES, LOCATED_IN

Text: {text}
"""

def extract_entities_and_relationships(text, llm):
    """Extract structured knowledge from text chunk."""
    response = llm.invoke(EXTRACTION_PROMPT.format(text=text))
    return parse_json_response(response.content)

# Step 2: Entity resolution (deduplicate)
def resolve_entities(entities):
    """Merge duplicate entities with fuzzy matching."""
    from rapidfuzz import fuzz

    resolved = []
    for entity in entities:
        matched = False
        for existing in resolved:
            if (existing['type'] == entity['type'] and
                fuzz.ratio(existing['name'].lower(), entity['name'].lower()) > 85):
                # Merge properties
                existing['properties'].update(entity['properties'])
                existing['aliases'] = existing.get('aliases', []) + [entity['name']]
                matched = True
                break
        if not matched:
            resolved.append(entity)
    return resolved

# Step 3: Store in Neo4j
def store_in_neo4j(graph, entities, relationships, source_doc):
    """Write extracted knowledge to Neo4j."""
    for entity in entities:
        graph.query("""
            MERGE (e:{type} {{name: $name}})
            SET e += $properties
            SET e.source_docs = coalesce(e.source_docs, []) + [$source]
        """.format(type=entity['type']),
        params={
            'name': entity['name'],
            'properties': entity['properties'],
            'source': source_doc,
        })

    for rel in relationships:
        graph.query("""
            MATCH (s {{name: $source}})
            MATCH (t {{name: $target}})
            MERGE (s)-[r:{type}]->(t)
            SET r += $properties
        """.format(type=rel['type']),
        params={
            'source': rel['source'],
            'target': rel['target'],
            'properties': rel.get('properties', {}),
        })

Pattern 2: Microsoft GraphRAG (Community Summaries)

• Use when: Need global queries ("What are the main themes?") or large corpus overview
• Concept: Build graph, detect communities (Leiden algorithm), summarize each community

# Step 1: Initialize
graphrag init --root ./ragproject

# Step 2: Configure settings.yaml (llm, chunks, entity_extraction, embeddings)

# Step 3: Index (builds graph + communities + summaries)
graphrag index --root ./ragproject

# Step 4: Query
graphrag query --root ./ragproject --method local --query "What is Company X's strategy?"
graphrag query --root ./ragproject --method global --query "What are the main industry trends?"

• When to use Local vs Global:

Pattern 3: Hybrid Graph + Vector Retrieval

• Use when: Need both semantic similarity and structural traversal
• Implementation:

from neo4j import GraphDatabase
import numpy as np

class HybridGraphVectorRetriever:
    """Combine vector similarity with graph traversal."""

    def __init__(self, neo4j_driver, embedding_model):
        self.driver = neo4j_driver
        self.embedder = embedding_model

    def retrieve(self, query, k_vector=10, k_graph=10, hop_depth=2):
        """
        Step 1: Vector search for relevant chunks
        Step 2: Extract entities from those chunks
        Step 3: Traverse graph from those entities
        Step 4: Merge and rank results
        """
        query_embedding = self.embedder.encode(query)

        # Step 1: Vector similarity search
        vector_results = self._vector_search(query_embedding, k=k_vector)

        # Step 2: Extract entities from vector results
        entity_names = self._extract_entities_from_chunks(vector_results)

        # Step 3: Graph traversal from entities
        graph_context = self._traverse_graph(entity_names, depth=hop_depth)

        # Step 4: Merge
        combined_context = self._merge_contexts(vector_results, graph_context)

        return combined_context

    def _vector_search(self, embedding, k):
        """Neo4j vector index search."""
        with self.driver.session() as session:
            result = session.run("""
                CALL db.index.vector.queryNodes('chunk_embeddings', $k, $embedding)
                YIELD node, score
                RETURN node.text AS text, node.source AS source, score
            """, k=k, embedding=embedding.tolist())
            return [dict(record) for record in result]

    def _traverse_graph(self, entity_names, depth):
        """Multi-hop graph traversal from seed entities."""
        with self.driver.session() as session:
            result = session.run("""
                UNWIND $entities AS entity_name
                MATCH (e {name: entity_name})
                CALL apoc.path.subgraphAll(e, {
                    maxLevel: $depth,
                    relationshipFilter: '>',
                    limit: 50
                })
                YIELD nodes, relationships
                RETURN nodes, relationships
            """, entities=entity_names, depth=depth)
            return self._format_graph_context(result)

    def _merge_contexts(self, vector_results, graph_context):
        """Interleave vector and graph results."""
        # Vector results: direct semantic matches
        # Graph results: structurally connected entities and relationships
        context_parts = []
        context_parts.append("## Relevant passages\n")
        for vr in vector_results[:5]:
            context_parts.append(f"- {vr['text']}")

        context_parts.append("\n## Related entities and relationships\n")
        context_parts.append(graph_context)

        return "\n".join(context_parts)

Pattern 4: Entity-Aware Chunking

• Use when: Standard chunking breaks entities across chunks
• Implementation:

def entity_aware_chunking(text, max_chunk_size=1000, overlap=100):
    """Chunk text while keeping entity mentions intact."""
    import spacy
    nlp = spacy.load("en_core_web_sm")
    doc = nlp(text)

    # Identify entity spans
    entity_spans = [(ent.start_char, ent.end_char) for ent in doc.ents]

    # Split on paragraph boundaries, respecting entity spans
    chunks = []
    current_chunk_start = 0

    for i, char in enumerate(text):
        if i - current_chunk_start >= max_chunk_size:
            # Find nearest paragraph break that doesn't split an entity
            split_point = find_safe_split(text, i, entity_spans)
            chunks.append(text[current_chunk_start:split_point])
            current_chunk_start = max(current_chunk_start, split_point - overlap)

    if current_chunk_start < len(text):
        chunks.append(text[current_chunk_start:])

    # Tag each chunk with its entities
    tagged_chunks = []
    for chunk in chunks:
        chunk_doc = nlp(chunk)
        entities = [(ent.text, ent.label_) for ent in chunk_doc.ents]
        tagged_chunks.append({
            'text': chunk,
            'entities': entities,
        })

    return tagged_chunks

Pattern 5: Subgraph Context Packing

• Use when: Packing retrieved graph context into LLM prompt efficiently
• Implementation:

def pack_subgraph_context(entities, relationships, max_tokens=2000):
    """
    Format graph context for LLM consumption.
    Priority: directly relevant entities > 1-hop > 2-hop
    """
    context_lines = []

    # Tier 1: Core entities (directly matched)
    context_lines.append("### Key Entities")
    for entity in entities[:10]:
        props = ", ".join(f"{k}: {v}" for k, v in entity.get('properties', {}).items())
        context_lines.append(f"- **{entity['name']}** ({entity['type']}): {props}")

    # Tier 2: Relationships
    context_lines.append("\n### Relationships")
    for rel in relationships[:20]:
        context_lines.append(
            f"- {rel['source']} --[{rel['type']}]--> {rel['target']}"
        )

    # Tier 3: Trim to token budget
    context = "\n".join(context_lines)
    if estimate_tokens(context) > max_tokens:
        context = truncate_to_tokens(context, max_tokens)

    return context

• Context format comparison:

Pattern 6: Graph Maintenance and Updates

• Use when: Keeping knowledge graph fresh as documents change
• Operations:
• New document: chunk → extract entities/relationships → resolve against existing graph → store
• Updated document: remove old extractions by source doc ID → re-extract
• Deleted document: remove entities/relationships where this was the only source doc
• Key rule: Track source_docs array on every node — only delete nodes with zero remaining sources

---

Anti-Patterns

---

Validation Checklist

• [ ] Use case analysis confirms graph RAG value (multi-hop, relational queries)
• [ ] Entity types and relationship types defined in ontology
• [ ] Entity resolution pipeline prevents duplicates
• [ ] Source provenance tracked on all nodes and edges
• [ ] Hybrid retrieval combines graph traversal + vector similarity
• [ ] Graph context packed within LLM token budget
• [ ] Incremental update pipeline handles add/update/delete
• [ ] Query latency measured and within SLA (<1s typical)
• [ ] Graph quality evaluated (entity coverage, relationship accuracy)
• [ ] Fallback to vector-only if graph retrieval returns empty

---

Cross-References

• ai-rag/references/embedding-model-guide.md — embeddings for vector component of hybrid search
• ai-rag/references/rag-caching-patterns.md — caching graph traversal results
• ai-mlops/references/experiment-tracking-patterns.md — tracking graph RAG quality experiments
• ai-mlops/references/cost-management-finops.md — LLM costs for entity extraction at scale

Grounding Checklists for RAG Systems

Tools and patterns for ensuring model outputs remain tied to retrieved evidence, preventing hallucinations and ensuring context compression.

---

1. Context Compression & Budgeting

Use when context window is tight.

Compression Strategies

1. Merge adjacent chunks - Combine semantically related chunks 2. Deduplicate repeated sentences - Remove redundant information 3. Summarize long chunks (LLM distillation) - Use smaller model to compress 4. Prioritize by relevance score - Include highest-scoring chunks first 5. Structure context as sections - Group by topic/source

Token Budget Management

• Calculate token budget: model_context_window - prompt_tokens - max_output_tokens
• Reserve 20-30% buffer for formatting overhead
• Track actual usage vs budget
• Example: Claude 200k context → reserve ~150k for retrieved content

Context Optimization Checklist

• [ ] Fits within model's token budget
• [ ] Includes top-ranked chunks
• [ ] Avoids filler / irrelevant content
• [ ] Document titles preserved
• [ ] Compression tested on eval set (no quality degradation)

---

2. Grounding Enforcement Pattern

Prompt Constraints

Use ONLY the provided context. If the answer is not found in the context, respond: "Not found in the documents."

Required

• No external facts
• No speculation
• All claims must cite chunks

Reinforcement Techniques

• Add negative examples in prompt (show what NOT to do)
• Add citations requirement (force explicit references)
• Apply reranker with answerability scoring
• Use constrained decoding or structured outputs

Grounding Enforcement Checklist

• [ ] Instructions explicitly forbid outside knowledge
• [ ] Tested on "not answerable" cases
• [ ] Model declines when context insufficient
• [ ] No hallucinated facts in outputs

---

3. Citation Pattern

Format

Answer: <text> Sources:

[1] <chunk_metadata> [2] <chunk_metadata>

Checklist

• [ ] All claims traceable to source chunk
• [ ] Chunk IDs stable
• [ ] No fabricated citations

---

4. Context Quality Checklist

• [ ] Top-ranked chunks relevant
• [ ] No noisy or empty chunks
• [ ] Context fits within token budget
• [ ] Ordered by relevance
• [ ] Headings/titles preserved

---

5. Hallucination Suppression Rules

Add negative constraints:

• "Do not guess."
• "If unsure, say: 'I don't know.'"
• "Do not infer facts not explicitly provided."

---

6. Answerability Validation Pattern

Use before generation:

1. Add classifier or shallow LLM to test if context is answerable 2. If unanswerable → return safe response 3. If answerable → generate output

---

7. Grounding Final Checklist

• [ ] Output references retrieved evidence
• [ ] No ungrounded claims
• [ ] Declines appropriately when context insufficient
• [ ] Citations are accurate and traceable
• [ ] Context fits within token budget
• [ ] Tested on edge cases (unanswerable questions)

---

Related Resources

• Advanced RAG Patterns - Context compression strategies
• Pipeline Architecture - Where grounding fits in the pipeline
• Retrieval Patterns - Improving context relevance
• ../assets/context/template-grounding.md - Implementation template

Hybrid Fusion Patterns (BM25 + Vector Search)

These patterns combine lexical and dense retrieval for maximum recall and relevance.

---

1. Why Use Hybrid Search

Use hybrid retrieval when:

• Queries vary between keyword and semantic
• Data includes structured + narrative content
• Domain terms or abbreviations affect relevance
• BM25 or vector alone is insufficient

---

2. Fusion Strategies

A. Weighted Sum Fusion

score = α bm25 + β similarity Typical weights:

• α = 0.3–0.6
• β = 0.4–0.7

---

B. Reciprocal Rank Fusion (RRF)

score = Σ (1 / (k + rank_i)) Characteristics:

• Stable
• Easy to tune
• Order-based, not score-based

---

C. Two-Stage Fusion

1. BM25 retrieves top K 2. Dense retrieves top K 3. Reranker fuses + scores

---

3. Hybrid Workflow Template

1. Preprocess & embed text 2. Run BM25 ranker 3. Run vector search 4. Combine lists 5. Rerank using cross-encoder 6. Output top N results

---

4. Fusion Tuning Guidelines

• Tune α/β using nDCG@10
• Test RRF for stability across query types
• Use BM25 to cover keyword-heavy queries
• Use vectors to cover paraphrases and synonyms

---

5. Hybrid Quality Checklist

• [ ] Both rankers tuned
• [ ] Fusion method selected & validated
• [ ] Recall@k improves vs baseline
• [ ] No quality loss on keyword queries

RAG Troubleshooting Guide

A structured triage tool for diagnosing retrieval, relevance, hallucination, and context issues.

---

1. Symptom → Cause → Fix Matrix

1.1 Irrelevant Results

Causes

• Poor chunking
• Weak embedding model
• Low K
• Bad index parameters

Fixes

• Increase chunk size or overlap
• Switch to stronger embeddings
• Tune ef_search / nprobe
• Use hybrid retrieval
• Add reranking

---

1.2 Missing Essential Information

Causes

• K too small
• Metadata filters too strict
• Incorrect query formulation

Fixes

• Increase K
• Loosen filters
• Add query rewriting

---

1.3 Hallucinations

Causes

• Context not strong enough
• Prompt allows speculation
• Chunks irrelevant

Fixes

• Add grounding constraints
• Enforce citation requirement
• Improve chunk quality

---

1.4 Slow Retrieval

Causes

• Large index without tuning
• High-dimensional embeddings
• Disk-based index not cached

Fixes

• Reduce embedding size
• Use HNSW
• Optimize ANN parameters
• Cache vectors in RAM

---

1.5 Repeated Content or Duplicates

Causes

• Duplicate documents
• Chunk explosion
• Overlap too high

Fixes

• Deduplicate source documents
• Reduce overlap
• Add chunk hashing

---

2. Debugging Workflow

1. Inspect retrieved chunks manually 2. Log embeddings & similarity scores 3. Test with simpler queries 4. Compare BM25 vs vector vs hybrid 5. Evaluate reranker performance

---

3. Logging Requirements

Log:

• Query text
• Retrieval K
• Index version
• Chunk IDs returned
• Similarity scores
• Reranker scores

---

4. Troubleshooting Checklist

• [ ] Retrieval validated
• [ ] Chunking validated
• [ ] Grounding constraints respected
• [ ] Hallucinations mitigated
• [ ] Reranker improves precision