Ai Llm

LLM Development & Engineering — Complete Reference

Build, evaluate, and deploy LLM systems with modern production standards.

This skill covers the full LLM lifecycle:

• Development: Strategy selection, dataset design, instruction tuning, PEFT/LoRA fine-tuning
• Evaluation: Automated testing, LLM-as-judge, metrics, rollout gates
• Deployment: Serving handoff, latency/cost budgeting, reliability patterns (see ai-llm-inference)
• Operations: Quality monitoring, change management, incident response (see ai-mlops)
• Safety: Threat modeling, data governance, layered mitigations (NIST AI RMF: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

Modern Best Practices (2026):

• Treat the model as a component with contracts, budgets, and rollback plans (not "magic").
• Separate core concepts (tokenization, context, training vs adaptation) from implementation choices (providers, SDKs).
• Gate upgrades with repeatable evals and staged rollout; avoid blind model swaps.
• Cost-aware engineering: Measure cost per successful outcome, not just cost per token; design tiering/caching early.
• Security-by-design: Threat model prompt injection, data leakage, and tool abuse; treat guardrails as production code.

For detailed patterns: See Resources and Templates sections below.

---

Quick Reference

---

Decision Tree: LLM System Architecture

Building LLM application: [Architecture Selection]
    ├─ Need current knowledge?
    │   ├─ Simple Q&A? → Basic RAG (page-level chunking + hybrid retrieval)
    │   └─ Complex retrieval? → Advanced RAG (reranking + contextual retrieval)
    │
    ├─ Need tool use / actions?
    │   ├─ Single task? → Simple agent (ReAct pattern)
    │   └─ Multi-step workflow? → Multi-agent (LangGraph, CrewAI)
    │
    ├─ Static behavior sufficient?
    │   ├─ Quick MVP? → Prompt engineering (CI/CD integrated)
    │   └─ Production quality? → Fine-tuning (PEFT/LoRA)
    │
    └─ Best results?
        └─ Hybrid (RAG + Fine-tuning + Agents) → Comprehensive solution

See [Decision Matrices](references/decision-matrices.md) for detailed selection criteria.

---

Cost-Quality Decision Framework

LLM spend is driven by usage-based inference (tokens/requests) plus supporting infra and engineering. Model selection is a cost-quality-latency-risk tradeoff.

Model Tier Strategy

| Tier | Typical profile | Use For | |------|--------|------|---------| | Value | Small/fast models | High-volume, simple tasks | | Balanced | General-purpose models | Most production workloads | | Premium | Frontier/large models | Hardest tasks, low volume |

Cost Optimization Levers

1. Model tiering: Route simple requests to cheaper models (often large savings at scale) 2. Prompt caching: Reuse stable prefixes/context (provider-specific discounts and constraints) 3. Prompt optimization: Compress examples and instructions (typically meaningful token reduction) 4. Output limits: Set appropriate max_tokens (prevents runaway costs)

When to Fine-Tune (ROI-Based)

Fine-tuning pays off when:

• Volume justifies it: >10k requests/month provides meaningful cost savings
• Domain is stable: Requirements unchanged for >6 months
• Data exists: >1,000 quality training examples available
• Break-even achievable: <12 months to recover investment

See [Cost Economics](references/cost-economics.md) for TCO modeling and [Fine-Tuning ROI Calculator](assets/selection/fine-tuning-roi-calculator.md) for investment analysis.

---

Core Concepts (Vendor-Agnostic)

• Model classes: encoder-only, decoder-only, encoder-decoder, multimodal; choose based on task and latency.
• Tokenization & limits: context window, max output, and prompt/template overhead drive both cost and tail latency.
• Adaptation options: prompting → retrieval → adapters (LoRA) → full fine-tune; choose by stability and ROI (LoRA: https://arxiv.org/abs/2106.09685).
• Evaluation: metrics must map to user value; report uncertainty and slice performance, not only global averages.
• Governance: data retention, residency, licensing, and auditability are product requirements (EU AI Act: https://eur-lex.europa.eu/eli/reg/2024/1689/oj; NIST GenAI Profile: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf).

Implementation Practices (Tooling Examples)

• Use a provider abstraction (gateway/router) to enable fallbacks and staged upgrades.
• Instrument requests with tokens, latency, and error classes (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).
• Maintain prompt/model registries with versioning, changelogs, and rollback criteria.

Do / Avoid

Do

• Do pin model + prompt versions in production, and re-run evals before any change.
• Do enforce budgets at the boundary: max tokens, max tools, max retries, max cost.
• Do plan for degraded modes (smaller model, cached answers, “unable to answer”).

Avoid

• Avoid model sprawl (unowned variants with no eval coverage).
• Avoid blind upgrades based on anecdotal quality; require measured impact.
• Avoid training on production logs without consent, governance, and leakage controls.

When to Use This Skill

Claude should invoke this skill when the user asks about:

• LLM preflight/project checklists, production best practices, or data pipelines
• Building or deploying RAG, agentic, or prompt-based LLM apps
• Prompt design, chain-of-thought (CoT), ReAct, or template patterns
• Troubleshooting LLM hallucination, bias, retrieval issues, or production failures
• Evaluating LLMs: benchmarks, multi-metric eval, or rollout/monitoring
• LLMOps: deployment, rollback, scaling, resource optimization
• Technology stack selection (models, vector DBs, frameworks)
• Production deployment strategies and operational patterns

---

Scope Boundaries (Use These Skills for Depth)

• Prompt design & CI/CD → ai-prompt-engineering
• RAG pipelines & chunking → ai-rag
• Search tuning (BM25, HNSW, hybrid) → ai-rag
• Agent architectures & tools → ai-agents
• Serving optimization/quantization → ai-llm-inference
• Production deployment/monitoring → ai-mlops
• Security/guardrails → ai-mlops

---

Resources (Best Practices & Operational Patterns)

Comprehensive operational guides with checklists, patterns, and decision frameworks:

Core Operational Patterns

• [Cost Economics & Decision Frameworks](references/cost-economics.md) - Cost modeling, unit economics, TCO analysis
• Pricing/discount assumptions (verify against current provider docs)
• Cost-quality tradeoff framework and decision matrix
• Total Cost of Ownership (TCO) calculation
• Fine-tuning ROI framework and break-even analysis
• Prompt caching economics
• Cost monitoring and budget guardrails

• [Project Planning Patterns](references/project-planning-patterns.md) - Stack selection, FTI pipeline, performance budgeting
• AI engineering stack selection matrix
• Feature/Training/Inference (FTI) pipeline blueprint
• Performance budgeting and goodput gates
• Progressive complexity (prompt → RAG → fine-tune → hybrid)

• [Production Checklists](references/production-checklists.md) - Pre-deployment validation and operational checklists
• LLM lifecycle checklist (modern production standards)
• Data & training, RAG pipeline, deployment & serving
• Safety/guardrails, evaluation, agentic systems
• Reliability & data infrastructure (DDIA-grade)
• Weekly production tasks

• [Common Design Patterns](references/common-design-patterns.md) - Copy-paste ready implementation examples
• Chain-of-Thought (CoT) prompting
• ReAct (Reason + Act) pattern
• RAG pipeline (minimal to advanced)
• Agentic planning loop
• Self-reflection and multi-agent collaboration

• [Decision Matrices](references/decision-matrices.md) - Quick reference tables for selection
• RAG type decision matrix (naive → advanced → modular)
• Production evaluation table with targets and actions
• Model selection matrix (tier-based, vendor-agnostic)
• Vector database, embedding model, framework selection
• Deployment strategy matrix

• [Anti-Patterns](references/anti-patterns.md) - Common mistakes and prevention strategies
• Data leakage, prompt dilution, RAG context overload
• Agentic runaway, over-engineering, ignoring evaluation
• Hard-coded prompts, missing observability
• Detection methods and prevention code examples

Domain-Specific Patterns

• [LLMOps Best Practices](references/llmops-best-practices.md) - Operational lifecycle and deployment patterns
• [Evaluation Patterns](references/eval-patterns.md) - Testing, metrics, and quality validation
• [Prompt Engineering Patterns](references/prompt-engineering-patterns.md) - Quick reference (canonical skill: ai-prompt-engineering)
• [Agentic Patterns](references/agentic-patterns.md) - Quick reference (canonical skill: ai-agents)
• [RAG Best Practices](references/rag-best-practices.md) - Quick reference (canonical skill: ai-rag)

Emerging Patterns

• [Structured Output Patterns](references/structured-output-patterns.md) - JSON mode, constrained decoding, schema enforcement, validation pipelines
• [Multimodal Patterns](references/multimodal-patterns.md) - Vision-language models, audio/image inputs, cross-modal pipelines, cost management
• [Model Migration Guide](references/model-migration-guide.md) - Provider migration playbook, eval-gated rollout, prompt adaptation, fallback strategies

Note: Each resource file includes preflight/validation checklists, copy-paste reference tables, inline templates, anti-patterns, and decision matrices.

---

Templates (Copy-Paste Ready)

Production templates by use case and technology:

Selection & Governance

• [Model Selection Matrix](assets/selection/model-selection-matrix.md) - Documented selection, scoring, licensing, and governance
• [Fine-Tuning ROI Calculator](assets/selection/fine-tuning-roi-calculator.md) - Investment analysis, break-even, go/no-go decisions

RAG Pipelines

• [Basic RAG](assets/rag-pipelines/template-basic-rag.md) - Simple retrieval-augmented generation
• [Advanced RAG](assets/rag-pipelines/template-advanced-rag.md) - Hybrid retrieval, reranking, contextual embeddings

Prompt Engineering

• [Chain-of-Thought](assets/prompt-engineering/template-cot.md) - Step-by-step reasoning pattern
• [ReAct](assets/prompt-engineering/template-react.md) - Reason + Act for tool use

Agentic Workflows

• [Reflection Agent](assets/agentic-workflows/template-reflection.md) - Self-critique and improvement
• [Multi-Agent](assets/agentic-workflows/template-multi-agent.md) - Manager-worker orchestration

Data Pipelines

• [Data Quality](assets/data-pipelines/template-data-quality.md) - Validation, deduplication, PII detection

Deployment

• [LLM Deployment](assets/deployment/template-llm-deployment.md) - Production deployment with monitoring

Evaluation

• [Multi-Metric Evaluation](assets/evaluation/template-multi-metric.md) - Comprehensive testing suite

---

Shared Utilities (Centralized patterns — extract, don't duplicate)

• ../software-clean-code-standard/utilities/llm-utilities.md — Token counting, streaming, cost estimation
• ../software-clean-code-standard/utilities/error-handling.md — Effect Result types, correlation IDs
• ../software-clean-code-standard/utilities/resilience-utilities.md — p-retry v6, circuit breaker for LLM API calls
• ../software-clean-code-standard/utilities/logging-utilities.md — pino v9 + OpenTelemetry integration
• ../software-clean-code-standard/utilities/observability-utilities.md — OpenTelemetry SDK, tracing, metrics
• ../software-clean-code-standard/utilities/config-validation.md — Zod 3.24+, secrets management for API keys
• ../software-clean-code-standard/utilities/testing-utilities.md — Test factories, fixtures, mocks
• ../software-clean-code-standard/references/clean-code-standard.md — Canonical clean code rules (CC-*) for citation

---

Trend Awareness Protocol

IMPORTANT: For “best/latest” recommendations, verify recency using current sources (official docs/release notes/benchmarks). If you can’t browse, state assumptions and ask for timeframe + constraints.

Trigger Conditions

• "What's the best LLM model for [use case]?"
• "What should I use for [RAG/fine-tuning/agents]?"
• "What's the latest in LLM development?"
• "Current best practices for [prompting/evaluation/deployment]?"
• "Is [model/framework] still relevant in 2026?"
• "[Model A] vs [Model B]?" or "[Framework A] vs [Framework B]?"
• "Best vector database for [use case]?"
• "What agent framework should I use?"

Minimal Verification Checklist

1. Confirm user constraints: latency, cost, privacy/compliance, deployment target, and toolchain. 2. Check at least 2 authoritative sources from data/sources.json (provider docs, release notes, pricing/quotas, deprecations). 3. Prefer stable guidance (tradeoffs + decision criteria) over “one best model/framework”.

What to Report

After searching, provide:

• Current landscape: What models/frameworks are popular NOW (not 6 months ago)
• Emerging trends: New models, frameworks, or techniques gaining traction
• Deprecated/declining: Models/frameworks losing relevance or support
• Recommendation: Based on fresh data, not just static knowledge

Example Topics (verify with fresh sources)

• Latest frontier models (GPT-4.5, Claude 4, Gemini 2.x, Llama 4)
• Agent frameworks (LangGraph, CrewAI, AutoGen, Semantic Kernel)
• Vector databases (Pinecone, Qdrant, Weaviate, pgvector)
• RAG techniques (contextual retrieval, agentic RAG, graph RAG)
• Inference engines (vLLM, TensorRT-LLM, SGLang)
• Evaluation frameworks (RAGAS, DeepEval, Braintrust)

---

Related Skills

This skill integrates with complementary Claude Code skills:

Core Dependencies

• [ai-rag](../ai-rag/SKILL.md) - Retrieval pipelines: chunking, hybrid search, reranking, evaluation
• [ai-prompt-engineering](../ai-prompt-engineering/SKILL.md) - Systematic prompt design, evaluation, testing, and optimization
• [ai-agents](../ai-agents/SKILL.md) - Agent architectures, tool use, multi-agent systems, autonomous workflows

Production & Operations

• [ai-llm-inference](../ai-llm-inference/SKILL.md) - Production serving, quantization, batching, GPU optimization
• [ai-mlops](../ai-mlops/SKILL.md) - Deployment, monitoring, incident response, security, and governance

---

External Resources

See [data/sources.json](data/sources.json) for 50+ curated authoritative sources:

• Official LLM platform docs - OpenAI, Anthropic, Gemini, Mistral, Azure OpenAI, AWS Bedrock
• Open-source models and frameworks - HuggingFace Transformers, open-weight models, PEFT/LoRA, distributed training/inference stacks
• RAG frameworks and vector DBs - LlamaIndex, LangChain 1.2+, LangGraph, LangGraph Studio v2, Haystack, Pinecone, Qdrant, Chroma
• Agent frameworks (examples) - LangGraph, Semantic Kernel, AutoGen, CrewAI
• RAG innovations (examples) - Graph-based retrieval, hybrid retrieval, online evaluation loops
• Prompt engineering - Anthropic Prompt Library, Prompt Engineering Guide, CoT/ReAct patterns
• Evaluation and monitoring - OpenAI Evals, HELM, Anthropic Evals, LangSmith, W&B, Arize Phoenix
• Production deployment - Model gateways/routers, self-hosted serving, managed endpoints

---

Usage

For New Projects

1. Start with [Production Checklists](references/production-checklists.md) - Validate all pre-deployment requirements 2. Use [Decision Matrices](references/decision-matrices.md) - Select technology stack 3. Reference [Project Planning Patterns](references/project-planning-patterns.md) - Design FTI pipeline 4. Implement with [Common Design Patterns](references/common-design-patterns.md) - Copy-paste code examples 5. Avoid [Anti-Patterns](references/anti-patterns.md) - Learn from common mistakes

For Troubleshooting

1. Check [Anti-Patterns](references/anti-patterns.md) - Identify failure modes and mitigations 2. Use [Decision Matrices](references/decision-matrices.md) - Evaluate if architecture fits use case 3. Reference [Common Design Patterns](references/common-design-patterns.md) - Verify implementation correctness

For Ongoing Operations

1. Follow [Production Checklists](references/production-checklists.md) - Weekly operational tasks 2. Integrate [Evaluation Patterns](references/eval-patterns.md) - Continuous quality monitoring 3. Apply [LLMOps Best Practices](references/llmops-best-practices.md) - Deployment and rollback procedures

---

Navigation Summary

Quick Decisions: Decision Matrices Pre-Deployment: Production Checklists Planning: Project Planning Patterns Implementation: Common Design Patterns Troubleshooting: Anti-Patterns

Domain Depth: LLMOps | Evaluation | Prompts | Agents | RAG

Templates: assets/ - Copy-paste ready production code

Sources: data/sources.json - Authoritative documentation links

---

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Multi-Agent Collaboration Workflow Template

Purpose: Scaffold for building LLM systems that coordinate multiple specialized agents—each with clear roles, hand-off logic, communication, and conflict resolution—for complex tasks and large workflows.

---

When to Use

Use this template when:

• Your system requires multiple agents (e.g., research, planning, coding, compliance, customer support)
• Each agent has a specialized role or expertise (domain, tool, task)
• Tasks must be handed off between agents, or require arbitration/consensus
• Full audit trail, error handling, and deadlock detection are required

---

Structure

This template has 5 sections:

1. Role Assignment – define agent roles, capabilities, and responsibilities 2. Task Routing/Hand-Off – protocol for delegating and escalating subtasks 3. Communication Protocol – message formats, APIs, shared memory/log 4. Arbitration/Consensus – resolving conflicts, merging outputs, final decisions 5. Auditability & Error Handling – log all actions, resolve deadlocks/loops, escalate failures

---

TEMPLATE STARTS HERE

Prompt Scaffold:

System: You are coordinating a team of AI agents, each with a specialized role.

[Agent Directory]
- ResearchAgent: finds information, summarizes sources.
- PlannerAgent: breaks down goals, allocates subtasks.
- QAAgent: verifies facts, checks for errors/hallucinations.
- ComplianceAgent: checks outputs for policy/compliance.

[Workflow Rules]
- Assign each incoming task to the appropriate agent.
- If a task requires multiple roles, PlannerAgent splits and routes.
- Agents communicate by logging actions and results to a shared record.
- If two agents disagree, escalate to ArbitrationAgent for final decision.
- All actions, messages, and outcomes are logged for audit.

[Example]
Task: "Draft a report on AI regulation in Europe."

PlannerAgent: Breaks into: (a) Research EU laws, (b) Summarize, (c) Compliance check.
ResearchAgent: Handles (a), posts findings.
PlannerAgent: Assigns (b) to itself or to SummaryAgent.
QAAgent: Reviews findings and summary for accuracy.
ComplianceAgent: Checks draft for legal/compliance issues.
ArbitrationAgent: Decides if QA and Compliance disagree.
All results are logged, and the final answer is posted only after audit.

Final Answer: [Composed, verified, and compliant report.]

---

COMPLETE EXAMPLE

Workflow (Pseudo-dialogue style):

Task: "Update our GDPR FAQ for customers."

PlannerAgent: Splits into: 1) Research updates, 2) Write draft, 3) Check compliance.
ResearchAgent: Finds latest GDPR changes. Logs sources.
WriterAgent: Drafts updated FAQ.
QAAgent: Reviews draft for errors or unsupported claims.
ComplianceAgent: Checks for legal compliance, flags ambiguous sections.
QAAgent: Flags a claim as unsupported. Logs conflict.
ArbitrationAgent: Reviews the conflict, decides to remove unsupported claim.
All agents log actions to shared record.

Final Answer: Updated FAQ, logged with all actions, decisions, and sources.

---

Quality Checklist

Before finalizing:

• [ ] All agent roles, tasks, and hand-off rules defined and coded
• [ ] Shared memory/log or API for communication/audit
• [ ] Arbitration logic present for conflicts or deadlocks
• [ ] All actions, decisions, escalations logged for audit/review
• [ ] At least one complete test case with hand-off and arbitration

---

For agentic reflection/self-correction, see [template-reflection.md]. For data quality, deployment, and evaluation, see other templates and [references/agentic-patterns.md].

Agentic Reflection Workflow Template

Purpose: Scaffold for agents that improve their reasoning, correctness, and robustness by reflecting, self-evaluating, and revising plans after each major step—field-tested in RAG, coding agents, complex QA, and self-correcting assistants.

---

When to Use

Use this template when:

• Building agents that should check, critique, or improve their own output before responding
• Tasks require high reliability (e.g., coding, complex reasoning, multi-step plans)
• Reducing hallucination, error propagation, or failed tool/API calls is critical
• You want agents to “think about their thinking” and adjust plans dynamically

---

Structure

This template has 5 sections:

1. Explicit Instructions – direct the agent to always reflect and self-evaluate after actions 2. Perception/Planning – model plans and acts as in standard agent workflow 3. Reflection Step – after each action or output, agent must pause and review results 4. Revision/Correction – agent proposes improvements, corrections, or re-plans if errors/uncertainty detected 5. Final Output – after one or more reflection cycles, agent delivers answer or outcome

---

TEMPLATE STARTS HERE

Prompt Scaffold:

You are an AI agent that must always reflect on your actions before finishing a task.
For each task:
- Plan and describe your next step.
- Take action (tool/API call, reasoning step, etc.).
- After each action, reflect: Did this action succeed? Is the result correct and sufficient? What could be improved?
- If any error or uncertainty, revise your plan or retry, then reflect again.
- When you are confident the answer is correct and complete, finish with:

Final Answer: [Your best, reviewed answer.]

---

COMPLETE EXAMPLE

QA with Reflection

Question: What is the capital of Australia?

Plan: I'll recall what I know or search trusted sources.
Action: Recall: I think it's Sydney.
Reflection: Wait—let me double-check. Is this correct?
Action: [Search "capital of Australia"]
Observation: The search result says the capital is Canberra.
Reflection: My first answer was incorrect. I'll update my answer.

Final Answer: The capital of Australia is Canberra.

Coding Agent with Reflection

Task: Write a Python function that computes Fibonacci numbers.

Plan: I'll write the function using recursion.
Action:

def fib(n): if n <= 1: return n return fib(n-1) + fib(n-2)

Reflection: Is this efficient? For large n, this will be slow.
Revision: I'll use memoization to improve performance.

def fib(n, memo={}): if n in memo: return memo[n] if n <= 1: return n memo[n] = fib(n-1, memo) + fib(n-2, memo) return memo[n]

Reflection: This is now efficient and correct.

Final Answer: [Returns improved function with memoization.]

Tool/API Use with Reflection

Task: Get the weather for Paris.

Plan: Use WeatherAPI.
Action: [WeatherAPI "Paris"]
Observation: API error, failed to fetch.
Reflection: My action failed. I'll try an alternative API.
Action: [BackupWeatherAPI "Paris"]
Observation: Success! 20°C, sunny.
Reflection: Now I have the correct result.

Final Answer: The weather in Paris is 20°C and sunny.

---

Quality Checklist

Before finalizing:

• [ ] Explicit reflection step after every major action/tool call in prompt
• [ ] Instructions require agent to revise/replan if error or uncertainty detected
• [ ] Prompt tested for both error correction and normal paths
• [ ] Output always ends with confident, reviewed Final Answer
• [ ] At least one end-to-end example for each target use case

---

For multi-agent collaboration, see [template-multi-agent.md]. For agentic workflow patterns and checklists, see [references/agentic-patterns.md].

Data Quality Pipeline Template

Purpose: Copy-paste scaffold for constructing data pipelines that ensure clean, deduplicated, relevant, and safe training/evaluation data for LLMs, RAG, or agentic systems.

---

When to Use

Use this template when:

• Preparing training, fine-tuning, or RAG data for any LLM or agent system
• You need to guarantee data freshness, consistency, deduplication, and PII safety
• You must pass production LLMOps or compliance review for data quality

---

Structure

This template has 5 sections:

1. Ingestion – load raw data from all sources 2. Filtering & Deduplication – remove junk, deduplicate, basic cleaning 3. PII & Safety Scanning – detect and remove personally identifiable info or toxic content 4. Labeling & Metadata – assign tags, data splits, version, and source info 5. Validation & Audit – test data pipeline, sample review, and version audit

---

TEMPLATE STARTS HERE

Pipeline Scaffold:

1. Ingestion
   - Load data from all required sources (docs, web, PDFs, code, etc)
   - Track source and collection date for each file or row

2. Filtering & Deduplication
   - Remove empty or near-duplicate records (use hashes or similarity)
   - Strip boilerplate, ads, footers, legal disclaimers
   - Normalize encoding and line endings

3. PII & Safety Scanning
   - Scan and redact emails, phone numbers, credit cards, names, locations, etc.
   - Use toxicity filter or blocklist to remove unsafe/abusive content
   - Document all PII redactions/removals

4. Labeling & Metadata
   - Assign split: train/valid/test or pretrain/finetune/eval
   - Add language, domain, or topic tags
   - Store version, source, and pipeline config for every output batch

5. Validation & Audit
   - Sample rows for human review (spot-check data, PII, toxic content)
   - Log pass/fail, run regression on pipeline after edits
   - Archive all data pipeline configs for reproducibility

---

COMPLETE EXAMPLE

Pythonic pseudo-code (pipeline script):

# 1. Ingest
docs = load_files("raw_data/", recursive=True)
docs = [{**doc, "source": src, "date": today()} for doc, src in docs]

# 2. Filter & Dedup
docs = filter_empty(docs)
docs = deduplicate_by_hash(docs)
docs = strip_boilerplate(docs)

# 3. PII & Safety Scan
docs = redact_pii(docs)
docs = remove_toxic_content(docs)

# 4. Label/Metadata
for doc in docs:
    doc['split'] = assign_split(doc)
    doc['lang'] = detect_language(doc['content'])
    doc['pipeline_ver'] = "v1.0"
    doc['source'] = doc['source']

# 5. Validate/Audit
sample = random_sample(docs, 0.01)
human_review(sample)
log_pipeline_version("v1.0", pipeline_config)

---

Quality Checklist

Before finalizing:

• [ ] All data sources logged, dates and source tracked
• [ ] Deduplication and filtering scripts run, verified (no near-duplicate rows)
• [ ] PII scan/removal complete and logged
• [ ] Toxic/abusive content filter applied
• [ ] Each row labeled with split, language, version, and source
• [ ] Manual review spot-checks at least 1% of data
• [ ] Pipeline config, audit logs, and version archived for reproducibility

---

For RAG/Retrieval, see [template-basic-rag.md] or [template-advanced-rag.md]. For deployment or agentic QA, see other templates and [references/llmops-best-practices.md].

LLM Deployment Checklist Template

Purpose: Field-ready checklist and playbook for safely deploying LLMs, RAG, or agentic systems to staging and production environments, with rollback, monitoring, and post-deploy quality gates.

---

When to Use

Use this template when:

• Releasing a new LLM-powered app, RAG/agentic service, or major update
• Moving from development/staging to production
• Need to pass security, compliance, and LLMOps review before go-live

---

Structure

This template has 5 main sections:

1. Preflight Readiness – staging, tests, and backup 2. Deployment Rollout – staged, canary, or blue/green releases 3. Monitoring & Observability – latency, cost, usage, error, and abuse monitoring 4. Incident & Rollback Plan – what to do if critical metrics fail or bugs are found 5. Post-Deployment Review – quality gates, regression checks, and user feedback

---

TEMPLATE STARTS HERE

1. Preflight Readiness

• [ ] All code, prompts, models, and pipeline configs in version control
• [ ] Regression, edge-case, and adversarial tests all passing
• [ ] Data pipeline/feeds validated and locked
• [ ] “Last known good” model/pipeline checkpoint archived
• [ ] Access/secret management reviewed (keys, API tokens, RBAC)
• [ ] Staging environment matches production as closely as possible

2. Deployment Rollout

• [ ] Deploy to staging and run smoke/regression tests
• [ ] Canary/blue-green: Route small % of traffic to new model/service first
• [ ] Monitor all key metrics live; abort if critical issues seen
• [ ] Full rollout only after canary passes quality gates

3. Monitoring & Observability

• [ ] Logs for all requests, outputs, and errors enabled
• [ ] Dashboards for latency (p50/p95), usage, and cost live
• [ ] Real-time alerts for high error, OOM, safety/abuse events
• [ ] Prompts and models versioned and tracked in all logs
• [ ] Output sampling for manual review

4. Incident & Rollback Plan

• [ ] Documented, tested rollback script (to last stable version)
• [ ] Incident thresholds: latency, hallucination, outage, cost, safety
• [ ] Escalation path for on-call/response (24/7 for critical prod)
• [ ] Failover or fallback model available if primary fails

5. Post-Deployment Review

• [ ] All quality gates (accuracy, faithfulness, latency, safety, cost) passed in prod
• [ ] Edge/adversarial cases checked live
• [ ] User feedback and error/abuse reports monitored daily
• [ ] Weekly regression re-run after go-live
• [ ] Archive all logs, incidents, and test results for audit

---

COMPLETE PLAYBOOK EXAMPLE

LLM/RAG/Agentic Prod Release Flow

1. Freeze code, data, prompt, and model versions.
2. Run full test suite in staging; archive passing logs.
3. Deploy canary to 5% of users/traffic.
4. Monitor: latency <2s, hallucination <3%, error rate <1%.
5. If metrics pass after X hours, rollout to 100%.
6. If any incident threshold hit:
   - Roll back to last stable deploy (scripted)
   - Notify on-call, triage root cause, fix or hot-patch
7. Post-launch: Sample outputs, gather user feedback, schedule postmortem.
8. Document all failures, actions, and lessons for future deployments.

---

Release-Readiness Checklist

• [ ] All code/prompt/model/data versioned and auditable
• [ ] Regression, edge, and abuse tests passing in prod
• [ ] Canary/staged rollout configured and tested
• [ ] Live monitoring and alerting enabled for all critical metrics
• [ ] Incident rollback and on-call escalation documented and tested
• [ ] All deployment, monitoring, and rollback steps reviewed by ops/QA/lead

---

For ongoing monitoring, see [references/llmops-best-practices.md]. For eval patterns and emergency playbooks, see [references/eval-patterns.md].

Multi-Metric LLM Evaluation Template

Purpose: Scaffold for evaluating LLM, RAG, or agentic systems on all critical dimensions (accuracy, faithfulness, latency, safety, cost, format, etc)—usable for pre-prod validation, regression, or ongoing LLMOps.

---

When to Use

Use this template when:

• Validating any new model, RAG pipeline, prompt, or agent release before production
• Monitoring ongoing system quality (hallucination, latency, cost, safety)
• Comparing different models, prompts, or RAG configs (A/B testing)
• Running regression after code/prompt/data/model changes

---

Structure

This template has 4 main sections:

1. Metric Selection – define all required metrics for system/goal 2. Test Suite – build gold set (QA pairs, edge/adversarial cases, format/abuse checks) 3. Evaluation Run – execute automated and/or human-in-the-loop evals 4. Results & Quality Gates – compare to thresholds, block release if any fail

---

TEMPLATE STARTS HERE

1. Metric Selection

Add others as needed: bias, toxicity, recall, F1, etc.

2. Test Suite

• Gold Q/A pairs with known correct answers
• Context-grounded test cases for faithfulness/hallucination
• Adversarial/edge cases (prompt injection, ambiguous queries)
• Format checks (valid JSON, table, etc)
• Abuse/toxicity samples (to trigger filters)
• Regression set from previous release

3. Evaluation Run

• Automated scoring:
• Compute accuracy, format, latency, cost from test batch
• Use faithfulness/hallucination checker (rule-based or LLM-powered)
• Human review:
• Sample N outputs for subjective criteria (helpfulness, style, unclear context)
• Mark pass/fail or rate by metric
• Log all results, compare to previous/baseline

4. Results & Quality Gates

• Pass if ALL critical metrics meet threshold (see table above)
• If any fail: block release, trigger bugfix/patch/rollback
• Store evaluation logs and metrics for audit and ongoing monitoring

---

COMPLETE EXAMPLE

Eval Script (Python-like pseudocode):

metrics = {
  "accuracy": [],
  "faithfulness": [],
  "latency": [],
  "cost": [],
  "format": [],
  "safety": []
}
for test in test_suite:
    result = run_llm(test.input)
    metrics["accuracy"].append(score_accuracy(result, test.gold))
    metrics["faithfulness"].append(score_faithful(result, test.context))
    metrics["latency"].append(result.latency)
    metrics["cost"].append(result.cost)
    metrics["format"].append(validate_format(result.output))
    metrics["safety"].append(scan_for_abuse(result.output))
# Compute means, compare to thresholds, print/pass/fail summary

---

Quality Checklist

Before finalizing:

• [ ] All metrics relevant to your system tracked and validated
• [ ] Test suite covers gold, edge, format, and safety cases
• [ ] Automated AND human-in-the-loop reviews where needed
• [ ] Results compared to clear thresholds (block on fail)
• [ ] All eval runs, metrics, and test cases versioned for audit

---

For live monitoring and incident playbooks, see [references/eval-patterns.md]. For deployment/rollback, see [deployment/].

Fine-Tuning Configuration Template

A reproducible config for SFT, instruction tuning, or LoRA.

---

1. Model Settings

base_model: <model_name> tokenizer: <model_tokenizer> max_seq_length: <length> gradient_checkpointing: true/false

---

2. Training Parameters

learning_rate: 1e-5 batch_size: <value> num_epochs: <value> warmup_steps: <value> seed: 42 eval_steps: <value> save_steps: <value>

---

3. LoRA (Optional)

lora: enable: true r: 8 alpha: 16 dropout: 0.05 target_modules: ["q_proj", "v_proj"]

---

4. Data

train_file: "<path/to/train.jsonl>" validation_file: "<path/to/validation.jsonl>" format: "instruction" | "chat" | "transform"

---

5. Output

output_dir: "<path/to/output>" save_total_limit: 3 log_to_file: true

---

6. Safety

• Validate dataset before training
• Remove harmful content
• Include refusal samples

Chain-of-Thought (CoT) Prompt Template

Purpose: Instantly scaffold prompts for LLMs that require multi-step reasoning, math, or explanations; maximizes answer faithfulness and transparency.

---

When to Use

Use this template when:

• The task requires logic, math, planning, or multi-step reasoning
• You want to make the model’s thinking explicit (“show your work”)
• Reducing hallucinations and increasing faithfulness is important
• Tasks include problem-solving, code explanation, root cause analysis, or planning

---

Structure

This template has 3 main sections:

1. Explicit Reasoning Cue – tell the model to “think step by step” or similar 2. User Question/Task – the prompt input 3. CoT Answer Format – request intermediate steps, then final answer

---

TEMPLATE STARTS HERE

Prompt Scaffold:

Q: {user_question}

A: Let's think step by step.
[Model writes out all steps, explains logic, makes calculations or deductions.]
The answer is: [final answer or result].

Add explicit cues:

• "Explain your reasoning."
• "List each step."
• "Show all calculations before the answer."

---

COMPLETE EXAMPLE

Math Reasoning

Q: If a train leaves at 8:00am and travels 120 miles at 40 mph, what time does it arrive?

A: Let's think step by step.
- The train speed is 40 mph.
- To go 120 miles at 40 mph: 120 / 40 = 3 hours.
- The train leaves at 8:00am, so it arrives at 8:00am + 3 hours = 11:00am.
The answer is: 11:00am.

Root Cause Analysis

Q: The website is down. What are the likely causes and how would you check?

A: Let's think step by step.
- First, check if the server is responding to pings.
- If not, check network connectivity and firewall rules.
- If server responds, check web server process status.
- Check for recent deploys or config changes.
The answer is: Start with server/network checks, then web server process, then configs.

Code Explanation

Q: What does this Python code do? [code snippet]

A: Let's think step by step.
- The code defines a function called foo(x).
- It loops from 1 to x, adding each number to a sum.
- Returns the final sum.
The answer is: It calculates the sum of all numbers from 1 to x.

---

Quality Checklist

Before finalizing:

• [ ] "Let's think step by step" or explicit reasoning cue included
• [ ] Prompt requires stepwise output, not just direct answer
• [ ] At least one test case per core use (math, logic, code, planning)
• [ ] Model output includes intermediate reasoning, not just result
• [ ] Final answer always clearly marked (e.g., “The answer is:”)

---

For tool use/agent patterns, see [template-react.md]. For structure and validation, see [references/prompt-engineering-patterns.md].

ReAct (Reason + Act) Prompt Template

Purpose: Scaffold LLM prompts that interleave reasoning and tool/API use—enabling complex workflows, tool-based retrieval, and multi-step planning in agentic and RAG systems.

---

When to Use

Use this template when:

• The LLM must reason through steps and take actions (API/tool calls, searches, etc)
• The workflow requires alternating between thought, action, observation, and next steps
• Use cases include multi-hop QA, research, web search, code exec, or agentic plans

---

Structure

This template has 5 main sections:

1. Explicit Instructions – clarify the Reason+Act workflow 2. User Question/Task – prompt input 3. ReAct Loop Format – repeatable [Thought] → [Action] → [Observation] sequence 4. Final Answer – LLM delivers answer only after reasoning and tool steps are done 5. Error/Fallback Handling – guide model to replan or escalate if actions fail

---

TEMPLATE STARTS HERE

Prompt Scaffold:

You are an AI assistant that can think step by step and use tools or APIs to solve tasks.
Follow this format:

Question: {user_question}

Thought: [Describe what you want to do next.]
Action: [If needed, specify the tool, API, or search and inputs.]
Observation: [Result/output from tool or previous action.]

(Repeat Thought/Action/Observation as needed...)

When you have enough information, finish with:

Final Answer: [Your answer to the original question.]

---

COMPLETE EXAMPLES

Web Search ReAct

Question: Who is the CEO of OpenAI?

Thought: I need to look up the latest information.
Action: [Search "OpenAI CEO"]
Observation: The search result says Sam Altman is the CEO of OpenAI.

Thought: I have found the answer.
Final Answer: The CEO of OpenAI is Sam Altman.

RAG + Tool Example

Question: What is the warranty policy for product X?

Thought: I need to search the company knowledge base.
Action: [RAG_Search "product X warranty policy"]
Observation: The retrieved chunk says "Product X has a 2-year limited warranty."

Thought: The answer is now clear.
Final Answer: Product X has a 2-year limited warranty.

Multi-Step/Agentic Example

Question: What’s the weather in Paris and send it to my email?

Thought: First, I need to look up the current weather in Paris.
Action: [WeatherAPI "Paris"]
Observation: It's 22°C, partly cloudy.

Thought: Now, I need to send this info via email.
Action: [SendEmail to="user@example.com" body="Weather in Paris is 22°C, partly cloudy."]
Observation: Email sent.

Final Answer: I checked the weather in Paris (22°C, partly cloudy) and emailed it to you.

Error Handling/Fallback

Observation: The API failed to return results.

Thought: The tool failed, so I will try an alternate approach or inform the user.
Action: [Alternative_Tool or escalate]
Observation: Alternate tool also failed.

Final Answer: I was unable to retrieve the information due to a system error.

---

Quality Checklist

Before finalizing:

• [ ] Explicit “Thought/Action/Observation” loop present in the prompt
• [ ] Instructions clarify how and when to use tools/APIs
• [ ] Prompt tested for both successful and failed tool calls
• [ ] Output always ends with “Final Answer” for clarity
• [ ] Handles action failures with fallback/escalation logic

---

For stepwise reasoning, see [template-cot.md]. For more agentic orchestration, see [agentic-workflows/]. For prompt anti-patterns, see [references/prompt-engineering-patterns.md].

Advanced RAG Pipeline Template

Purpose: Deploy a robust, high-recall RAG system for production or high-value use cases (large corpora, mixed formats, high accuracy/faithfulness demands).

---

When to Use

Use this template when:

• Your knowledge base is large, diverse, or multi-format (e.g., >50k docs, mixed PDF/web/Markdown)
• Retrieval precision, faithfulness, or latency is critical
• You need hybrid retrieval (dense + keyword/BM25), reranking, or automated context compression
• The LLM must reliably ground answers in external, up-to-date, or compliance-sensitive data

---

Structure

This template has 6 sections:

1. Advanced Chunking – structure-aware, semantic windowing, deduplication 2. Hybrid Retrieval – dense + keyword search, metadata filtering 3. Post-Retrieval Reranking – LLM/cross-encoder or custom rankers 4. Context Compression – select/summarize to fit context window 5. Prompt Assembly – structured, citation-required, with fallback 6. Evaluation & Monitoring – log, validate, and monitor recall, latency, faithfulness

---

TEMPLATE STARTS HERE

1. Advanced Chunking

• Parse docs by structure (headings, sections, tables)
• Use semantic chunkers if possible (not just token windows)
• Chunk size: 300–800 tokens (adjust per doc type)
• Deduplicate near-duplicate chunks (hash or similarity match)
• Track metadata: doc/source, section, timestamp, tags

2. Hybrid Retrieval

• At query:
• Dense retrieval: Embed question, search vector DB (e.g., BGE, ada-002, E5, etc)
• Keyword/BM25 retrieval: Run keyword search, e.g. with Elastic or built-in
• Combine results, remove duplicates
• Filter by metadata if needed (date, type, tags)

3. Post-Retrieval Reranking

• Pass candidate chunks to reranker:
• LLM-based: e.g., “Does this chunk answer the question? Y/N”
• Cross-encoder model or BERT re-ranker
• Select top-N (usually 2–6) for final prompt

4. Context Compression

• If context window exceeded:
• Summarize lower-priority chunks, or
• Select most relevant sentences within chunk
• Hard truncate only as last resort

5. Prompt Assembly

Prompt Template:

Answer ONLY using the context. If context is insufficient, say "Not found."
Cite sources (e.g., [doc1], [doc2]) for every claim.

Context:
{ranked_context_with_sources}

Question: {user_question}

Answer:

6. Evaluation & Monitoring

• Log retrievals, LLM generations, user feedback
• Periodically re-benchmark retrieval recall, faithfulness, latency, cost
• Trigger alert/rollback if key metrics degrade

---

COMPLETE EXAMPLE

Python (LangChain/Hybrid/LLM rerank pseudo-code):

# 1. Chunking
docs = load_docs("corpus/")
chunks = semantic_chunk(docs)
chunks = deduplicate_chunks(chunks)
store_chunks(chunks, metadata=["doc", "section", "date"])

# 2. Hybrid Retrieval
def hybrid_search(query):
    dense_hits = vector_search(query, top_k=12)
    keyword_hits = bm25_search(query, top_k=12)
    hits = merge_dedup(dense_hits, keyword_hits)
    return hits

# 3. Reranking
reranked = llm_rerank(query, hits, top_n=4)

# 4. Context Compression
context = compress_context(reranked, max_tokens=1800)

# 5. Prompt Assembly
sources = "\n\n".join([f"[{c['doc']}] {c['content']}" for c in context])
prompt = f"""
Answer ONLY using the context. If context is insufficient, say "Not found."
Cite sources (e.g., [doc1], [doc2]) for every claim.

Context:
{sources}

Question: {query}

Answer:
"""
answer = call_llm(prompt)
print(answer)

---

Quality Checklist

Before finalizing:

• [ ] Semantic or structure-aware chunking, dedup complete
• [ ] Hybrid retrieval (dense + keyword) implemented, tuned
• [ ] Reranker (LLM/cross-encoder) validated for precision
• [ ] Context fits LLM input window (with compression)
• [ ] Prompt requires citations, fallback "Not found"
• [ ] Retrieval recall, latency, faithfulness monitored, alerts configured

---

For minimal use cases, see [template-basic-rag.md]. For multi-agent or multimodal, see [agentic-workflows/]. For eval, see [references/eval-patterns.md].

Basic RAG Pipeline Template

Purpose: Instantly scaffold a minimal, working Retrieval-Augmented Generation (RAG) pipeline for document search, QA, or LLM grounding—usable in LangChain, LlamaIndex, or similar frameworks.

---

When to Use

Use this template when:

• You need a fast, reliable RAG implementation for internal KB, FAQ, or doc search
• LLM must ground answers in external data
• Simple, single-language, text-only use case (expand for advanced/hybrid needs)

---

Structure

This template has 4 main sections:

1. Chunking – split source docs for retrieval 2. Embedding – convert chunks to vector space 3. Retrieval – retrieve top-k similar chunks at query time 4. Prompt Assembly & Generation – compose context and generate grounded answer

---

TEMPLATE STARTS HERE

1. Chunking

Script/Process:

• Load docs (markdown, PDF, etc)
• Split into chunks (e.g., 400–600 tokens each, 50–100 token overlap)
• Save chunk metadata: source, position, doc ID

2. Embedding

Code Example (Python/LangChain):

from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
vector_store = Chroma()  # or Pinecone, Qdrant, etc.
vector_store.add_documents(chunks, embeddings)

3. Retrieval

Query-Time Process:

• User submits question
• Embed question (same model)
• Retrieve top-k (e.g., k=4) similar chunks from vector DB

4. Prompt Assembly & Generation

Prompt Template:

You must answer based only on the provided context. If the answer is not found, say "Not found."

Context:
{retrieved_chunks}

Question: {user_question}

Answer:

Generation:

• Send assembled prompt to LLM (Claude, GPT-4, Gemini, etc)
• Return answer (optionally: highlight source or chunk citations)

---

COMPLETE EXAMPLE

Python (Pseudo-code using LangChain):

# 1. Chunking
docs = load_docs("docs/*.md")
chunks = chunk_docs(docs, chunk_size=512, overlap=64)

# 2. Embedding + Indexing
embeddings = OpenAIEmbeddings()
vector_store = Chroma()
vector_store.add_documents(chunks, embeddings)

# 3. Retrieval at query time
question = "What is the warranty policy?"
q_embedding = embeddings.embed_query(question)
retrieved = vector_store.similarity_search(q_embedding, k=4)

# 4. Prompt Assembly
context = "\n\n".join([c['content'] for c in retrieved])
prompt = f"""
You must answer based only on the provided context. If the answer is not found, say "Not found."

Context:
{context}

Question: {question}

Answer:
"""
answer = call_llm(prompt)
print(answer)

---

Quality Checklist

Before finalizing:

• [ ] All docs chunked, deduped, overlap set (50–100 tokens)
• [ ] Embedding model consistent (same for indexing/query)
• [ ] Vector DB indexed, queryable, low-latency (<1s)
• [ ] Retrieval recall tested (>85% for key questions)
• [ ] Prompt instructs LLM to answer only from context, fallback “Not found”
• [ ] Outputs logged for eval/debug

---

For advanced/hybrid, see [template-advanced-rag.md]. For evaluation, see [references/eval-patterns.md].

Fine-Tuning ROI Calculator

Purpose: Determine whether fine-tuning investment is justified for your use case.

---

ROI Framework

Core Formula

Net ROI = (Annual Benefits - Total Investment) / Total Investment × 100

Where:
- Annual Benefits = Cost Savings + Quality Value Improvement
- Total Investment = Data + Compute + Engineering + Maintenance

---

1. Current State Assessment

Baseline Metrics

Baseline Calculations

Annual LLM Cost = Monthly Spend × 12 = $_______

Quality Gap = Target Quality - Current Quality = ____%

---

2. Fine-Tuning Investment Estimate

One-Time Costs

Ongoing Costs (Annual)

Total Investment

Year 1 Total = One-Time + Annual = $_______
Year 2+ Total = Annual Only = $_______

---

3. Expected Benefits

Cost Reduction Benefits

Calculate Annual Cost Savings

Current Annual Cost: $_______

After Fine-Tuning:
- Token reduction: ____% → New input cost: $_______
- Model tier change: From $___/1M to $___/1M
- Retry reduction: ____% → Request reduction: ____

New Annual Cost: $_______

Annual Cost Savings: Current - New = $_______

Quality Improvement Value

---

4. ROI Calculation

Summary Table

ROI Metrics

Break-Even Point = One-Time Investment / Monthly Net Benefit = ___ months

Year 1 ROI = (Year 1 Benefits - Year 1 Investment) / Year 1 Investment = ____%

3-Year ROI = (3-Year Benefits - 3-Year Investment) / 3-Year Investment = ____%

NPV (10% discount) = Σ (Annual Net Benefit / (1.10)^n) - Initial Investment = $_______

---

5. Decision Framework

Go / No-Go Criteria

Decision Tree

Have >1,000 quality examples?
├─ No → Use prompt engineering (collect data first)
│
└─ Yes → Request volume >10k/month?
    ├─ No → Calculate: Is quality improvement worth $15-50k?
    │   ├─ Yes → Fine-tune (quality-driven)
    │   └─ No → Use prompt engineering
    │
    └─ Yes → Break-even <12 months?
        ├─ Yes → PASS Fine-tune (cost-driven)
        └─ No → Is quality improvement critical?
            ├─ Yes → Fine-tune (quality-driven)
            └─ No → Use prompt engineering + optimize

---

6. Risk Assessment

Risk Factors

Risk-Adjusted ROI

Risk Factor = Σ(Probability × Impact) = ____

Risk-Adjusted ROI = Base ROI × (1 - Risk Factor) = ____%

---

7. Recommendation Template

Executive Summary

Recommendation: [ ] Proceed with Fine-Tuning [ ] Defer [ ] Alternative Approach

Rationale: 1. ____________________ 2. ____________________ 3. ____________________

Key Metrics:

• Expected break-even: ___ months
• Year 1 ROI: ___%
• 3-Year NPV: $______

Risks and Mitigations:

• Risk 1: ____________________
• Mitigation: ____________________

Next Steps: 1. ____________________ 2. ____________________ 3. ____________________

---

Example Calculation

Scenario: Customer Support Chatbot

Current State:

• 50,000 requests/month
• $0.05/request (Claude Sonnet with long prompts)
• 82% quality score (target: 90%)
• Annual cost: $30,000

Fine-Tuning Investment:

• Data preparation: $15,000
• Training (PEFT): $500
• Evaluation: $5,000
• Engineering: $10,000
• Total one-time: $30,500
• Annual maintenance: $12,000

Expected Benefits:

• Shorter prompts (40% reduction): $12,000/year savings
• Smaller model possible: $0.02/request → $6,000/year savings
• Total cost savings: $18,000/year
• Quality improvement (82%→90%): Worth $25,000/year (fewer escalations)
• Total annual benefits: $43,000

ROI Calculation:

• Year 1 net: $43,000 - $42,500 = $500 (break-even)
• Year 2 net: $43,000 - $12,000 = $31,000
• 3-Year ROI: ($97,500 - $66,500) / $66,500 = 47%

Decision: Proceed - modest Year 1, strong Years 2-3

---

Related Resources

• [Cost Economics](../../references/cost-economics.md) - Cost modeling fundamentals
• [Fine-Tuning Recipes](../../references/fine-tuning-recipes.md) - Implementation patterns
• [Model Selection Matrix](model-selection-matrix.md) - Model comparison
• [Decision Matrices](../../references/decision-matrices.md) - Technology selection

---

LLM Model Selection Matrix

Purpose: Systematic model comparison with documented rationale for enterprise decisions.

---

Template Contract

Goals

• Select a model that meets quality, latency, cost, and governance requirements.
• Document rationale to enable repeatability, auditability, and future re-evaluation.

Inputs

• Task definition and acceptance criteria.
• Representative traffic samples and a golden eval set.
• Non-functional requirements: latency, throughput, availability, and budget.
• Governance requirements: privacy, residency, licensing, retention, compliance.

Decisions

• Primary model, fallback model(s), and routing criteria.
• Deployment mode (managed API, self-hosted, hybrid) and data handling posture.
• Change management gates (eval pass, canary, rollback).

Risks

• Vendor lock-in, deprecations, and pricing changes.
• Data leakage/retention and compliance violations.
• Silent regressions on upgrades (quality, safety, latency).

Metrics

• Task success rate, refusal correctness, and safety violation rate.
• Latency p95/p99, cost per request, and error rate.
• Stability metrics: variance across runs, drift sensitivity.

1. Task Requirements

Primary Task

• [ ] Task type: [ ] Generation [ ] Classification [ ] Extraction [ ] Reasoning [ ] Code [ ] Multimodal
• [ ] Quality requirement: [ ] High (critical path) [ ] Medium (user-facing) [ ] Acceptable (internal)
• [ ] Latency budget: ___ms P95
• [ ] Cost budget: $___/1K requests
• [ ] Context length needed: ___tokens
• [ ] Output length typical: ___tokens

Constraints

• [ ] Data residency: [ ] Any [ ] US-only [ ] EU-only [ ] Specific: ___
• [ ] Deployment: [ ] API-only [ ] On-premise [ ] Hybrid
• [ ] Fine-tuning needed: [ ] Yes [ ] No [ ] Maybe future
• [ ] Compliance: [ ] SOC2 [ ] HIPAA [ ] GDPR [ ] FedRAMP [ ] None

---

2. Model Comparison Matrix

Scoring Guide (1-10)

• Quality: Task-specific benchmark performance
• Latency: Time to first token + generation speed
• Cost: Per 1K token (input + output weighted)
• Context: Maximum context window size
• Reliability: Uptime, rate limits, consistency

Weights (must sum to 1.0)

• Quality: ___
• Latency: ___
• Cost: ___
• Context: ___
• License fit: ___

---

3. Licensing Matrix

---

4. Cost Projection

Per Request Estimate

• Average input tokens: ___
• Average output tokens: ___
• Requests per day: ___
• Requests per month: ___

---

5. Risk Assessment

---

6. Evaluation Results

Benchmark Scores (Task-Specific)

Latency Testing (P50/P95/P99)

---

7. Decision

Primary Model: _______________

Rationale: 1. _______________ 2. _______________ 3. _______________

Fallback Model: _______________

Fallback Trigger:

• [ ] Primary unavailable >30s
• [ ] Rate limit exceeded
• [ ] Cost threshold exceeded
• [ ] Other: _______________

Re-evaluation Date: _______________

---

8. Governance

Approval Chain

• [ ] Technical Lead: _______________ Date: ___
• [ ] Security Review: _______________ Date: ___
• [ ] Legal/Compliance: _______________ Date: ___
• [ ] Budget Approval: _______________ Date: ___

Change Management

• Model changes require: [ ] A/B test [ ] Evaluation suite pass [ ] Stakeholder approval
• Minimum notice for deprecation: ___ days
• Rollback procedure documented: [ ] Yes [ ] No

Advanced LLM Development Patterns

Advanced techniques for RLHF, pretraining, task-specific tuning, and production feedback loops.

---

Pattern 1: RLHF / Feedback Alignment Loop (Production-Friendly)

Use when: You need tighter alignment than SFT alone (safety, refusals, tone control)

Loop (Minimal Viable)

1. Collect preference data

• Pairwise rankings or scalar scores on model outputs (reward modeling)
• Include safety/refusal edge cases
• Gather from production logs or human labelers
• Ensure diverse coverage of task types

2. Train reward model (RM)

• Small model or head on frozen encoder/decoder
• Validate on held-out preferences
• Check for overfitting to labeler biases
• Measure inter-rater agreement

3. Policy optimization

• PPO / DPO / ORPO; prefer DPO/ORPO for simpler stacks (no rollout infra)
• Constrain KL divergence to base model
• Stop if degradation on held-out tasks
• Monitor reward hacking (gaming the RM)

4. Safety & regression eval

• Safety red-team set + task eval + format adherence
• Gate on "no new regressions"
• Verify refusal behavior intact
• Check for capability degradation

5. Package

• Ship RM + policy + configs
• Log KL, reward distribution, eval metrics
• Document training hyperparameters
• Maintain rollback capability

Checklist: RLHF pass

• [ ] Preference dataset balanced (task + safety)
• [ ] RM validated on held-out set
• [ ] KL/constraint tracked per step
• [ ] Regression + safety eval passed
• [ ] Policy + RM + configs versioned together
• [ ] Reward hacking monitored and mitigated
• [ ] Inter-rater agreement measured (if human labels)

---

Pattern 2: Pretraining Path (Tokenizer → Corpus → Schedule)

Use when: Building or heavily adapting a base model (from-scratch or major domain shift)

Tokenizer & Vocab Fit

• Train BPE/unigram on domain corpus
• Audit splits on code, math, URLs, PII markers
• Lock tokenizer before corpus filtering
• Validate coverage on representative samples
• Test efficiency (tokens per character)

Corpus Pipeline

• Mix domains (code/docs/web/structured)
• Dedupe (exact + near-dup with MinHash)
• Contamination scan against evals
• Filter low-quality/boilerplate (perplexity filters, heuristics)
• Balance multilingual if needed
• Document corpus composition and lineage

Training Schedule

• Warmup → cosine decay learning rate
• Gradient clipping (1.0 typical)
• EMA (Exponential Moving Average) optional
• Checkpoint/adapter save cadence
• Loss spikes watchdog (pause if gradient norm spikes)

Long-Context Plan

• Position encodings (RoPE/YaRN)
• Sliding-window attention where needed
• Budget KV cache size early
• Test on long-context benchmarks

Eval During Pretrain

• Perplexity slices per domain
• Probe tasks (code/math/narrative)
• Long-context stress set
• Stop if loss flattens while probes regress

Checklist: Pretraining ready

• [ ] Tokenizer validated on domain sample
• [ ] Corpus deduped, filtered, contamination-checked
• [ ] Learning rate schedule + clip + checkpoints defined
• [ ] Long-context attention + KV budget selected
• [ ] Probe eval + early-stop rules wired
• [ ] Corpus composition documented
• [ ] Contamination scan completed

---

Pattern 3: Task-Specific Tuning (Classify, Embed, Multimodal)

Use when: Going beyond chat to task/format-specific models

Classification/Extraction

• Use instruction or seq2seq format
• Add calibration (logits/temperature scaling)
• Enforce schema with constrained decoding
• Class-balance the dataset (oversample minority classes)
• Test on class-imbalanced holdout set

Embedding Models

• Optimize for retrieval/ranking tasks
• Mine hard negatives (similar but incorrect)
• Evaluate on Recall@K / nDCG metrics
• Test multilingual/domain drift
• Use contrastive loss (InfoNCE, triplet loss)

Multimodal Adapters

• Choose vision encoder + connector (CLIP, SigLIP)
• Align image tokens with text prompts
• Cap resolution to balance quality/compute
• Cache vision tower outputs
• Add safety filters on images (NSFW, harmful content)

Latency/Cost Fit

• Smaller heads/adapters where possible
• Quantize heads (int8, int4)
• Restrict max output length for classification/extraction
• Batch inference for throughput

Monitoring

• Per-class metrics (precision, recall, F1)
• Schema violations (malformed outputs)
• Drift on embeddings (centroid/dispersion)
• Multimodal failure modes (misaligned image-text)

Checklist: Task tuning safe

• [ ] Format/schema fixed and validated in eval
• [ ] Negatives/hard examples included
• [ ] Multimodal connector latency/cost measured (if used)
• [ ] Per-class/embedding drift monitored
• [ ] Safety filters for text + images active
• [ ] Calibration validated on holdout set

---

Pattern 4: Context Engineering Best Practices

Use when: Managing context and memory across LLM interactions

Key Insight: Context structure matters more than model selection. Even weaker LLMs perform well with proper context.

Progressive Disclosure

• Load context on-demand, not upfront
• Route by domain before retrieve
• Prioritize recent and relevant over exhaustive
• Use lazy loading for large knowledge bases

Session Management

• Treat sessions as conversation containers
• Honor framework differences (LangChain vs LlamaIndex)
• Share session handles safely across agents with scoped replay
• Implement session timeout and cleanup

Memory Provenance

• Track lineage (source, timestamp, approvals)
• Store only verifiable data
• Tag memory with confidence scores
• Version memory snapshots

Generation Triggers

• Extract/consolidate memory at phase boundaries
• Trigger after confidence drops below threshold
• Generate when new entities appear
• Periodic snapshots for long conversations

Background vs Blocking

• Run heavy writes async (embeddings, summarization)
• Keep blocking writes minimal for critical state
• Use queues for non-critical memory updates
• Prioritize read latency over write latency

Retrieval Timing

• Retrieve before high-impact actions
• Re-retrieve after state changes
• Enforce recency windows (e.g., last 24h for news)
• Cache frequently accessed contexts

Multimodal Context

• Normalize metadata across modalities
• Store text + embeddings separately
• Tag modalities (text/image/audio/video)
• Align timestamps across modalities

Fresh Contexts

• Spawn new agents with clean state
• Hydrate from validated memory only
• Avoid context pollution from previous tasks
• Test with and without context carryover

Checklist: Context engineering ready

• [ ] Progressive disclosure implemented
• [ ] Session management configured
• [ ] Memory provenance tracked
• [ ] Generation triggers defined
• [ ] Background/blocking writes separated
• [ ] Retrieval timing optimized
• [ ] Multimodal metadata normalized
• [ ] Fresh context spawning tested

---

Pattern 5: Production Monitoring & Observability

Use when: Deploying LLMs to production environments

Key Metrics to Track

Quality Metrics:

• Task success rate (did it complete the task?)
• Correctness score (is the output accurate?)
• Format compliance (schema violations)
• Refusal rate (appropriate vs over-cautious)

Performance Metrics:

• Latency (p50, p95, p99)
• Throughput (requests/second)
• Token usage (input + output)
• Cost per request

Safety Metrics:

• PII detection rate
• Toxicity/harmful content rate
• Jailbreak attempt detection
• Policy violation rate

Instrumentation

• Log every request/response with trace IDs
• Capture prompt templates and versions
• Store model outputs with timestamps
• Record user feedback (thumbs up/down, edits)

Alerting Rules

• Latency spike > 2x baseline
• Error rate > 5%
• Cost spike > 1.5x budget
• Safety violations > threshold
• Refusal rate drift > 10%

A/B Testing Framework

• Shadow new prompts/models before rollout
• Split traffic (5-10% canary)
• Compare metrics side-by-side
• Automated rollback on regression

Drift Detection

• Monitor output distribution shifts
• Track new entity types appearing
• Detect format changes over time
• Alert on vocabulary drift

Checklist: Monitoring ready

• [ ] All key metrics instrumented
• [ ] Trace IDs propagated
• [ ] Alerting rules configured
• [ ] A/B testing framework ready
• [ ] Drift detection active
• [ ] Cost tracking enabled

---

Pattern 6: Synthetic Data Generation

Use when: Insufficient real data for fine-tuning or evaluation

Use Cases

• Bootstrapping datasets for new domains
• Augmenting sparse classes in imbalanced datasets
• Generating edge cases and adversarial examples
• Creating evaluation sets for specific phenomena

Generation Strategies

Distillation:

• Use stronger model (GPT-4, Claude) to generate from weaker model prompts
• Validate outputs manually or with automated checks
• Ensure diversity in generated examples

Paraphrasing:

• Rephrase existing examples with semantic preservation
• Use multiple paraphrase models for diversity
• Validate meaning equivalence

Backtranslation:

• Translate to intermediate language and back
• Creates natural variations
• Test with multiple language pairs

Rule-Based Templates:

• Create templates with variable slots
• Fill slots programmatically
• Validate logical consistency

Quality Control

• Manual review of random samples (10-20%)
• Automated filters (length, perplexity, toxicity)
• Deduplication against real data
• Diversity metrics (unique n-grams, entity coverage)

Contamination Prevention

• Keep synthetic data separate from eval sets
• Track lineage (synthetic vs real)
• Hash all synthetic samples
• Periodic audits for leakage

Checklist: Synthetic data ready

• [ ] Generation strategy selected
• [ ] Quality control implemented
• [ ] Manual review completed
• [ ] Deduplication run
• [ ] Contamination prevention active
• [ ] Lineage tracking configured

---

Pattern 7: Model Compression & Optimization

Use when: Deploying to resource-constrained environments or reducing costs

Quantization

Post-Training Quantization (PTQ):

• int8: 2x smaller, minimal accuracy loss
• int4: 4x smaller, slight accuracy loss
• GPTQ, AWQ for weight-only quantization

Quantization-Aware Training (QAT):

• Simulate quantization during training
• Better accuracy preservation
• Requires full training access

Pruning

• Remove low-magnitude weights
• Structured pruning (entire neurons/layers)
• Iterative magnitude pruning
• Test after each pruning step

Distillation

• Train smaller "student" model from larger "teacher"
• Match logits or intermediate activations
• Combine with quantization for maximum compression
• Validate on diverse test set

Knowledge Distillation Workflow

1. Train large teacher model (or use existing) 2. Generate soft labels from teacher 3. Train smaller student model on soft + hard labels 4. Validate student performance 5. Iterate on student architecture if needed

Checklist: Compression ready

• [ ] Quantization strategy selected
• [ ] Accuracy validated post-compression
• [ ] Inference latency measured
• [ ] Model size reduction achieved (target: 2-4x)
• [ ] Production deployment tested

---

Pattern 8: Multi-Task Learning

Use when: Training a single model for multiple related tasks

Task Selection

• Choose related tasks with shared representations
• Ensure sufficient data per task
• Balance task difficulty
• Test negative transfer (tasks hurting each other)

Architecture Patterns

Shared Encoder + Task-Specific Heads:

• Common for classification tasks
• Efficient parameter sharing
• Easy to add new tasks

Multi-Task Transformer:

• Task tokens or prompts to indicate task
• Single unified output space
• More flexible but needs more data

Training Strategies

Task Sampling:

• Proportional to dataset size
• Temperature-based (flatten/sharpen distribution)
• Dynamic (based on task performance)

Loss Weighting:

• Equal weights baseline
• Uncertainty weighting (learn task weights)
• GradNorm (gradient-based balancing)

Evaluation

• Per-task metrics
• Average across tasks
• Worst-task performance (important for robustness)
• Test for negative transfer

Checklist: Multi-task ready

• [ ] Tasks selected and validated
• [ ] Architecture chosen
• [ ] Task sampling strategy defined
• [ ] Loss weighting configured
• [ ] Per-task evaluation implemented
• [ ] Negative transfer monitored

---

Agentic Patterns Best Practices

Purpose: Ready-to-apply design patterns and validation checklists for building, orchestrating, and safely operating agentic AI systems (LLM-based agents, multi-agent workflows, and tool-using LLMs).

---

Core Patterns

---

Pattern 1: Agent Workflow Loop

Use when: Designing any LLM-powered agent (single or multi-tool), especially those that must interact with APIs, databases, users, or other agents.

Structure:

1. Receive mission/task input (user or upstream agent)
2. Perceive (gather info, context, or state)
3. Plan (select tools/steps, break into subtasks)
4. Act (call tools/APIs, make changes, send messages)
5. Observe results (validate, update memory/context)
6. Loop: Replan or finish if task completed/failed
7. Escalate or fallback if goal unreachable or error encountered

Checklist:

• [ ] Mission/goal always explicit and parsed
• [ ] Perception step: can access latest context/state
• [ ] Plan step decomposes nontrivial tasks, logs plan
• [ ] Action step: every tool/API call logged, has error handling
• [ ] Observations update agent memory/context
• [ ] Loop capped (step limit, time limit, watchdog)
• [ ] Escalation or fallback for dead ends/errors

---

Pattern 2: Tool Use & Reflection

Use when: Your agent needs to interact with external APIs, retrieve data, or perform multi-step reasoning.

Structure:

1. Recognize when tool/API is needed (tool call trigger)
2. Format inputs (normalize, check for required params)
3. Call tool/API, handle possible errors/timeouts
4. Reflect on results: was the output usable? (if not, replan or escalate)
5. Update memory/log of tool usage for auditability

Checklist:

• [ ] Tools/API schemas versioned and validated
• [ ] Inputs validated before each call
• [ ] Outputs parsed/validated before use
• [ ] Tool call failures handled (retry, fallback, escalate)
• [ ] Reflection step checks if tool solved subgoal

---

Pattern 3: Multi-Agent Collaboration

Use when: Building systems where multiple agents specialize, collaborate, or compete to solve complex tasks.

Structure:

1. Assign roles/capabilities to each agent
2. Define hand-off rules (when/how to delegate)
3. Establish communication protocol (messages, APIs, shared memory)
4. Synchronize state as needed (avoid race conditions)
5. Arbitration: resolve conflicting actions or deadlocks
6. Monitor, audit, and escalate when cooperation fails

Checklist:

• [ ] Roles and responsibilities of each agent defined
• [ ] Hand-off protocols tested for all task boundaries
• [ ] All agent-agent comms logged and auditable
• [ ] Deadlock/loop detection in place
• [ ] Arbitration logic for conflicting outcomes
• [ ] Multi-agent eval test suite (edge, error, and abuse cases)

---

Decision Matrices

Agent Type Selection Table

---

Step Limit and Escalation Matrix

---

Common Mistakes & Anti-Patterns

---

[FAIL] No step/loop cap: Agent runs indefinitely, racks up costs, or gets stuck. [OK] Instead: Always enforce max step/timeout per agent/task; log and abort if limit hit.

[FAIL] No tool validation: Assumes API/tool will always return correct/expected data. [OK] Instead: Always parse and validate tool output before use; retry or escalate on error.

[FAIL] Agent hand-off chaos: No clear protocol for when/how agents delegate, causing confusion, missed hand-offs, or data races. [OK] Instead: Define hand-off conditions and message schemas; log every transfer.

[FAIL] Memory bleed: Agents keep growing memory/context unchecked, eventually failing or hallucinating. [OK] Instead: Use context pruning, summarize memory, enforce hard context/window limits.

[FAIL] Reflection step missing: Agent blindly acts without evaluating outcome of tool/API use. [OK] Instead: After every action, explicitly check if goal was advanced, else replan/escalate.

---

Quick Reference

Agentic System Production Checklist

• [ ] Workflow loop (perceive → plan → act → observe → replan) coded and tested
• [ ] Tool/API schema versioning and validation
• [ ] Hand-off and comms protocols documented, tested, and logged
• [ ] All loops/steps capped, watchdog in place
• [ ] All agent actions, plans, and tool calls auditable
• [ ] Reflection and fallback/escalation paths coded
• [ ] Multi-agent eval suite with edge/error/abuse scenarios

---

Emergency Playbook

• If agent stuck in loop:

1. Abort after max step/time, log event 2. Output best effort/partial result, escalate for review

• If tool/API fails or abused:

1. Retry, fallback to backup tool 2. Block tool if abuse detected, alert/triage

• If multi-agent hand-off fails:

1. Trigger arbitration/fallback agent 2. Escalate to operator if still unresolved

---

Further Resources

See data/sources.json for:

• Agent frameworks: LangChain, LangGraph, CrewAI, Google Agent Developer Kit
• Reference patterns: ReAct, Reflection, Multi-agent, Tool Use, Guardrails

---

Next: See references/prompt-engineering-patterns.md for ready-to-use prompt templates, checklists, and validation guides.

Anti-Patterns

Common LLM engineering mistakes and how to prevent them. Learn from production failures.

---

Data Leakage

Problem: Test set data or user content included in training set, inflating performance metrics.

Symptoms:

• Suspiciously high evaluation scores
• Poor performance in production vs test
• Model memorizing specific examples
• Evaluation metrics don't match user experience

Prevention:

# Hash-based deduplication
def deduplicate_splits(train_data, test_data):
    """Ensure zero overlap between train and test"""
    train_hashes = {hash_example(ex) for ex in train_data}
    test_hashes = {hash_example(ex) for ex in test_data}

    # Check for overlap
    overlap = train_hashes & test_hashes
    if overlap:
        raise ValueError(f"Found {len(overlap)} overlapping examples")

    return train_data, test_data

# Time-based split for temporal data
def temporal_split(data, test_start_date):
    """Ensure test data is strictly after training data"""
    train = [ex for ex in data if ex.date < test_start_date]
    test = [ex for ex in data if ex.date >= test_start_date]
    return train, test

# Monitor for leakage events
def monitor_leakage():
    """Continuous monitoring for data leakage"""
    alerts = []

    # Check: Test examples appearing in training logs
    if test_examples_in_training_logs():
        alerts.append("Test data in training logs")

    # Check: User data in training pipeline
    if user_data_in_training():
        alerts.append("User data leaked into training")

    # Check: Eval metrics too good to be true
    if eval_metrics_suspicious():
        alerts.append("Suspicious evaluation scores")

    return alerts

Detection:

• Compare train/test set overlap with checksums
• Monitor evaluation vs production metrics divergence
• Audit training data provenance
• Regular data lineage reviews

Best practices:

• Isolate splits before any processing
• Use content hashing for deduplication
• Version all datasets with timestamps
• Automated leakage detection in CI/CD

---

Prompt Dilution

Problem: Too many instructions/examples, exceeding context window, causing model to ignore key task.

Symptoms:

• Model ignoring critical instructions
• Inconsistent behavior across requests
• Following some instructions but not others
• Performance degrading with prompt complexity

Prevention:

# Test prompt lengths
def validate_prompt_length(prompt, max_tokens=4000):
    """Ensure prompt fits in context window with room for response"""
    token_count = count_tokens(prompt)

    if token_count > max_tokens:
        raise ValueError(
            f"Prompt too long: {token_count} tokens (max: {max_tokens})"
        )

    # Check: Instructions are clear and prioritized
    if not has_clear_priority(prompt):
        warnings.warn("Prompt lacks clear instruction priority")

    return token_count

# Trim low-value context
def optimize_prompt(prompt, max_tokens=4000):
    """Remove low-value content to fit budget"""
    sections = parse_prompt_sections(prompt)

    # Priority order
    priorities = [
        "core_task",         # Always keep
        "constraints",       # Keep if space
        "examples",          # Trim to 2-3
        "background_context" # Remove if needed
    ]

    optimized = []
    token_budget = max_tokens

    for priority in priorities:
        section = sections.get(priority)
        if section:
            section_tokens = count_tokens(section)

            if section_tokens <= token_budget:
                optimized.append(section)
                token_budget -= section_tokens
            elif priority == "examples":
                # Keep fewer examples
                trimmed = trim_examples(section, token_budget)
                optimized.append(trimmed)
                break

    return "\n\n".join(optimized)

# Scoring metrics for prompt quality
def score_prompt_quality(prompt):
    """Evaluate prompt effectiveness"""
    scores = {
        "clarity": measure_instruction_clarity(prompt),
        "conciseness": measure_conciseness(prompt),
        "completeness": measure_completeness(prompt),
        "token_efficiency": measure_token_efficiency(prompt)
    }

    # Fail if any dimension below threshold
    if any(score < 0.7 for score in scores.values()):
        warnings.warn(f"Low prompt quality: {scores}")

    return scores

Detection:

• Monitor prompt token lengths
• Test with varying prompt complexity
• A/B test simplified vs complex prompts
• Track instruction-following metrics

Best practices:

• Single clear task per prompt
• Prioritize most important instructions
• Use structured format (XML tags, JSON)
• Test at max expected prompt length
• Remove redundant or conflicting instructions

---

RAG Context Overload

Problem: Too many irrelevant chunks in context, degrading LLM accuracy.

Symptoms:

• Model ignoring relevant retrieved information
• Hallucinating despite having correct context
• Low groundedness scores
• Citing wrong sources

Prevention:

# Tighten retrieval
def retrieve_with_threshold(query, min_score=0.7):
    """Only retrieve highly relevant chunks"""
    results = vector_db.search(query, top_k=20)

    # Filter by relevance score
    filtered = [r for r in results if r.score >= min_score]

    if len(filtered) == 0:
        # Fallback: lower threshold or return None
        filtered = results[:3]  # Top 3 as backup

    return filtered[:5]  # Max 5 chunks

# Apply rerankers
def retrieve_with_reranking(query, initial_k=20, final_k=5):
    """Two-stage retrieval with reranking"""
    # Stage 1: Fast vector search
    candidates = vector_db.search(query, top_k=initial_k)

    # Stage 2: Cross-encoder reranking
    reranked = reranker.rank(
        query=query,
        documents=[c.text for c in candidates]
    )

    # Take top-k after reranking
    return reranked[:final_k]

# Compress or filter context
def compress_context(chunks, max_tokens=2000):
    """Compress retrieved chunks to fit budget"""
    if sum(count_tokens(c) for c in chunks) <= max_tokens:
        return chunks

    # Strategy 1: Summarize each chunk
    compressed = [
        summarize_chunk(c, max_tokens=200)
        for c in chunks
    ]

    # Strategy 2: Extract key sentences
    compressed = [
        extract_key_sentences(c, max_sentences=3)
        for c in chunks
    ]

    # Strategy 3: Deduplicate information
    compressed = deduplicate_chunks(chunks)

    return compressed

# Monitor context quality
def monitor_rag_quality():
    """Track RAG performance metrics"""
    metrics = {
        "retrieval_recall": measure_recall(),      # >85%
        "groundedness": measure_groundedness(),    # >95%
        "hallucination_rate": measure_hallucination(), # <3%
        "citation_accuracy": measure_citations()  # >90%
    }

    for metric, value in metrics.items():
        if value < thresholds[metric]:
            alert(f"{metric} below threshold: {value}")

    return metrics

Detection:

• Monitor groundedness metrics
• Track hallucination rates
• Measure citation accuracy
• A/B test different retrieval strategies

Best practices:

• Retrieve fewer, higher-quality chunks (5-10 optimal)
• Use reranking for better relevance
• Set minimum relevance score threshold
• Compress context if needed (summarization, key sentence extraction)
• Monitor retrieval quality continuously

---

Agentic Runaway

Problem: Agents stuck in loop, redundant tool calls, or unsafe escalation.

Symptoms:

• Agent exceeding step limits
• Repeated identical tool calls
• Oscillating between states
• High costs from redundant API calls
• Unsafe actions without proper validation

Prevention:

# Max step limits
class Agent:
    def __init__(self, max_steps=10):
        self.max_steps = max_steps
        self.step_count = 0
        self.action_history = []

    def run(self, task):
        while self.step_count < self.max_steps:
            # Detect loops
            if self.is_looping():
                return self.handle_loop()

            action = self.plan_next_action(task)

            # Validate action
            if not self.is_action_safe(action):
                return self.escalate_unsafe_action(action)

            result = self.execute_action(action)
            self.step_count += 1
            self.action_history.append(action)

            if self.is_task_complete():
                return self.finalize()

        return self.handle_max_steps_exceeded()

    def is_looping(self):
        """Detect if agent is stuck in a loop"""
        if len(self.action_history) < 3:
            return False

        # Check: Same action repeated
        last_3 = self.action_history[-3:]
        if len(set(last_3)) == 1:
            return True

        # Check: Oscillating between two actions
        if len(set(last_3)) == 2 and last_3[0] == last_3[2]:
            return True

        return False

    def handle_loop(self):
        """Recovery from detected loop"""
        # Option 1: Try different approach
        alternative_plan = self.generate_alternative_plan()
        if alternative_plan:
            return self.execute_plan(alternative_plan)

        # Option 2: Escalate to human
        return self.escalate("Agent stuck in loop")

# Tool rate limits
class RateLimitedTool:
    def __init__(self, tool, max_calls_per_minute=10):
        self.tool = tool
        self.max_calls = max_calls_per_minute
        self.call_history = []

    def execute(self, params):
        """Execute with rate limiting"""
        # Check rate limit
        recent_calls = self.count_recent_calls(window=60)
        if recent_calls >= self.max_calls:
            raise RateLimitError(
                f"Tool {self.tool.name} rate limit exceeded"
            )

        result = self.tool.execute(params)
        self.call_history.append(time.time())
        return result

# Explicit fallback/abort paths
class SafeAgent:
    def __init__(self, fallback_strategy="escalate"):
        self.fallback_strategy = fallback_strategy

    def execute_with_fallback(self, action):
        """Execute action with fallback handling"""
        try:
            result = self.execute_action(action)

            # Validate result
            if not self.is_result_valid(result):
                return self.fallback()

            return result

        except ToolError as e:
            return self.fallback(error=e)

    def fallback(self, error=None):
        """Fallback strategy for failures"""
        if self.fallback_strategy == "escalate":
            return self.escalate_to_human(error)

        elif self.fallback_strategy == "retry_alternative":
            return self.try_alternative_approach()

        elif self.fallback_strategy == "graceful_degradation":
            return self.provide_best_effort_response()

        else:
            raise ValueError(f"Unknown fallback: {self.fallback_strategy}")

Detection:

• Monitor step count distribution
• Track repeated tool calls
• Detect oscillating patterns
• Alert on excessive retries

Best practices:

• Set max step limits (5-20 depending on task complexity)
• Implement loop detection
• Tool rate limiting per agent
• Explicit fallback strategies
• Human escalation for failures
• Audit trail for debugging

---

Over-Engineering

Problem: Building complex systems when simple solutions would work.

Symptoms:

• High maintenance burden
• Difficult to debug
• Slow iteration speed
• Engineers don't understand the system

Prevention:

Start simple:

# DON'T: Build multi-agent system for simple task
class ComplexSystem:
    def __init__(self):
        self.orchestrator = OrchestratorAgent()
        self.specialist1 = SpecialistAgent("domain1")
        self.specialist2 = SpecialistAgent("domain2")
        self.validator = ValidatorAgent()
        self.router = RouterAgent()

# DO: Single prompt for simple task
def simple_solution(query):
    prompt = f"""Answer the question concisely: {query}"""
    return llm.generate(prompt)

Progressive complexity: 1. Start with single prompt 2. Add RAG if knowledge needed 3. Add tools if actions needed 4. Add agents if orchestration needed

Complexity checklist:

• [ ] Can this be solved with a better prompt?
• [ ] Can this be solved with RAG?
• [ ] Can this be solved with a single agent?
• [ ] Do I really need multiple agents?

---

Ignoring Evaluation

Problem: Deploying without measuring quality, leading to production failures.

Symptoms:

• Users reporting poor quality
• No baseline for improvement
• Can't measure impact of changes
• Regressions going unnoticed

Prevention:

# Automated regression tests
def test_regression():
    """Run on every prompt/model change"""
    golden_set = load_golden_test_set()

    results = []
    for example in golden_set:
        prediction = llm.generate(example.input)
        score = evaluate(prediction, example.expected)
        results.append(score)

    avg_score = sum(results) / len(results)

    # Block deployment if regression
    if avg_score < QUALITY_THRESHOLD:
        raise RegressionError(f"Quality below threshold: {avg_score}")

# Multi-metric evaluation
def evaluate_llm_system(test_set):
    """Comprehensive evaluation suite"""
    metrics = {
        "accuracy": measure_accuracy(test_set),
        "hallucination_rate": measure_hallucination(test_set),
        "groundedness": measure_groundedness(test_set),
        "latency_p95": measure_latency(test_set),
        "cost_per_request": measure_cost(test_set),
        "user_satisfaction": measure_satisfaction(test_set)
    }

    # All metrics must pass
    failed = [
        m for m, v in metrics.items()
        if v < thresholds[m]
    ]

    if failed:
        raise EvaluationError(f"Failed metrics: {failed}")

    return metrics

Best practices:

• Create golden test set (100+ examples)
• Run automated tests on every change
• Track multiple metrics (not just accuracy)
• Set quality thresholds and gates
• Regular human evaluation

---

Hard-Coded Prompts

Problem: Prompts embedded in code instead of versioned and tested separately.

Symptoms:

• Difficult to iterate on prompts
• No version history
• Can't A/B test easily
• Code changes required for prompt updates

Prevention:

# DON'T: Hard-code prompts
def generate_response(query):
    prompt = "You are a helpful assistant. Answer: " + query
    return llm.generate(prompt)

# DO: Version and test prompts separately
class PromptManager:
    def __init__(self, version="v1"):
        self.prompts = self.load_prompts(version)

    def load_prompts(self, version):
        """Load prompts from versioned files"""
        return yaml.safe_load(
            open(f"prompts/{version}/prompts.yaml")
        )

    def get_prompt(self, name, **kwargs):
        """Get prompt template with variables"""
        template = self.prompts[name]
        return template.format(**kwargs)

# Usage
prompt_manager = PromptManager(version="v2")
prompt = prompt_manager.get_prompt("qa_assistant", query=query)
response = llm.generate(prompt)

Best practices:

• Store prompts in separate files (YAML, JSON)
• Version control prompts
• CI/CD for prompt testing
• A/B testing framework
• Prompt template system with variables

---

Missing Observability

Problem: No logging/tracing, making debugging impossible.

Symptoms:

• Can't debug production issues
• No visibility into failures
• Can't measure performance
• User complaints with no context

Prevention:

# Comprehensive logging
import structlog

logger = structlog.get_logger()

def llm_call_with_logging(prompt, trace_id):
    """LLM call with full observability"""
    start_time = time.time()

    logger.info(
        "llm_call_start",
        trace_id=trace_id,
        prompt_tokens=count_tokens(prompt),
        model=model_name
    )

    try:
        response = llm.generate(prompt)

        logger.info(
            "llm_call_success",
            trace_id=trace_id,
            latency_ms=(time.time() - start_time) * 1000,
            response_tokens=count_tokens(response),
            cost=calculate_cost(prompt, response)
        )

        return response

    except Exception as e:
        logger.error(
            "llm_call_failed",
            trace_id=trace_id,
            error=str(e),
            latency_ms=(time.time() - start_time) * 1000
        )
        raise

# Distributed tracing
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("rag_query")
def rag_query(query, trace_id):
    """RAG with distributed tracing"""
    span = trace.get_current_span()
    span.set_attribute("query", query)
    span.set_attribute("trace_id", trace_id)

    # Retrieval span
    with tracer.start_as_current_span("retrieval"):
        chunks = retrieve(query)
        span.set_attribute("chunks_retrieved", len(chunks))

    # Generation span
    with tracer.start_as_current_span("generation"):
        response = generate(query, chunks)
        span.set_attribute("response_length", len(response))

    return response

Best practices:

• Log all LLM calls with full context
• Distributed tracing with trace IDs
• Structured logging (JSON)
• Metrics dashboard (latency, cost, quality)
• Error tracking and alerting

---

Related Resources

• [Common Design Patterns](common-design-patterns.md) - Correct implementation patterns
• [Production Checklists](production-checklists.md) - Pre-deployment validation
• [LLMOps Best Practices](llmops-best-practices.md) - Operational standards
• [Evaluation Patterns](eval-patterns.md) - Quality measurement and testing

---

Dataset Formatting Guide (Instruction, Chat, Transformation)

Templates and rules for building clean, consistent datasets for SFT and instruction tuning.

---

1. Instruction Format (Recommended)

Each example:

{ "instruction": "<what user wants>", "input": "<optional context>", "output": "<ideal response>" }

Rules

• Use empty string for missing inputs
• Keep outputs concise
• Avoid multi-step reasoning unless required

---

2. Chat Format (Multi-Turn)

{ "messages": [ {"role": "system", "content": "<policy/role>"}, {"role": "user", "content": "<query>"}, {"role": "assistant", "content": "<ideal reply>"} ] }

Rules

• No overlapping roles
• Ensure each conversation is self-contained
• Avoid leaking system prompts in assistant outputs

---

3. Transformation Format (Simple I/O)

{ "input": "<raw text>", "output": "<transformed text>" }

Use for:

• Rewriting
• Summarization
• Classification
• Extraction

---

4. Dataset Hygiene Rules

A. No Leakage

• Do not include system prompts in model outputs
• Do not include personal info
• Remove timestamps or IDs that encode answers

B. Deduplication

• Remove near-duplicate samples
• Deduplicate across categories

C. Quality Enforcement

• Each output = ideal model response
• Avoid ambiguous tasks
• Avoid mixed languages unless intentional

---

5. Formatting Deliverables Checklist

• [ ] JSONL validated with jq
• [ ] UTF-8 encoded
• [ ] No trailing commas
• [ ] Uniform field names
• [ ] Balanced sample distribution
• [ ] Full dataset documented in README

LLMOps Best Practices

Purpose: Practical guidance for executing and maintaining production-grade LLM systems, from project validation through deployment, monitoring, and continuous improvement.

---

Core Patterns

---

Pattern 1: LLM Project Preflight & Lifecycle

Use when: Starting any LLM, RAG, or agentic AI project—especially for production or critical deployments.

Structure:

1. Define business/user goal (clear success criteria)
2. Identify target data sources (access, licensing, update schedule)
3. Data curation pipeline: deduplication, filtering, PII scan, split
4. Model path: choose (API, OSS, custom fine-tune)
5. Plan for eval: select metrics, test suites, human-in-the-loop
6. Deployment target: resource plan, scaling, rollback/upgrade
7. Observability: logs, alerts, usage/latency dashboards
8. Safety & guardrails: input/output filtering, abuse/escalation

Checklist:

• [ ] Clear user/business goal defined
• [ ] Data curation pipeline (dedup, filter, PII) in place
• [ ] Pretraining/fine-tune plan with rollback/checkpoints
• [ ] Evaluation plan (metrics, regression, human review)
• [ ] Deployment resource + rollback
• [ ] Observability: logs, dashboards, alerting
• [ ] Safety: guardrails, escalation, abuse handling

---

Pattern 2: LLMOps Lifecycle—End-to-End Steps

---

Pattern 3: Production Readiness—Quality Gates

Before launch, pass all:

• [ ] Data: Source checked, up-to-date, deduped, filtered
• [ ] Model: Eval pass on regression, hallucination <3%
• [ ] Prompts: All core prompts versioned & regression-tested
• [ ] RAG: Retrieval recall >85%, context window fits key data
• [ ] Agents: Tool use/plan reproducible, no infinite loops
• [ ] Safety: Abuse cases filtered, critical output escalation

---

Decision Matrices

Model Path Selection Table

---

Deployment Pattern Matrix

---

Common Mistakes & Anti-Patterns

---

[FAIL] Data Drift Ignored: No schedule for re-curating data, leading to outdated/irrelevant model behavior. [OK] Instead: Automate data refresh, log drift, set up re-ingestion triggers.

[FAIL] No Prompt Versioning: Ad-hoc edits break workflows, regressions sneak in. [OK] Instead: Store all prompt templates in version control, run regression suites after edit.

[FAIL] No Rollback Plan: Deployments go live without a way to revert on failure. [OK] Instead: Canary deployments, automatic rollback, fast “last known good” fallback.

[FAIL] Observability Gaps: No logging/alerts for latency, OOM, or safety. [OK] Instead: Add logs, health pings, resource use monitors, safety/abuse event logs.

[FAIL] Ignoring Edge Cases in Eval: Models tested only on “happy path.” [OK] Instead: Test all common and edge cases—short, long, adversarial, weird user input.

---

Quick Reference

LLMOps Production Checklist

• [ ] Data freshness + deduplication validated
• [ ] Model eval: hallucination, faithfulness, latency pass
• [ ] Prompt templates: versioned, tested, edge cases
• [ ] RAG: top-k retrieval recall and source tracking
• [ ] Agent workflows: tool flow, fallback, step limit
• [ ] Monitoring: latency, cost, abuse, drift
• [ ] Rollback path and escalation ready

---

Emergency Playbook

• If hallucinations spike:

1. Route queries to RAG/grounded mode 2. Tighten retrieval, compress context 3. Trigger rollback to previous checkpoint/model

• If cost/latency spikes:

1. Switch to quantized model, reduce context window 2. Batch/stream processing, enable autoscaling 3. Alert/auto-disable non-critical features

• If user abuse detected:

1. Auto-block, log event, alert on-call 2. Escalate critical cases, triage, patch prompts/filters

---

Further Resources

See data/sources.json for:

• OpenAI, Anthropic, Gemini, HuggingFace, vLLM, LlamaIndex, LangChain, PEFT, DeepSpeed, LangSmith, W&B, and more.

---

Next: See references/rag-best-practices.md for copy-paste RAG/Retrieval patterns and validation guides.