Ai Agents

AI Agents Development — Production Skill Hub

Modern Best Practices (March 2026): deterministic control flow, bounded tools, auditable state, MCP-based tool integration, handoff-first orchestration, multi-layer guardrails, OpenTelemetry tracing, and human-in-the-loop controls (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

This skill provides production-ready operational patterns for designing, building, evaluating, and deploying AI agents. It centralizes procedures, checklists, decision rules, and templates used across RAG agents, tool-using agents, OS agents, and multi-agent systems.

No theory. No narrative. Only operational steps and templates.

---

When to Use This Skill

Codex should activate this skill whenever the user asks for:

• Designing an agent (LLM-based, tool-based, OS-based, or multi-agent).
• Scoping capability maturity and rollout risk for new agent behaviors.
• Creating action loops, plans, workflows, or delegation logic.
• Writing tool definitions, MCP tools, schemas, or validation logic.
• Generating RAG pipelines, retrieval modules, or context injection.
• Building memory systems (session, long-term, episodic, task).
• Creating evaluation harnesses, observability plans, or safety gates.
• Preparing CI/CD, rollout, deployment, or production operational specs.
• Producing any template in /references/ or /assets/.
• Implementing MCP servers or integrating Model Context Protocol.
• Setting up agent handoffs and orchestration patterns.
• Configuring multi-layer guardrails and safety controls.
• Evaluating whether to build an agent (build vs not decision).
• Calculating agent ROI, token costs, or cost/benefit analysis.
• Assessing hallucination risk and mitigation strategies.
• Deciding when to kill an agent project (kill triggers).
• For prompt scaffolds, retrieval tuning, or security depth, see Scope Boundaries below.

Scope Boundaries (Use These Skills for Depth)

• Prompt scaffolds & structured outputs → ai-prompt-engineering
• RAG retrieval & chunking → ai-rag
• Search tuning (BM25/HNSW/hybrid) → ai-rag
• Security/guardrails → ai-mlops
• Inference optimization → ai-llm-inference

Default Workflow (Production)

• Pick an architecture with the Decision Tree (below); default to workflow/FSM/DAG for production.
• Draft an agent spec with `assets/core/agent-template-standard.md` (or `assets/core/agent-template-quick.md`).
• Specify tools and handoffs with JSON Schema using `assets/tools/tool-definition.md` and `references/api-contracts-for-agents.md`.
• Add retrieval only when needed; start with `assets/rag/rag-basic.md` and scale via `assets/rag/rag-advanced.md` + `references/rag-patterns.md`.
• Add eval + telemetry early via `references/evaluation-and-observability.md`.
• Run the go/no-go gate with `assets/checklists/agent-safety-checklist.md`.
• Plan deploy/rollback and safety controls via `references/deployment-ci-cd-and-safety.md`.

---

Quick Reference

Framework Selection (March 2026)

Tier 1 — Production-Grade

Tier 2 — Specialized

Tier 3 — Niche

See `references/modern-best-practices.md` for detailed comparison and selection guide.

Framework Deep Dives

• Claude Agent SDK - `references/claude-agent-sdk-patterns.md`

Agent definition, built-in tools (Bash, TextEditor, Computer), MCP servers, guardrails, multi-agent, streaming events

• Pydantic AI - `references/pydantic-ai-patterns.md`

Type-safe agents, MCP toolsets, native A2A, pydantic-graph FSM, durable execution, HITL, TestModel testing

---

Decision Tree: Choosing Agent Architecture

What does the agent need to do?
    ├─ Answer questions from knowledge base?
    │   ├─ Simple lookup? → RAG Agent (LangChain/LlamaIndex + vector DB)
    │   └─ Complex multi-step? → Agentic RAG (iterative retrieval + reasoning)
    │
    ├─ Perform external actions (APIs, tools, functions)?
    │   ├─ 1-3 tools, linear flow? → Tool-Using Agent (LangGraph + MCP)
    │   └─ Complex workflows, branching? → Planning Agent (ReAct/Plan-Execute)
    │
    ├─ Write/modify code autonomously?
    │   ├─ Single file edits? → Tool-Using Agent with code tools
    │   └─ Multi-file, issue resolution? → Code/SWE Agent (HyperAgent pattern)
    │
    ├─ Delegate tasks to specialists?
    │   ├─ Fixed workflow? → Multi-Agent Sequential (A → B → C)
    │   ├─ Manager-Worker? → Multi-Agent Hierarchical (Manager + Workers)
    │   └─ Dynamic routing? → Multi-Agent Group Chat (collaborative)
    │
    ├─ Control desktop/browser?
    │   └─ OS Agent (Anthropic Computer Use + MCP for system access)
    │
    └─ Hybrid (combination of above)?
        └─ Planning Agent that coordinates:
            - Tool-using for actions (MCP)
            - RAG for knowledge (MCP)
            - Multi-agent for delegation (A2A)
            - Code agents for implementation

Protocol Selection:

• Use MCP for: Tool access, data retrieval, single-agent integration
• Use A2A for: Agent-to-agent handoffs, multi-agent coordination, task delegation

Framework Selection (after choosing architecture):

Which framework?
    ├─ MVP/Prototyping?
    │   ├─ Python → OpenAI Agents SDK or CrewAI
    │   └─ TypeScript → Mastra or Claude Agent SDK
    │
    ├─ Production →
    │   ├─ Auditability/compliance? → LangGraph
    │   ├─ Type safety + MCP/A2A native? → Pydantic AI
    │   ├─ Anthropic models + Computer Use? → Claude Agent SDK
    │   ├─ Google Cloud / Gemini? → Google ADK
    │   ├─ Azure / .NET / Java? → MS Agent Framework
    │   ├─ AWS managed? → Bedrock Agents
    │   └─ RAG-heavy? → LlamaIndex Workflows
    │
    ├─ Minimalist / Research →
    │   ├─ Fewest LLM calls? → SmolAgents
    │   └─ Optimize prompts automatically? → DSPy
    │
    └─ Enterprise pipeline → Haystack

---

Core Concepts (Vendor-Agnostic)

Control Flow Options

• Reactive: direct tool routing per user request (fast, brittle if unbounded).
• Workflow (FSM/DAG): explicit states and transitions (default for deterministic production).
• Planner/Executor: plan with strict budgets, then execute step-by-step (use when branching is unavoidable).
• Orchestrated multi-agent: separate roles with validated handoffs (use when specialization is required).

Memory Types (Tradeoffs)

• Short-term (session): cheap, ephemeral; best for conversational continuity.
• Episodic (task): scoped to a case/ticket; supports audit and replay.
• Long-term (profile/knowledge): high risk; requires consent, retention limits, and provenance.

Failure Handling (Production Defaults)

• Classify errors: retriable vs fatal vs needs-human.
• Bound retries: max attempts, backoff, jitter; avoid retry storms.
• Fallbacks: degraded mode, smaller model, cached answers, or safe refusal.

Do / Avoid

Do

• Do keep state explicit and serializable (replayable runs).
• Do enforce tool allowlists, scopes, and idempotency for side effects.
• Do log traces/metrics for model calls and tool calls (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).

Avoid

• Avoid runaway autonomy (unbounded loops or step counts).
• Avoid hidden state (implicit memory that cannot be audited).
• Avoid untrusted tool outputs without validation/sanitization.

Navigation: Economics & Decision Framework

Should You Build an Agent?

• Build vs Not Decision Framework - `references/build-vs-not-decision.md`
• 10-second test (volume, cost, error tolerance)
• Red flags and immediate disqualifiers
• Alternatives to agents (usually better)
• Full decision tree with stage gates
• Kill triggers during development and post-launch
• Pre-build validation checklist

Agent ROI & Token Economics

• Agent Economics - `references/agent-economics.md`
• Token pricing by model (January 2026)
• Cost per task by agent type
• ROI calculation formula and tiers
• Hallucination cost framework and mitigation ROI
• Investment decision matrix
• Monthly tracking dashboard

---

Navigation: AI Engine Layers

Five-layer architecture for production agent systems. Start with the overview, then drill into layer-specific patterns.

• AI Engine Architecture — `references/ai-engine-layers.md`

5-layer composition model, layer interaction matrix, implementation phases

• Context Graph Patterns — `references/context-graph-patterns.md`

Node/edge schema, traversal patterns, graph-RAG, memory tiers, conflict detection

• Inbox Engine Patterns — `references/inbox-engine-patterns.md`

Event-driven intake, signal classification, deduplication, priority routing, dead letter

• Knowledge Base Architecture — `assets/knowledge-base/kb-architecture.md`

Unified KB schema (vector + graph + doc index), provenance, freshness, multi-tenant

Action Graph → covered by `references/operational-patterns.md` + `references/agent-operations-best-practices.md`

Data Agent → covered by `../ai-rag/SKILL.md` + `references/rag-patterns.md`

---

Navigation: Core Concepts & Patterns

Governance & Maturity

• Agent Maturity & Governance - `references/agent-maturity-governance.md`
• Capability maturity levels (L0-L4)
• Identity & policy enforcement
• Fleet control and registry management
• Deprecation rules and kill switches

Modern Best Practices

• Modern Best Practices - `references/modern-best-practices.md`
• Model Context Protocol (MCP)
• Agent-to-Agent Protocol (A2A)
• Agentic RAG (Dynamic Retrieval)
• Multi-layer guardrails
• LangGraph over LangChain
• OpenTelemetry for agents

Context Management

• Context Engineering - `references/context-engineering.md`
• Progressive disclosure
• Session management
• Memory provenance
• Retrieval timing
• Multimodal context

Core Operational Patterns

• Operational Patterns - `references/operational-patterns.md`
• Agent loop pattern (PLAN → ACT → OBSERVE → UPDATE)
• OS agent action loop
• RAG pipeline pattern
• Tool specification
• Memory system pattern
• Multi-agent workflow
• Safety & guardrails
• Observability
• Evaluation patterns
• Deployment & CI/CD

---

Navigation: Protocol Implementation

• MCP Practical Guide - `references/mcp-practical-guide.md`

Building MCP servers, tool integration, and standardized data access

• MCP Server Builder - `references/mcp-server-builder.md`

End-to-end checklist for workflow-focused MCP servers (design → build → test)

• A2A Handoff Patterns - `references/a2a-handoff-patterns.md`

Agent-to-agent communication, task delegation, and coordination protocols

• Protocol Decision Tree - `references/protocol-decision-tree.md`

When to use MCP vs A2A, decision framework, and selection criteria

---

Navigation: Agent Capabilities

• Agent Operations - `references/agent-operations-best-practices.md`

Action loops, planning, observation, and execution patterns

• RAG Patterns - `references/rag-patterns.md`

Contextual retrieval, agentic RAG, and hybrid search strategies

• Memory Systems - `references/memory-systems.md`

Session, long-term, episodic, and task memory architectures

• Tool Design & Validation - `references/tool-design-specs.md`

Tool schemas, validation, error handling, and MCP integration

Skill Packaging & Sharing

• Skill Lifecycle - `references/skill-lifecycle.md`

Scaffold, validate, package, and share skills with teams (Slack-ready)

• API Contracts for Agents - `references/api-contracts-for-agents.md`

Request/response envelopes, safety gates, streaming/async patterns, error taxonomy

• Multi-Agent Patterns - `references/multi-agent-patterns.md`

Manager-worker, sequential, handoff, and group chat orchestration

• OS Agent Capabilities - `references/os-agent-capabilities.md`

Desktop automation, UI grounding, and computer use patterns

• Code/SWE Agents - `references/code-swe-agents.md`

SE 3.0 paradigm, autonomous coding patterns, SWE-Bench, HyperAgent architecture

Framework-Specific Patterns

• Pydantic AI Patterns - `references/pydantic-ai-patterns.md`

Type-safe agents, MCP toolsets (Stdio/SSE/StreamableHTTP), A2A via to_a2a(), pydantic-graph FSM, durable execution, TestModel testing

---

Navigation: Production Operations

• Evaluation & Observability - `references/evaluation-and-observability.md`

OpenTelemetry GenAI, metrics, LLM-as-judge, and monitoring

• Deployment, CI/CD & Safety - `references/deployment-ci-cd-and-safety.md`

Multi-layer guardrails, HITL controls, NIST AI RMF, production checklists

• Agent Debugging Patterns - `references/agent-debugging-patterns.md`

Systematic debugging for agentic systems: trace analysis, tool call failures, loop detection, state corruption

• Voice & Multimodal Agents - `references/voice-multimodal-agents.md`

Voice-first and multimodal agent patterns: speech pipelines, vision grounding, cross-modal orchestration

• Guardrails Implementation - `references/guardrails-implementation.md`

Multi-layer guardrail patterns: input/output validation, content filtering, PII detection, cost caps

---

Navigation: Templates (Copy-Paste Ready)

Checklists

• Agent Design & Safety Checklist - `assets/checklists/agent-safety-checklist.md`

Go/No-Go safety gate: permissions, HITL triggers, eval gates, observability, rollback

Core Agent Templates

• Standard Agent Template - `assets/core/agent-template-standard.md`

Full production spec: memory, tools, RAG, evaluation, observability, safety

• Specialized Agent Template - `assets/core/agent-template-specialized.md`

Domain-specific agents with custom capabilities and constraints

• Quick Agent Template - `assets/core/agent-template-quick.md`

Minimal viable agent for rapid prototyping

RAG Templates

• Basic RAG - `assets/rag/rag-basic.md`

Simple retrieval-augmented generation pipeline

• Advanced RAG - `assets/rag/rag-advanced.md`

Contextual retrieval, reranking, and agentic RAG patterns

• Hybrid Retrieval - `assets/rag/hybrid-retrieval.md`

Semantic + keyword search with BM25 fusion

Tool Templates

• Tool Definition - `assets/tools/tool-definition.md`

MCP-compatible tool schemas with validation and error handling

• Tool Validation Checklist - `assets/tools/tool-validation-checklist.md`

Testing, security, and production readiness checks

Multi-Agent Templates

• Manager-Worker Template - `assets/multi-agent/manager-worker-template.md`

Orchestration pattern with task delegation and result aggregation

• Evaluator-Router Template - `assets/multi-agent/evaluator-router-template.md`

Dynamic routing with quality assessment and domain classification

Service Layer Templates

• FastAPI Agent Service - `../dev-api-design/assets/fastapi/fastapi-complete-api.md`

Auth, pagination, validation, error handling; extend with model lifespan loads, SSE, background tasks

---

External Sources Metadata

• Curated References - `data/sources.json`

Authoritative sources spanning standards, protocols, and production agent frameworks

---

Shared Utilities (Centralized patterns — extract, don't duplicate)

• ../software-clean-code-standard/utilities/llm-utilities.md — Token counting, streaming, cost estimation
• ../software-clean-code-standard/utilities/error-handling.md — Effect Result types, correlation IDs
• ../software-clean-code-standard/utilities/resilience-utilities.md — p-retry v6, circuit breaker for API calls
• ../software-clean-code-standard/utilities/logging-utilities.md — pino v9 + OpenTelemetry integration
• ../software-clean-code-standard/utilities/observability-utilities.md — OpenTelemetry SDK, tracing, metrics
• ../software-clean-code-standard/utilities/testing-utilities.md — Test factories, fixtures, mocks
• ../software-clean-code-standard/references/clean-code-standard.md — Canonical clean code rules (CC-*) for citation

---

Trend Awareness Protocol

IMPORTANT: When users ask framework recommendations or "what's best for X" questions, use WebSearch to verify current landscape before answering. If unavailable, use data/sources.json and state what was verified vs assumed.

Trigger: framework comparisons, "best for [use case]", "is X still relevant?", "latest in AI agents", MCP server availability.

Report: current landscape, emerging trends, deprecated patterns, recommendation with rationale.

---

Related Skills

This skill integrates with complementary skills:

Core Dependencies

• `../ai-llm/` - LLM patterns, prompt engineering, and model selection for agents
• `../ai-rag/` - Deep RAG implementation: chunking, embedding, reranking
• `../ai-prompt-engineering/` - System prompt design, few-shot patterns, reasoning strategies

Production & Operations

• `../qa-observability/` - OpenTelemetry, metrics, distributed tracing
• `../software-security-appsec/` - OWASP Top 10, input validation, secure tool design
• `../ops-devops-platform/` - CI/CD pipelines, deployment strategies, infrastructure

Supporting Patterns

• `../dev-api-design/` - REST/GraphQL design for agent APIs and tool interfaces
• `../ai-mlops/` - Model deployment, monitoring, drift detection
• `../qa-debugging/` - Agent debugging, error analysis, root cause investigation
• `../dev-ai-coding-metrics/` - Team-level AI coding metrics: adoption, DORA/SPACE, ROI, DX surveys (this skill covers per-task agent economics)

Usage pattern: Start here for agent architecture, then reference specialized skills for deep implementation details.

---

Usage Notes

• Modern Standards: Default to MCP for tools, agentic RAG for retrieval, handoff-first for multi-agent
• Lightweight SKILL.md: Use this file for quick reference and navigation
• Drill-down resources: Reference detailed resources for implementation guidance
• Copy-paste templates: Use templates when the user asks for structured artifacts
• External sources: Reference data/sources.json for authoritative documentation links
• No theory: Never include theoretical explanations; only operational steps

---

AI-Native SDLC Template

• Use `assets/agent-template-ainative-sdlc.md` for the Delegate → Review → Own runbook (guardrails + outputs checklist).

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

AI-Native SDLC Agent Template

Purpose: Delegate mechanical SDLC work to the agent while humans own intent, architecture, and release. Use for feature delivery, refactors, or hotfixes.

Inputs:

• Spec or ticket
• Repo context (paths, constraints, coding standards)
• AGENTS.md / tool scopes (allowed commands, time caps, kill switch)
• Required tests and deploy checks

Preflight:

• Set max runtime and token budget; require explicit kill switch
• Allowlist commands/tools; block package installs unless approved
• Enable logging (plan, actions, diffs, test output)
• Require PLAN.md creation or planning tool output before coding

Runbook (Delegate → Review → Own)

• Plan
• Agent drafts PLAN.md with scope, code paths, dependencies, risks, and exit criteria
• Human reviews/edits plan; reject until risks/edge cases captured
• Design
• Agent maps mocks/specs to components; applies design tokens/style guides
• Call MCP component library; list accessibility gaps
• Human signs off on architecture changes or schema migrations
• Build
• Agent scaffolds end-to-end: models/APIs/UI/tests/docs in one run
• Enforce conventions (telemetry, errors, lint format, feature flags)
• Block commits/merges; diff-only output; no secrets
• Test
• Require failing test first; agent adds/updates tests and runs suite
• Capture coverage delta and flaky-test notes
• Human verifies assertions/fixtures reflect intent
• Review
• Agent performs first-pass review focused on P0/P1 bugs and policy violations
• Human reviews architecture, performance, safety, migrations; owns merge
• Document
• Agent writes PR summary, file/module notes, mermaid diagram if useful
• Human adds “why” and approvals; ensure docs ship with code
• Deploy & Maintain
• Agent links logs/metrics via MCP; proposes hotfix with rollback plan
• Human approves rollout; track evals/drift/regressions

Guardrails

• Time cap per run; abort on unexpected prompts or new permission requests
• No package install/network without approval; no credential edits
• Require explicit test run and results before proposing merge
• Always surface uncertainties and blocked items; never self-approve

Outputs Checklist

• PLAN.md (or planner output), code diffs, tests run + results, doc updates, PR summary, risk/edge list, next steps/rollout notes

Agent Design & Safety Checklist

Purpose: Ensure production-ready agent development with multi-layer safety controls and observability baseline.

---

Template Contract

Goals

• Ensure the agent is bounded, auditable, and rollbackable.
• Prevent unsafe tool actions, data leakage, and uncontrolled spend.
• Make quality and safety measurable before production rollout.

Inputs

• Agent spec (purpose, users, permissions).
• Tool inventory (APIs, data stores, side effects).
• Data classification (PII, confidential, public).
• SLOs/budgets (latency, cost per request, failure rate).

Decisions

• Autonomy level and step/time/cost caps.
• Tool allowlist + authorization model per tool.
• HITL triggers and escalation paths.
• Evaluation gates and rollout strategy (canary, shadow, rollback).

Risks

• Prompt injection and tool abuse via untrusted inputs.
• Data exfiltration via tools, logs, or citations.
• Runaway loops (cost/latency explosions) and cascading retries.
• Non-reproducible behavior due to hidden state or missing traces.

Metrics

• Task success rate, tool success rate, refusal correctness.
• Guardrail violation rate, PII leakage rate.
• Latency (TTFT/total) p50/p95/p99 and cost per request.

Pre-Development

Scope Definition

• [ ] Agent purpose documented (single responsibility)
• [ ] Tool allowlist defined (no "all tools" access)
• [ ] Maximum autonomy level specified (L1-L5)
• [ ] HITL triggers identified (financial, destructive, legal actions)

Risk Assessment

• [ ] Blast radius documented (what can go wrong)
• [ ] Data access classified (PII, confidential, public)
• [ ] Destructive actions identified (delete, modify, send)
• [ ] Regulatory constraints checked (GDPR, HIPAA, SOX, EU AI Act)

---

Implementation

Guardrails (Multi-Layer Defense Required)

Layer 1: Input Validation

• [ ] PII redaction configured
• [ ] Content filtering enabled
• [ ] Prompt injection detection active

Layer 2: Authorization

• [ ] RBAC/ABAC configured per tool
• [ ] Fine-grained permissions defined
• [ ] Principle of least privilege applied

Layer 3: Tool Gating

• [ ] Tool signatures verified (artifact signing)
• [ ] Human approval required for high-risk operations
• [ ] Rate limits per tool configured

Layer 4: Output Filtering

• [ ] PII detection in responses
• [ ] Policy compliance validation
• [ ] Content moderation active

Layer 5: Observability

• [ ] OpenTelemetry spans configured
• [ ] SIEM integration active
• [ ] Real-time alerts defined

OpenTelemetry Spans (Required)

spans:
  - llm_call: {prompt, response, tokens, latency, model}
  - tool_call: {name, params, result, duration, success}
  - retrieval: {query, chunks, scores, method}
  - memory_op: {operation, type, key, size}
  - agent_handoff: {source, target, schema_version, trace_id}

Failure Handling

• [ ] Retry policy defined (max retries, exponential backoff)
• [ ] Fallback behavior specified
• [ ] Timeout limits set (per-step and total)
• [ ] Error classification (retriable vs fatal)
• [ ] Graceful degradation path documented

---

Pre-Production

Evaluation Suite

• [ ] Golden dataset created (minimum 50 test cases)
• [ ] Final answer evaluation (correctness, grounding, clarity)
• [ ] Trajectory evaluation (step quality, tool use)
• [ ] Safety evaluation (policy violations, harmful content)
• [ ] Adversarial testing completed

Security Testing

• [ ] OWASP GenAI Top 10 checked
• [ ] Prompt injection testing passed
• [ ] Tool abuse scenarios tested
• [ ] PII leakage testing passed

Deployment Readiness

• [ ] Canary deployment configured
• [ ] Rollback procedure documented and tested
• [ ] Incident runbook created
• [ ] On-call rotation assigned

---

Production Metrics

---

Post-Launch Monitoring

Daily Checks

• [ ] Error rate within threshold
• [ ] Cost within budget
• [ ] No guardrail violations
• [ ] Latency stable

Weekly Reviews

• [ ] Evaluation score trends
• [ ] User feedback analysis
• [ ] Cost optimization opportunities
• [ ] Security incident review

Monthly Reviews

• [ ] Model performance degradation check
• [ ] Tool usage patterns analysis
• [ ] Capacity planning update
• [ ] Compliance audit

---

Anti-Patterns to Avoid

---

Sign-Off

Quick Agent Template

Purpose: Rapidly define a functional agent with minimal configuration. Suitable for prototypes, internal tooling, or simple production agents.

---

When to Use

Use this template when you need:

• A compact agent specification
• A starting point for rapid iteration
• A simple, single-agent design
• Minimal boilerplate
• Quick prototyping with tools or RAG

---

TEMPLATE STARTS HERE

1. Agent Overview

Agent Name: [Name]

Primary Goal: [Short description]

Key Behaviors:

• [Behavior 1]
• [Behavior 2]
• [Behavior 3]

Limitations:

• [Out-of-scope items]
• [Disallowed operations]

Capability Level & Policy: L[0-4] and allowed tools/scopes, approvals, and HITL gates at this level.

Contracts & Handoffs (if applicable): Schemas, trace_id, escalation rules, and any negotiation/subcontract needs.

---

2. System Instructions

You are a production agent designed to complete tasks using a plan → act → observe loop.
You must use only authorized tools.
You must ground factual statements in retrieved evidence.
You must ask for confirmation before irreversible or high-risk actions.
If you cannot perform a task, say so clearly.
Keep all responses short, structured, and operational.

---

3. Tools

3.1 Tool List

3.2 Tool Rule Summary

• Validate all parameters.
• Never hallucinate paths/IDs/fields.
• Retry only transient errors.
• Verify tool output before using it.

---

4. RAG (Optional)

Retrieval Pipeline

query → embed → retrieve → rerank → inject → answer

Injection Format

<retrieved>
[chunk_1]
[chunk_2]
</retrieved>

RAG Rules

• Use retrieval before answering fact-based questions.
• Cite retrieved evidence directly.
• Remove irrelevant chunks.

---

5. Memory (Optional)

Memory Rules

• Store only user-approved, non-sensitive, stable facts.
• Summarize session history when long.
• Retrieve memory only when relevant.
• Session design: scope/handle, sharing rules, replay limits.
• Write triggers: phase completion, confidence drop, new entity, pre-handoff.
• Provenance: source, timestamp, origin agent/tool, approvals, confidence.

---

6. Safety

High-Risk Confirmation Required For

• OS/system actions
• File modifications
• Financial or legal operations
• External system mutations

Safety Rules

• Reject unsupported or dangerous tasks.
• Sanitize all user inputs.
• Block hallucinated tools/actions.

---

7. Observability

Required Logs

• Input
• Plan
• Tool calls
• Tool outputs
• Final answer

Required Traces

• LM call
• Tool call
• Retrieval (if used)

---

8. Deployment (Minimal)

• [ ] Evaluation tests pass
• [ ] Safety checks pass
• [ ] Version pinned
• [ ] Rollback path defined

---

COMPLETE EXAMPLE (Optional)

1. Agent Overview

Agent Name: Internal Search Assistant Primary Goal: Answer internal policy questions using RAG.

3. Tools (Example)

search_policies:
  description: "Search internal policy index by query"
  input_schema:
    query: string
  output_schema:
    results: list
  confirm: no
  error_handling:
    retry: 1
    timeout: 10

4. RAG (Example)

<retrieved>
[Policy Section 3.2: VPN Requirements]
</retrieved>

---

End of Template

Specialized Agent Template

Purpose: Provide a structured template for designing specialized agents with domain-specific rules, advanced RAG, complex tool use, multi-agent roles, OS automation, or high-risk operational constraints.

---

When to Use

Use this template when:

• The agent performs complex retrieval, multiple tools, or multi-step reasoning.
• The agent operates in regulated, sensitive, or high-risk domains.
• The agent integrates with OS, browser, or external systems.
• The agent is part of a multi-agent orchestration.
• The agent requires strict safety, advanced evaluation, or custom workflows.

---

Structure

This template contains 14 specialized sections:

1. Specialization Summary 2. System Instructions (Specialized Form) 3. Operational Scope & Boundaries 4. Domain Rules & Constraints 5. Advanced Tools & Execution Rules 6. Advanced RAG (Domain-Aware) 7. Memory Strategy (Domain-Specific) 8. Multi-Agent Role Definition (If Used) 9. Planning Framework (Custom) 10. Safety Enforcement Layer 11. Validation Layer 12. Observability (Deep Mode) 13. Evaluation Framework (Domain-Specific) 14. Deployment Requirements

---

TEMPLATE STARTS HERE

1. Specialization Summary

Agent Name: [Name]

Domain: [e.g., legal QA, financial modeling, clinical data extraction, OS automation]

Primary Functions:

• [Function 1]
• [Function 2]
• [Function 3]

Special Notes:

• [High-risk constraints, compliance rules, tool limitations, etc.]

Capability Level & Policy: L[0-4]; allowed scopes/tools, approvals, HITL gates, and audit requirements for this level.

Contracts & Handoffs (if multi-agent/external): Input/output Schemas, contract version, trace_id requirements, escalation rules, negotiation/subcontract handling.

---

2. System Instructions (Specialized)

You are a specialized agent operating in the [domain] domain.
You must perform all tasks using a strict plan → act → observe → update loop.

You MUST:
- Use only approved tools.
- Ground all facts in retrieved evidence.
- Adhere to domain-specific rules and constraints.
- Ask for confirmation before high-risk actions.
- Produce structured outputs as required.

You MUST NOT:
- Invent facts, policies, legal interpretations, numbers, or system paths.
- Perform disallowed or irreversible actions without explicit confirmation.
- Use tools not listed in the Tools section.

Output Format Requirements:

• JSON
• Markdown tables
• Action blocks
• RAG evidence sections

---

3. Operational Scope & Boundaries

Allowed Tasks:

• [Explicit task types]

Out-of-Scope Tasks:

• [Must decline or redirect]

Authority Level:

• [read-only? modify? execute?]

Escalation Rules:

• [When to ask for confirmation or clarification]
• [When to escalate to human/manager agent]

---

4. Domain Rules & Constraints

Domain Standards:

• [e.g., legal citations, medical terminology, finance accuracy]

Regulatory Requirements:

• [HIPAA / GDPR / SOC2 / internal policies]

Accuracy Requirements:

• [Zero hallucination allowed?]
• [Evidence-backed answers required?]

Forbidden Behaviors:

• [Domain-specific limitations]

---

5. Advanced Tools & Execution Rules

5.1 Tool List (Specialized)

5.2 Tool Definition (Example)

tool_name:
  description: [operational purpose]
  input_schema:
    param_a: string
    param_b: integer
  output_schema:
    result: object
  confirm: [yes/no]
  error_handling:
    retry: 1
    timeout: 20

5.3 Tool Execution Rules

• Validate all parameters.
• Reject hallucinated IDs / paths / fields.
• Use high-risk confirmation logic.
• Retry only transient errors (e.g., timeouts).
• Verify output fields before plan continues.

---

6. Advanced RAG (Domain-Aware)

6.1 RAG Pipeline

query → rewrite → embed → retrieve → rerank → filter → enrich → inject → answer

6.2 Index Specifications

6.3 Evidence Injection

<retrieved>
[chunk_1]
[chunk_2]
</retrieved>

6.4 Domain Filters

• Enforce domain matching.
• Remove irrelevant or conflicting chunks.
• Require citations for all factual claims.

---

7. Memory Strategy (Domain-Specific)

Memory Types Enabled:

• Session memory
• Long-term preferences
• Episodic events
• Domain knowledge only when safe

Write Rules

Write memory only if:

• User explicitly confirms
• Information is non-sensitive
• Information is verifiable
• Information is stable over time

Write triggers: Phase completion, confidence drop, new entities, pre-handoff consolidation. Provenance: Source, timestamp, originating agent/tool, approvals, confidence. Session design: Scope/handle, sharing across agents, replay limits.

Retrieval Rules

• Retrieve only relevant entries
• Summaries required >150 tokens
• Apply domain filters

---

8. Multi-Agent Role Definition (Optional)

Example Roles

Manager: Decompose tasks. Worker-Research: Perform RAG + summaries. Worker-Execution: Execute tool operations. Evaluator: Score correctness / grounding / safety. Router: Domain routing.

Optional Multi-Agent Structure

roles:
  manager:
    responsibilities: [planning, coordination]
  worker_research:
    responsibilities: [retrieval, summarization]
  worker_execution:
    responsibilities: [tool-use, OS actions]
  evaluator:
    responsibilities: [scoring, verification]

---

9. Planning Framework (Custom)

Planning Pattern (Specialized)

1. analyze(query)
2. retrieve / collect needed context
3. produce plan (atomic steps)
4. execute each step:
      observe → ground → act → verify
5. consolidate results
6. produce final output

Plan Requirements

• Each step explicit
• Each step lists expected outputs
• No speculative steps
• Revise plan after each observation

---

10. Safety Enforcement Layer

Safety Checks

• Domain restrictions
• High-risk action detection
• Data sensitivity detection
• Tool misuse prevention
• OS-action constraints

Safety Gates

check_domain()
check_action_risk()
check_tool_scope()
sanitize_inputs()

Confirmation Prompts

Include exact parameters:

You requested a high-risk operation:
Action: [...]
Parameters: [...]
Please confirm "yes" to proceed.

---

11. Validation Layer

Validation Pattern

validate_input()
validate_tool_params()
validate_rag_chunks()
validate_memory()
validate_output()

Required Validations

• Type checking
• Range checking
• Domain consistency
• Evidence alignment

---

12. Observability (Deep Mode)

Required Logs

• Input
• Plan
• Tool calls
• Tool results
• RAG retrieval
• Memory reads/writes
• Evaluator scores
• Final answer

Required Traces

• LM spans
• Tool spans
• Retrieval spans
• Memory spans
• Safety spans

Metrics

---

13. Evaluation Framework (Domain-Specific)

Evaluation Categories

• Correctness
• Grounding
• Domain compliance
• Safety performance
• Tool execution accuracy
• OS action accuracy (if used)

LLM-as-Judge Template

{
  "correctness": 1-5,
  "grounding": 1-5,
  "domain_accuracy": 1-5,
  "tool_usage": 1-5,
  "safety": "pass|fail",
  "notes": "..."
}

---

14. Deployment Requirements

Pre-Deployment Checklist

• [ ] All evaluation tests passed
• [ ] RAG pipeline validated
• [ ] Tool-call tests validated
• [ ] Safety tests passed
• [ ] Version pinned
• [ ] Canary rollout configured
• [ ] Rollback plan ready

Deployment Flow

dev → CI → staging → canary (1%) → expand (25%) → full production

---

COMPLETE EXAMPLE (Optional)

1. Specialization Summary

Agent Name: Medical Safety Summarizer Domain: Clinical documentation (read-only) Primary Functions:

• Extract key information
• Detect unsafe statements
• Summarize with evidence

Constraints:

• No diagnosis generation
• Must cite sections

6. Advanced RAG (Example)

Injection Format

<retrieved>
[Section 2.3: Symptoms]
[Section 4.0: Contraindications]
</retrieved>

---

End of Template

Standard Agent Operations Template

Purpose: Create a full production-ready agent specification including memory, tools, RAG, evaluation, observability, safety, and deployment.

---

Related Resources

Best Practices:

• Agent Operations - Action loops, planning patterns
• Tool Design & Validation - MCP tools, schemas, error handling
• RAG Patterns - Contextual retrieval, hybrid search
• Deployment & Safety - Multi-layer guardrails, HITL

Related Skills:

• Prompt Engineering - System prompt optimization
• Observability - OpenTelemetry, metrics
• Security - Input validation, OWASP Top 10

---

When to Use

Use this template when:

• Designing a new production agent.
• Adding memory, RAG, or tools to an existing agent.
• Creating multi-agent configurations.
• Preparing for evaluation, staging, or deployment.

---

Structure

This template has 11 sections:

1. Agent Overview 2. System Instructions 3. Tools 4. Memory 5. RAG (Retrieval-Augmented Grounding) 6. Multi-Agent Configuration (Optional) 7. Safety & Guardrails 8. Observability 9. Evaluation 10. Deployment 11. OS Agent Integration (If Applicable)

---

TEMPLATE STARTS HERE

1. Agent Overview

Agent Name: [Name]

Primary Goal: [What the agent must achieve]

Key Behaviors:

• [Behavior 1]
• [Behavior 2]
• [Behavior 3]

Constraints:

• [Safety restrictions]
• [Budget / token caps]
• [Disallowed actions]

Capability Level & Policy: L[0-4] (static → tool → strategic → multi-agent → self-evolving); allowed tools, scopes, approvals, and HITL gates at this level.

Contracts & Handoffs (if multi-agent or external): Input/output JSON Schemas, contract version, trace_id requirements, escalation rules, negotiation/subcontract needs.

---

2. System Instructions

Core Behavior:

You are a production agent.
You must follow a plan → act → observe loop for every step.
You may only use tools you are authorized to use.
Ground all factual statements in retrieved evidence when available.
Ask for confirmation before performing irreversible actions.
Keep answers concise and operational.

Style Requirements:

• [tone, brevity, formatting rules]
• [structured outputs: JSON / markdown / tables]

---

3. Tools

3.1 Available Tools

3.2 Tool Definitions

tool_name:
  description: [clear operational purpose]
  input_schema:
    field_1: string
    field_2: number
  output_schema:
    result: string
  confirm: yes/no
  error_handling:
    retry: 1
    timeout: 30

3.3 Tool Use Rules

• Validate all parameters before calling.
• Never hallucinate IDs, paths, or coordinates.
• Use tools when external data or action is required.
• Do not chain tools without verifying each result.

---

4. Memory

4.1 Memory Types Used

• Session memory: [yes/no]
• Long-term memory: [yes/no]
• Episodic memory: [yes/no]
• Task-specific scratchpad: [yes/no]

Session design: Scope/handle, sharing rules across agents, replay limits.

4.2 Memory Write Rules

Write memory only when:

• User explicitly confirms.
• Fact is verifiable and non-sensitive.
• Fact will be reused later.
• Provenance (source, timestamp) can be stored.

Write triggers: Phase completion, confidence drop, new entity detected, pre-handoff consolidation. Provenance fields: Source, timestamp, tool/agent of origin, approvals, confidence.

4.3 Memory Retrieval Rules

• Retrieve only relevant memories.
• Summarize if > 200 tokens.
• Apply recency filters when appropriate.

---

5. RAG (Retrieval-Augmented Grounding)

5.1 Retrieval Pipeline

query → rewrite → embed → retrieve → rerank → filter → inject → answer

5.2 Indexes Used

5.3 RAG Injection Format

<retrieved>
[chunk_1]
[chunk_2]
...
</retrieved>

5.4 RAG Rules

• Always rerank retrieved results.
• Discard irrelevant or stale chunks.
• All factual claims should be traceable to chunks.

---

6. Multi-Agent Configuration (Optional)

Pattern: Manager / Worker / Router / Evaluator

---

7. Safety & Guardrails

7.1 Input Safety Filters

• Block prompt injection attempts.
• Block unsupported domains.
• Normalize and sanitize inputs.

7.2 High-Risk Actions

Require explicit confirmation for:

• Financial or legal actions.
• OS-level commands.
• File deletion or modification.
• External system writes.

7.3 Output Safety

• Must avoid disallowed content.
• Must not expose secrets or PII.
• Must refuse unsupported dangerous requests.

---

8. Observability

8.1 Required Logs

• User input (sanitized).
• Agent plan.
• Tool calls (name + parameters).
• Tool results (status + output).
• RAG retrieval details.
• Final answer.

8.2 Required Traces

One span per:

• LM call
• Tool call
• Retrieval step
• Memory read/write

8.3 Metrics

---

9. Evaluation

9.1 Evaluation Dimensions

• Effectiveness: task success, correctness.
• Grounding: evidence-backed answers.
• Tool Use: correct tool selection and parameters.
• Safety: refusal and safe-handling correctness.
• Performance: latency and cost.

9.2 Test Cases

9.3 LLM-as-Judge Template

Evaluate the agent output on:
- Correctness (1–5)
- Grounding (1–5)
- Tool usage (1–5)
- Safety (pass/fail)

Return JSON:
{
  "correctness": n,
  "grounding": n,
  "tool_usage": n,
  "safety": "pass|fail",
  "justification": "short explanation"
}

---

10. Deployment

10.1 Pre-Deployment Checklist

• [ ] All evaluation tests passed.
• [ ] Tool success rate ≥ threshold.
• [ ] Safety tests passed.
• [ ] Logging and tracing enabled.
• [ ] Version pinned (models, prompts, tools).
• [ ] Rollback strategy defined.

10.2 Promotion Flow

dev → CI eval → staging → canary → production

---

11. OS Agent Integration (If Applicable)

11.1 OS Action Loop

OBSERVE(window_state)
GROUND(element)
ACT(click/type/scroll/shortcut)
VERIFY(state_change)

11.2 OS Action Safety

• Avoid blind coordinate clicking.
• Always verify element visibility.
• Require confirmation for destructive OS operations.

---

COMPLETE EXAMPLE

1. Agent Overview (Example)

Agent Name: Docs Support Agent Primary Goal: Answer questions about internal documentation with grounded, cited responses.

Key Behaviors:

• Retrieve relevant documents via RAG.
• Cite all answers from retrieved content.
• Refuse questions outside allowed domains.

Constraints:

• Cannot access external internet.
• Must not invent policy or legal statements.

---

2. System Instructions (Example)

You are a production Docs Support Agent.
You answer questions using only internal documentation passed via <retrieved> tags.
If the answer is not present, you say you don't know.
Cite specific sections or filenames when answering.
Never fabricate policies, legal clauses, or user data.
Ask before performing any irreversible action.

---

3. Tools (Example)

search_docs:
  description: "Search internal docs index by query."
  input_schema:
    query: string
    limit: integer
  output_schema:
    results: list
  confirm: no
  error_handling:
    retry: 1
    timeout: 10

---

5. RAG (Example)

Indexes Used

Injection

<retrieved>
[chunk_1]
[chunk_2]
</retrieved>

---

10. Deployment (Example)

Pre-Deployment Checks

• [x] 50 test questions passed.
• [x] Grounding ≥ 4.5 average.
• [x] Tool success rate ≥ 97%.
• [x] Safety: pass on all red-team prompts.

---

Quality Checklist (Before Finalizing Spec)

• [ ] All sections 1–11 filled.
• [ ] Tools defined with schemas and safety rules.
• [ ] Memory rules declared and safe.
• [ ] RAG pipeline fully specified.
• [ ] Evaluation metrics and thresholds set.
• [ ] Deployment and rollback clearly defined.
• [ ] OS integration defined (if relevant).

---

End of Template

Knowledge Base Architecture — Unified Agent Memory

Purpose: Architecture template for building a unified Knowledge Base that combines vector store, knowledge graph, and document index with provenance tracking. This is the persistent semantic memory layer for agent systems.

---

1. Unified KB Schema

Pattern: Three-Store Architecture

┌─────────────────────────────────────────────────┐
│                 QUERY INTERFACE                   │
│  semantic search | entity lookup | keyword filter │
├────────────┬──────────────┬─────────────────────┤
│  VECTOR    │  KNOWLEDGE   │   DOCUMENT          │
│  STORE     │  GRAPH       │   INDEX             │
│  (embeddings│ (entities,  │  (full-text,        │
│   + cosine) │  relations) │   filters, facets)  │
├────────────┴──────────────┴─────────────────────┤
│              PROVENANCE LAYER                    │
│  source | timestamp | confidence | lineage       │
├─────────────────────────────────────────────────┤
│              STORAGE ENGINE                      │
│  (provider-specific: Pinecone, Neo4j, ES, etc.) │
└─────────────────────────────────────────────────┘

Schema Definition

knowledge_base:
  # Layer 1: Vector Store — semantic similarity search
  vector_store:
    provider: "pinecone | qdrant | pgvector | chroma | weaviate"
    config:
      embedding_model: "text-embedding-3-large"
      dimensions: 3072
      distance_metric: "cosine"  # cosine | euclidean | dot_product
      index_type: "hnsw"
      namespace_strategy: "per_source"  # per_source | per_domain | single
    record_schema:
      id: "string (deterministic hash of content + source)"
      embedding: "float[3072]"
      text: "string (original chunk text)"
      metadata:
        source_url: "string"
        source_type: "api | document | web | database"
        domain: "string"
        ingested_at: "ISO 8601"
        chunk_index: "int"
        parent_doc_id: "string"

  # Layer 2: Knowledge Graph — entity relationships
  knowledge_graph:
    provider: "neo4j | falkordb | amazon_neptune | memgraph"
    config:
      persistence: "disk"
      consistency: "eventual"  # strong | eventual
    schema:
      entity_types:
        - "person"
        - "organization"
        - "concept"
        - "document"
        - "event"
        - "tool"
        - "metric"
      relation_types:
        - "authored_by"
        - "belongs_to"
        - "references"
        - "contradicts"
        - "supersedes"
        - "depends_on"
        - "measured_by"
      entity_properties:
        - name: "string"
        - type: "enum (entity_types)"
        - source: "string"
        - confidence: "float"
        - created_at: "ISO 8601"
        - updated_at: "ISO 8601"

  # Layer 3: Document Index — keyword search + filtering
  document_index:
    provider: "elasticsearch | typesense | meilisearch | opensearch"
    config:
      analyzers: ["standard", "keyword"]
      shards: 1
      replicas: 0
    fields:
      - name: "title"
        type: "text"
        searchable: true
      - name: "content"
        type: "text"
        searchable: true
      - name: "source_url"
        type: "keyword"
        filterable: true
      - name: "domain"
        type: "keyword"
        filterable: true
      - name: "ingested_at"
        type: "date"
        sortable: true
      - name: "tags"
        type: "keyword[]"
        filterable: true

---

2. Provenance Tracking

Pattern: Every Record Has Lineage

provenance_record:
  lineage_id: "string (uuid — traces full lifecycle)"
  source:
    url: "string (where the data came from)"
    type: "api | document | web | database | user_input | inference"
    fetch_method: "crawl | webhook | poll | upload | A2A"
  timestamps:
    source_created_at: "ISO 8601 (when source published)"
    ingested_at: "ISO 8601 (when we fetched it)"
    last_validated_at: "ISO 8601 (when we last checked freshness)"
    expires_at: "ISO 8601 (TTL expiration)"
  quality:
    confidence: "float (0.0 - 1.0)"
    validation_method: "checksum | schema_match | llm_verify | human_review"
    error_rate: "float (historical accuracy of this source)"
  lineage:
    parent_doc_id: "string (if chunked from larger doc)"
    transformation: "string (chunked | summarized | translated | extracted)"
    pipeline_version: "string (which pipeline version produced this)"

Checklist: Provenance Requirements

• [ ] Every record has a lineage_id that traces back to its origin.
• [ ] source.url is populated — never store data without knowing where it came from.
• [ ] ingested_at is set at write time — never backdate.
• [ ] confidence is set based on source type (user_input=1.0, inference=0.5-0.8).
• [ ] expires_at is set based on freshness policy (see Section 3).
• [ ] transformation records what happened to the data (chunked, summarized, etc.).

---

3. Freshness Management

Pattern: TTL + Invalidation + Re-Index

freshness_policy:
  # Default TTL by source type
  ttl_by_source:
    api_data: 3600           # 1 hour
    web_page: 86400          # 24 hours
    document: 604800         # 7 days
    user_input: 2592000      # 30 days
    reference_data: 7776000  # 90 days

  # Invalidation triggers (immediate re-fetch)
  invalidation_triggers:
    - webhook_received          # source pushes update
    - schema_change_detected    # source structure changed
    - confidence_below: 0.3     # quality degraded
    - contradiction_detected    # conflicting data found
    - user_reported_stale       # user flags outdated info

  # Re-indexing strategy
  re_index:
    strategy: "incremental"     # full | incremental | differential
    schedule: "0 2 * * *"       # daily at 2 AM
    priority_sources_first: true
    max_concurrent_fetches: 10
    backoff_on_failure:
      base_ms: 5000
      max_ms: 300000
      max_retries: 3

Freshness Check Pattern

Before serving a KB result:
1. Check expires_at against current time
2. If expired:
   a. Return stale result with staleness warning
   b. Trigger background re-fetch
   c. Mark record as "stale_pending_refresh"
3. If not expired:
   a. Return result normally
4. After re-fetch:
   a. Compare new content hash with stored hash
   b. If changed: update record, bump ingested_at, recalculate embedding
   c. If unchanged: bump last_validated_at only

Decision Tree: When to Invalidate

What triggered the check?
├── Webhook received? → Invalidate immediately, re-fetch
├── Scheduled re-index? → Check content hash, update if changed
├── Query returned low confidence? → Flag for review, don't invalidate
├── Contradiction detected? → Invalidate both records, fetch fresh
└── User reported stale? → Invalidate, re-fetch, log user feedback

---

4. Access Control and Multi-Tenant Patterns

Pattern: Namespace Isolation

multi_tenant:
  isolation_strategy: "namespace"  # namespace | separate_index | row_level
  namespace_key: "tenant_id"

  # Vector store: separate namespace per tenant
  vector_store_namespaces:
    tenant_a: "ns-tenant-a"
    tenant_b: "ns-tenant-b"
    shared: "ns-shared"        # shared knowledge (docs, policies)

  # Knowledge graph: label-based isolation
  knowledge_graph_labels:
    tenant_a: "TenantA"
    tenant_b: "TenantB"
    shared: "Shared"

  # Document index: filter-based isolation
  document_index_filter:
    field: "tenant_id"
    enforce_on_every_query: true

Access Control Matrix

Checklist: Multi-Tenant Safety

• [ ] Every query includes tenant_id filter — never return cross-tenant data by accident.
• [ ] Shared namespace is read-only for tenant-scoped agents.
• [ ] Data Agent writes enforce tenant_id on every record.
• [ ] Audit log tracks all cross-tenant queries.
• [ ] PII is encrypted at rest and tenant-scoped encryption keys are isolated.

---

5. Integration with Data Agent

Pattern: Data Agent → KB Write Pipeline

Data Agent output → KB write path:

1. RECEIVE transformed data from Data Agent
2. VALIDATE against KB schema (reject malformed)
3. EMBED text fields using configured model
4. CHECK for existing record (same source + content hash)
   ├── New record → INSERT across all three stores
   └── Updated record → UPSERT (vector + doc index), UPDATE (graph)
5. SET provenance metadata (lineage_id, ingested_at, confidence)
6. CONFIRM write success, return record IDs

Write Consistency

write_transaction:
  strategy: "best_effort_ordered"  # not ACID across stores
  order: ["vector_store", "document_index", "knowledge_graph"]
  on_partial_failure:
    rollback_completed_writes: true
    retry_failed_store: true
    max_retries: 2
    alert_on_inconsistency: true

---

6. Access Protocol: MCP

Agents access the Knowledge Base through Model Context Protocol (MCP) — the standard interface for agent-to-data connectivity (the "USB-C for AI").

Pattern: MCP-First KB Access

kb_mcp_server:
  name: "knowledge-base"
  transport: "stdio | sse | streamable-http"
  tools:
    - name: "kb_semantic_search"
      description: "Search KB by semantic similarity"
      input_schema:
        query: "string"
        top_k: "int (default: 10)"
        namespace: "string (optional, for multi-tenant)"
        filters: "object (optional, date/source/domain)"
      output: "array of {text, score, provenance}"

    - name: "kb_entity_lookup"
      description: "Look up entity and relationships in knowledge graph"
      input_schema:
        entity: "string (name or ID)"
        max_hops: "int (default: 2)"
      output: "entity node + edges + neighbor nodes"

    - name: "kb_keyword_search"
      description: "Keyword search with filters and facets"
      input_schema:
        terms: "string"
        filters: "object (date, source, tags)"
      output: "array of {title, content_snippet, highlights, provenance}"

    - name: "kb_hybrid_search"
      description: "Combined semantic + keyword + entity enrichment"
      input_schema:
        query: "string"
        filters: "object (optional)"
      output: "array of {text, score, entities, provenance}"

Why MCP over direct DB access: Agents should never connect directly to vector stores or graph databases. MCP provides tool-level abstraction with schema validation, rate limiting, access control, and audit logging — all enforced at the protocol layer rather than relying on each agent to implement correctly.

Pluggable Driver Architecture

Use a database-agnostic core with swappable backend drivers. This prevents vendor lock-in and enables per-environment configuration (e.g., Chroma for dev, Pinecone for production).

driver_abstraction:
  interface_operations:
    - "upsert_record"
    - "delete_record"
    - "search_semantic"
    - "search_keyword"
    - "get_entity"
    - "traverse_graph"
    - "batch_write"
  drivers:
    pinecone:
      vector_store: true
      config: { api_key: "${PINECONE_API_KEY}", index: "kb-prod" }
    neo4j:
      knowledge_graph: true
      config: { uri: "${NEO4J_URI}", auth: "${NEO4J_AUTH}" }
    typesense:
      document_index: true
      config: { host: "${TYPESENSE_HOST}", api_key: "${TYPESENSE_API_KEY}" }

Reference: Graphiti implements this pattern with 11 operation abstractions across Neo4j, FalkorDB, Kuzu, and Neptune drivers.

---

7. Query Patterns

Pattern: Unified Query Interface

query_interface:
  # Semantic search (vector store)
  semantic:
    input: "natural language query"
    method: "embed_query → cosine_similarity → top_k"
    returns: "ranked documents with scores"

  # Entity lookup (knowledge graph)
  entity:
    input: "entity_id or entity_name + type"
    method: "graph traversal (BFS, max 2 hops)"
    returns: "entity + relationships + neighbors"

  # Keyword/filter (document index)
  keyword:
    input: "search terms + filters (date, source, tags)"
    method: "full-text search + faceted filter"
    returns: "matching documents with highlights"

  # Hybrid (all three)
  hybrid:
    input: "natural language + optional filters"
    method: |
      1. Semantic search → top 20
      2. Keyword search → top 20
      3. Entity enrichment → add related entities
      4. Reciprocal rank fusion → merged top 10
    returns: "enriched results with provenance"

Decision Tree: Which Query?

What does the agent need?
├── "Find similar content" → Semantic search
├── "What is entity X?" → Entity lookup
├── "All docs matching [filter]" → Keyword/filter
├── "Answer question about X" → Hybrid (semantic + entity enrichment)
└── "Cross-reference X and Y" → Entity lookup → Semantic on results

---

Implementation Checklist

Phase 1: Single Store (MVP)

• [ ] Choose primary vector store (Pinecone, Qdrant, or pgvector).
• [ ] Define record schema with provenance fields.
• [ ] Implement embed → upsert → query pipeline.
• [ ] Add TTL-based freshness checks.
• [ ] Connect Data Agent write path.

Phase 2: Add Document Index

• [ ] Add Typesense/Meilisearch for keyword search.
• [ ] Implement hybrid query (semantic + keyword with RRF).
• [ ] Add filter/facet support (source, date, domain).
• [ ] Sync document index with vector store on writes.

Phase 3: Add Knowledge Graph

• [ ] Add Neo4j/FalkorDB for entity relationships.
• [ ] Implement entity extraction on ingest (NER or LLM).
• [ ] Build graph-augmented retrieval pipeline.
• [ ] Add contradiction detection across stores.

Phase 4: Production Hardening

• [ ] Implement multi-tenant namespace isolation.
• [ ] Add write consistency with ordered rollback.
• [ ] Deploy freshness management (TTL + invalidation + re-index).
• [ ] Add OpenTelemetry metrics (query latency, index size, freshness).
• [ ] Load test with 10× expected query volume.

---

Commercial Reference Implementations (March 2026)

Products that validate and extend our three-store KB architecture:

Memory Layer Platforms

Maps to our architecture: Mem0 covers our Knowledge Base + Context Graph layers with a single-API opinionated approach. Redis is a strong implementation choice for our mid-scale KB tier (Section 1 provider options).

Enterprise KB Platforms

Pattern validated: Tabnine's vector + graph + agentic retrieval confirms our three-store architecture (vector store + knowledge graph + document index) as the production pattern for enterprise KB.

Vector/Search Infrastructure

Key Industry Patterns

1. MCP as KB access protocol — Pinecone, Confluent, and others ship native MCP servers. Our MCP-First KB Access pattern (Section 6) is aligned with industry direction. 2. Driver abstraction — Graphiti's 11-operation abstraction across Neo4j, FalkorDB, Kuzu, and Neptune validates our Pluggable Driver Architecture (Section 6). 3. Freshness as non-negotiable — Materialize identifies three context engine requirements: freshness (current reality, not snapshots), correctness (no partial/stale state), composability (derived views stack without gaps). Our TTL + invalidation + re-index pattern (Section 3) addresses all three.

---

Related Resources

Evaluator + Router Multi-Agent Template

Purpose: Define production-grade Evaluator and Router agents used in multi-agent systems for domain routing, scoring, quality control, grounding, and safety enforcement with validated handoffs.

Modern Update: All handoffs between agents must use validated JSON Schema payloads with trace_id propagation.

---

When to Use

Use this template when:

• You need deterministic domain routing across multiple worker agents
• You need an Evaluator to score worker outputs
• You must enforce quality, grounding, and safety before integration
• Multiple workers require domain specialization
• You want a modular routing layer for future expansion
• You need versioned handoff contracts for reliability

---

TEMPLATE STARTS HERE

1. Multi-Agent Overview

System Name: [Name]

Roles Included:

• Router Agent — selects appropriate worker agent
• Evaluator Agent — scores worker outputs
• Optional: Manager + Workers (covered in other template)

---

2. Router Agent Template

2.1 Router Role

The Router classifies the user query or subtask, assigns it to the correct Worker, and returns the routing decision to the Manager.

Router does not:

• Execute tasks
• Use tools
• Modify subtasks
• Perform planning

Router only:

• Classifies
• Routes
• Validates domain
• Rejects ambiguous mappings

---

2.2 Router System Instructions

You are the Router agent.
Your job is to:
1. Analyze each subtask or query.
2. Classify it into the correct domain.
3. Select the appropriate worker.
4. Request clarification when classification is uncertain.
5. Never execute tasks or call tools.

Output ONLY routing decisions in structured format.

---

2.3 Routing Table

routing_table:
  code:
    keywords: ["function", "compile", "stack trace", "error", "API"]
    worker: worker_code
  research:
    keywords: ["summarize", "explain", "compare", "analyze"]
    worker: worker_research
  operations:
    keywords: ["create ticket", "schedule", "inventory", "order"]
    worker: worker_ops
  rag:
    keywords: ["retrieve", "find", "search", "documents"]
    worker: worker_rag

---

2.4 Routing Logic Template (Modern Handoff Pattern)

Validated handoff payload:

handoff_to_worker:
  # Handoff metadata (modern standard)
  version: "v1.2"
  trace_id: "req-abc-123"  # Propagated from original request
  timestamp: "2025-11-18T10:30:00Z"
  source_agent: "router-001"
  target_agent: "[assigned_worker]"

  # Routing decision
  domain: "[classified_domain]"
  worker: "[assigned_worker]"
  confidence: [0.0-1.0]

  # Task definition
  task:
    id: "task-456"
    type: "[domain]"
    instruction: "[original user query or subtask]"
    expected_output: "Structured result with citations"
    constraints:
      max_duration_seconds: 300
      require_citations: true

  # Context
  context:
    user_query: "Original user question"
    prior_findings: []
    domain_specific_data: {}

  # Validation
  validation:
    schema_version: "v1.2"
    required_fields: ["task.instruction", "trace_id", "worker"]
    checksum: "sha256-hash"

Validation checklist before handoff:

• [ ] JSON Schema validation passed
• [ ] trace_id propagated
• [ ] All required fields present
• [ ] Confidence ≥ threshold (0.65)
• [ ] Worker exists in routing table
• [ ] Task constraints defined

---

2.5 Routing Decision Rules

• Minimum confidence threshold: ≥ 0.65
• If below threshold → ask user for clarification
• If multiple domains match → request clarification
• If domain unknown → fallback to general worker or manager

---

3. Evaluator Agent Template

3.1 Evaluator Role

Evaluator scores Worker outputs along multiple dimensions:

• Correctness
• Grounding
• Completeness
• Structure
• Safety

Evaluator does NOT:

• Modify outputs
• Execute tasks
• Perform planning
• Generate new content

---

3.2 Evaluator System Instructions

You are the Evaluator agent.
Your job is to:
1. Score worker outputs for correctness, grounding, structure, and safety.
2. Reject unsafe or incorrect outputs.
3. Request worker redo when scores fall below threshold.
4. Output structured scores only.

---

3.3 Evaluation Scoring Template

evaluation:
  task_id: "task-001"
  correctness: 1-5
  grounding: 1-5
  completeness: 1-5
  structure: 1-5
  safety: "pass" | "fail"
  notes: "short explanation only"

---

3.4 Evaluation Thresholds

If any score < threshold → worker redo required.

---

3.5 Evaluation Rules

• Evidence must align 1:1 with worker output
• Citations must match retrieved text
• No hallucinations allowed
• No contradictions
• No safety red flags
• No unsupported claims
• All required fields must be present

---

3.6 Safety Scan Pattern

scan_for:
  - hallucinated actions/tools
  - unsupported domain instructions
  - sensitive or private data
  - high-risk unconfirmed actions

If detected → safety: fail.

---

4. End-to-End Router + Evaluator Flow

Manager → Router → Worker → Evaluator → Manager

Steps

1. Manager creates subtask 2. Router classifies → selects worker 3. Worker executes subtask 4. Evaluator scores output 5. Manager integrates or requests redo

---

5. Validation Checklists

Router Validation Checklist

• [ ] Domain classification correct
• [ ] Worker selected from routing table
• [ ] Confidence ≥ threshold
• [ ] No ambiguous domains
• [ ] Request clarification on domain conflict

Evaluator Validation Checklist

• [ ] All score fields present
• [ ] Safety scanned
• [ ] Grounding validated
• [ ] Output structure correct
• [ ] Score thresholds enforced

---

6. Anti-Patterns

Router Anti-Patterns

• AVOID: Routing without confidence threshold
• AVOID: Assigning to multiple workers
• AVOID: Hallucinating unknown domains
• AVOID: Acting like a Worker

Evaluator Anti-Patterns

• AVOID: Changing Worker outputs
• AVOID: Executing tasks
• AVOID: Ignoring missing fields
• AVOID: Accepting unsafe outputs
• AVOID: Soft-failing without rejecting

---

7. Complete Example (Optional)

Router Output Example

router_output:
  domain: "code"
  worker: "worker_code"
  confidence: 0.82

Evaluator Output Example

evaluation:
  task_id: "t003"
  correctness: 5
  grounding: 4
  completeness: 4
  structure: 4
  safety: "pass"
  notes: "Output correctly grounded in provided logs."

---

End of Template

Manager–Worker Multi-Agent Template

Purpose: Provide a production-grade template for building a multi-agent system where a Manager agent delegates tasks to Worker agents and integrates their outputs.

---

Related Resources

Best Practices:

• Multi-Agent Patterns - Orchestration patterns and coordination
• A2A Handoff Patterns - Agent-to-agent communication protocol
• Evaluation & Observability - Multi-agent tracing and metrics

Protocol Guides:

• Protocol Decision Tree - MCP vs A2A selection
• MCP Practical Guide - Tool integration for workers

Related Skills:

• LLM Engineering - Model selection per agent role
• Observability - Distributed tracing across agents

---

When to Use

Use this template when:

• Tasks must be decomposed into atomic subtasks
• Workers require specialized tools or domain expertise
• The system needs separation of planning vs execution
• Output must be validated, scored, and integrated
• Multi-agent orchestration is required

---

TEMPLATE STARTS HERE

1. Multi-Agent Overview

System Name: [Name]

Architecture: Manager → Worker(s) → Evaluator (optional) → Manager → Final output

Goals:

• Decompose complex tasks
• Route to correct worker
• Execute subtasks deterministically
• Validate and integrate results

Agents Involved:

• Manager
• Worker(s): [worker_1, worker_2, …]
• Evaluator (optional)

---

2. Manager Agent Specification

2.1 Role

The Manager plans, decomposes, orchestrates, and integrates. It never executes tasks or calls tools.

2.2 System Instructions

You are the Manager agent.
Your job is to:
1. Understand the user query.
2. Decompose it into atomic subtasks.
3. Assign each subtask to the correct Worker.
4. Validate Worker outputs.
5. Integrate results into a final answer.
6. Replan when Worker outputs contradict expectations.

Do NOT execute tasks or call tools.
You only plan, delegate, validate, and integrate.

2.3 Subtask Format

subtask:
  id: "task-001"
  description: "[what must be done]"
  expected_output: "[format or fields]"
  worker: "[assigned_worker]"

2.4 Manager Delegation Rules

• Decompose into logical, minimal steps
• Assign each step to one worker only
• Include expected output schema
• Revise plan if Worker output is invalid

---

3. Worker Agent Specification

3.1 Role

Workers execute subtasks, using tools, RAG, OS actions, or domain logic. Workers do not break down tasks or create new tasks.

3.2 System Instructions

You are a Worker agent.
Your job is to execute exactly the subtask assigned to you.

You MUST:
- Use tools appropriately.
- Perform retrieval if required.
- Output structured results.
- Stay within the subtask scope.

You MUST NOT:
- Modify or create subtasks.
- Delegate work.
- Perform Manager duties.

3.3 Output Format

worker_output:
  id: "task-001"
  output: {...}
  evidence: [...]
  confidence: 0.0-1.0

3.4 Worker Execution Pattern

plan_step()
if retrieval needed: run RAG
if tools needed: validate → execute → verify
format output
return to Manager

---

4. Optional: Evaluator Agent Specification

4.1 Role

Evaluator scores Worker outputs for:

• Correctness
• Grounding
• Safety
• Structure

4.2 Scoring Template

evaluation:
  task_id: "task-001"
  correctness: 1-5
  grounding: 1-5
  safety: "pass|fail"
  notes: "..."

4.3 Evaluator Rules

• Reject unsafe or incorrect outputs
• Request Worker redo if score < threshold

---

5. End-to-End Flow

User Request
→ Manager decomposes
→ Router (optional) routes subtasks
→ Workers execute
→ Evaluator scores (optional)
→ Manager integrates
→ Final Answer

---

6. Integration Logic (Manager)

6.1 Manager Integration Pattern

collect(worker_outputs)
validate_all()
resolve_conflicts()
merge_into_final_answer()

6.2 Conflict Resolution Rules

• Prefer higher evaluator score
• Prefer more recent or direct evidence
• Discard outputs that contradict retrieved evidence

6.3 Final Output Format

final_answer:
  summary: "..."
  combined_results: [...]
  evidence: [...]

---

7. Safety Rules for Multi-Agent Systems

• Manager must confirm high-risk actions
• Workers must not bypass confirmation logic
• Evaluator must run safety scan when enabled
• No Worker can act outside its assigned scope
• No agent stores sensitive data

---

8. Multi-Agent Validation Checklist

Manager

• [ ] Subtasks atomic
• [ ] Correct worker chosen
• [ ] Expected output defined

Workers

• [ ] Tools validated pre-call
• [ ] Evidence included
• [ ] Output structured

Evaluator (optional)

• [ ] Scored each output
• [ ] Flagged unsafe items
• [ ] Requested redo where needed

System

• [ ] Conflicts resolved
• [ ] Final output grounded
• [ ] No hallucinated subtasks/workers

---

COMPLETE EXAMPLE (Optional)

Example Decomposition (Manager)

subtask_1:
  id: "t001"
  description: "Retrieve uptime metrics for service X."
  expected_output: "JSON with metrics + timestamps"
  worker: "worker_metrics"

subtask_2:
  id: "t002"
  description: "Summarize and highlight anomalies."
  expected_output: "Markdown summary + anomalies list"
  worker: "worker_analysis"

Example Worker Output

worker_output:
  id: "t001"
  output:
    uptime_percent: 99.2
    outages: ["2024-01-03 03:21 UTC"]
  evidence:
    - "logs/service_x.log: lines 42–55"
  confidence: 0.94

Example Final Answer

final_answer:
  summary: "Service X shows strong uptime with one minor outage."
  combined_results:
    - uptime: 99.2
    - outage_events: ["2024-01-03"]
  evidence:
    - "logs/service_x.log"

---

End of Template

Hybrid Retrieval Template

Purpose: Provide a structured template for implementing hybrid retrieval (semantic + keyword) with reranking, domain filtering, metadata scoring, and conflict resolution.

---

When to Use

Use this template when:

• Retrieval requires both semantic similarity and exact matching.
• Data includes technical, legal, financial, or code-heavy content.
• Users ask for factual, numeric, or terminology-sensitive answers.
• You need higher precision than semantic-only retrieval.

---

TEMPLATE STARTS HERE

1. Hybrid Retrieval Overview

Goal: [Describe what the hybrid retrieval solves or enhances.]

Sources / Indexes:

• [Semantic index]
• [Keyword index]
• [Optional rule-based index]
• [Optional metadata index]

Constraints:

• All chunks must be relevant.
• Keyword hits have priority for factual accuracy.
• Semantic matches fill conceptual context.

---

2. Retrieval Pipeline (Hybrid)

query
→ optional_rewrite
→ embed
→ semantic_retrieve(top_k_semantic)
→ keyword_retrieve(top_k_keyword)
→ merge_and_dedupe
→ rerank
→ filter
→ inject
→ answer

---

3. Parameters

Semantic Retrieval

• top_k_semantic: [20–50]
• Embedding model: [model_name]

Keyword Retrieval

• top_k_keyword: [10–20]
• Engine: [BM25 / keyword index]

Reranking

• Model: [cross-encoder / domain reranker]
• Keep: top 3–7

Filtering

• Domain check
• Term match
• Conflict resolution
• Metadata validation

---

4. Merge & Deduplication

Rule Set

• Deduplicate by chunk hash or paragraph ID.
• Prefer keyword hits for exact terms.
• Merge based on semantic similarity threshold (e.g., 0.8).

Example Pseudocode

results = semantic_results + keyword_results
results = dedupe(results)
results = rerank(results)

---

5. Filtering (Hybrid-Specific)

Required Filters

• Domain alignment
• Relevance threshold
• Term consistency
• Metadata validity

Domain Enforcement Example

if chunk.domain != expected_domain:
    discard

Term Check Example

if query contains exact_term:
    ensure keyword_result contains exact_term

---

6. Evidence Injection

Injection Format

<retrieved>
[chunk_1]
[chunk_2]
[chunk_3]
</retrieved>

Injection Rules

• Max tokens: 500–700
• Keep only highly relevant chunks
• Include metadata: source, page, section, timestamp

---

7. Answer Generation Rules

• Use only injected evidence.
• Cite chunk metadata.
• Avoid mixing external world knowledge.
• If evidence contradicts: surface conflict explicitly.

Answer Template

## Answer
[Short grounded answer]

## Evidence
- [chunk_1_source]
- [chunk_2_source]

---

8. Validation

Checklist

• [ ] Query rewritten (if needed)
• [ ] Semantic retrieval executed
• [ ] Keyword retrieval executed
• [ ] Results merged correctly
• [ ] Reranking applied
• [ ] All chunks relevant
• [ ] No duplicates
• [ ] Evidence injected properly
• [ ] Final answer grounded

Anti-Patterns

• AVOID: Using semantic-only for fact-heavy queries
• AVOID: Injecting irrelevant keyword hits
• AVOID: Overweighting semantic similarity
• AVOID: Using >700 tokens as context
• AVOID: Responding without citations
• AVOID: Mixing multi-domain results

---

9. Complete Example (Optional)

Pipeline Summary

Rewrite: enabled
Semantic top_k: 30
Keyword top_k: 15
Rerank: cross-encoder-finance
Final Chunks: 3

Injection Example

<retrieved>
[Annual Report 2023 - Section 4.2: Revenue Breakdown]
[Annual Report 2023 - Section 4.3: Cost of Goods Sold]
[Annual Report 2023 - Appendix A: Terminology]
</retrieved>

Answer Example

Revenue increased due to higher unit sales and expanded distribution channels (see Section 4.2).

---

End of Template

Advanced RAG Template

Purpose: Provide a full production-grade template for complex Retrieval-Augmented Generation, including routing, HyDE, multi-index retrieval, hierarchical search, enrichment, and advanced filtering.

---

When to Use

Use this template when:

• Retrieval spans multiple domains or indexes
• Queries are complex, ambiguous, or sparse
• Strict grounding and accuracy are required
• You need hierarchical, hybrid, or enriched retrieval
• You must enforce domain-based filtering
• You require structured outputs

---

TEMPLATE STARTS HERE

1. RAG Overview (Advanced)

Goal: [Describe precise retrieval goals]

Sources / Indexes:

Core Requirements:

• Multi-step retrieval
• Multi-domain routing
• Heavy reranking
• Strict evidence-only generation
• No hallucinations

---

2. Advanced RAG Pipeline

query
→ detect_domain
→ route_to_index
→ rewrite (optional)
→ embed
→ retrieve (semantic + keyword)
→ rerank
→ hierarchical_refine
→ context_enrich
→ filter (domain + relevance)
→ inject
→ answer

---

3. Domain Detection & Routing

3.1 Classification Pattern

domain = classify(query)
index = route(domain)

3.2 Routing Table

3.3 Routing Rules

• Reject multi-domain queries → request clarification
• Use fallback index only if domain = “unknown”

---

4. Query Rewrite (Advanced)

Rewrite Logic

• Expand acronyms
• Add domain-specific terminology
• Convert vague queries → explicit format
• Split multi-intent queries into subqueries

Rewrite Template

rewrite(query) → domain-specific, explicit, unambiguous query.

---

5. Embedding & Retrieval

5.1 Embedding

• Use consistent embedding model
• Use deterministic pre-processing

5.2 Hybrid Retrieval

semantic_top_k = [20–50]
keyword_top_k = [10–20]
combine → dedupe → rerank

5.3 Retrieval Rules

• Never rely on raw top-k
• Deduplicate before reranking
• Resolve conflicts using domain priority

---

6. HyDE (Hypothetical Document Embedding)

When to Use

• Sparse queries
• Retrieval fails or low hit rate
• Queries with abstract terms

Pattern

hyde_doc = generate_hypothetical_doc(query)
embed(hyde_doc)
retrieve_using_hyde()

HyDE Rules

• hyde_doc ≤ 150 tokens
• Must reflect domain constraints
• Must not include fabricated details

---

7. Hierarchical Retrieval

Pattern

retrieve(topic-level)
→ retrieve(section-level)
→ retrieve(paragraph-level)

Rules

• Use hierarchical steps only when needed
• Limit final extraction to 3–7 chunks
• Collapse similar content into summaries

---

8. Context Enrichment

Pattern

add(metadata)
add(linked_entities)
add_relevant_history()

Use Cases

• Entity-based tasks
• Multi-turn workflows
• Cross-document synthesis

Allowed Metadata

• IDs
• Dates
• Sections
• Entity names
• Structured fields

---

9. Filtering (Advanced)

Filters

• Domain match
• Relevance threshold
• Recency filter (if applicable)
• Deduplication
• Conflict resolution

Conflict Resolution Rules

• Prefer more recent content
• Prefer domain-specific over generic
• Prefer higher reranker score

---

10. Evidence Injection

Injection Format

<retrieved>
[chunk_1]
[chunk_2]
[chunk_3]
...
</retrieved>

Requirements

• Max injected length: 500–700 tokens
• Chunks must be topic-pure
• Include metadata (source, page, hash)

---

11. Answer Generation (Strict)

Answer Rules

• Use ONLY injected evidence
• Cite exact chunks
• No external world knowledge
• No hallucinated claims
• When evidence is missing → “insufficient data”

Answer Format

## Answer
[Short grounded answer]

## Evidence
- [chunk_1_source]
- [chunk_2_source]

---

12. Validation Pipeline

Checklist

• [ ] Domain classified accurately
• [ ] Routed to correct index
• [ ] Query rewritten properly
• [ ] Hybrid retrieval used
• [ ] Reranking applied
• [ ] HyDE applied (if needed)
• [ ] Hierarchical retrieval validated
• [ ] Enrichment consistent
• [ ] All chunks relevant
• [ ] No duplicates
• [ ] Answer grounded & cited

---

13. Anti-Patterns (Advanced)

• AVOID: Skipping reranking
• AVOID: Injecting > 700 tokens
• AVOID: Mixing domain-chunks
• AVOID: Summaries with fabricated content
• AVOID: Relying solely on semantic search
• AVOID: Using HyDE without domain consistency
• AVOID: Answering from memory instead of evidence
• AVOID: Citation mismatch

---

14. Complete Example (Optional)

Example Retrieval Summary

Domain: Legal
Index: legal_idx
Rewrite: "Summarize the obligations in Section 12 of Contract A."
Hybrid Retrieval: semantic_k=30, keyword_k=10
Rerank: cross-encoder-legal
Final Chunks: 3

Example Injection

<retrieved>
[Contract A - Section 12: Obligations]
[Contract A - Section 12.1: Deliverables]
[Contract A - Section 12.3: Compliance Requirements]
</retrieved>

Example Answer

Section 12 requires the vendor to deliver the agreed-upon services, comply with listed requirements, and maintain proper documentation (see evidence above).

---

End of Template

Basic RAG Template

Purpose: Provide a minimal, production-ready template for implementing a simple Retrieval-Augmented Generation pipeline.

---

When to Use

Use this template when:

• You need a lightweight RAG pipeline.
• Retrieval requirements are simple.
• You want a fast, minimal baseline.
• Advanced features (HyDE, routing, enrichment) are not required.

---

TEMPLATE STARTS HERE

1. RAG Overview

RAG Purpose: [Describe what the RAG pipeline retrieves and why.]

Sources / Indexes:

• [Index name 1]
• [Index name 2]

Constraints:

• Only use retrieved evidence for factual claims.
• No external internet unless explicitly allowed.

---

2. RAG Rules (Minimal)

• Always retrieve before answering fact-based questions.
• Never answer without evidence.
• Remove irrelevant or duplicate chunks.
• Keep injected evidence ≤ 500 tokens.
• Use reranking on retrieved results.

---

3. Retrieval Pipeline

query → rewrite (if needed) → embed → retrieve → rerank → filter → inject → answer

3.1 Query Rewrite (Optional)

rewrite(query) → improved_query

3.2 Embedding

• Use embedding model: [model_name]
• Use vector store: [db_name]

3.3 Retrieval

Parameters:

• Top-k retrieved: [5–20]

3.4 Reranking

• Apply cross-encoder or reranker model.
• Keep top 3–7 chunks.

3.5 Filtering Rules

• Remove chunks with low relevance.
• Remove stale/conflicting content.
• Remove duplicates.

---

4. Evidence Injection

Injection Format

<retrieved>
[chunk_1]
[chunk_2]
...
</retrieved>

Chunk Requirements

• 150–350 tokens each.
• Single-topic per chunk.
• Include metadata (source, page).

---

5. Answer Generation

Answer Rules

• Use only retrieved evidence.
• Cite chunks directly.
• No hallucinated facts allowed.
• No claims outside of injected context.

Example Format

### Answer

[Short, grounded answer here.]

### Evidence

- [chunk_1_source]
- [chunk_2_source]

---

6. Validation

Checklist

• [ ] Query rewritten (if ambiguous).
• [ ] Embeddings computed with correct model.
• [ ] Retrieval executed with correct k.
• [ ] Reranking applied.
• [ ] All chunks relevant.
• [ ] Evidence injected before reasoning.
• [ ] Final answer grounded in retrieved text.

Anti-Patterns

• AVOID: Answering without retrieval
• AVOID: Ignoring reranking
• AVOID: Using more than 500–700 tokens of context
• AVOID: Mixing unsupported external knowledge
• AVOID: Summaries not aligned with evidence

---

COMPLETE EXAMPLE (Optional)

Retrieval Pipeline (Example)

Top-k: 10 Rerank to: 3

<retrieved>
[Policy 4.1: Password Requirements]
[Policy 7.2: MFA Procedures]
</retrieved>

Answer (Example)

Your password must meet all requirements listed in Section 4.1, including minimum length and rotation (see evidence above).

---

End of Template

Tool Definition Template

Purpose: Define a production-ready tool with clear schema, safety rules, validation, and error-handling.

---

When to Use

Use this template when:

• Creating a new tool for an agent
• Connecting MCP or API functions
• Designing OS actions, retrieval tools, or system integrations
• Adding high-risk or domain-specific tools
• Upgrading tool schemas for production readiness

---

TEMPLATE STARTS HERE

1. Tool Overview

Tool Name: [tool_name]

Purpose (1 sentence): [What this tool does operationally]

Tool Category:

• Retrieval
• Action
• OS Automation
• API Integration
• Computation
• Transformation
• Other

---

2. Tool Specification (Full YAML)

tool_name:
  description: "[Clear operational purpose]"
  input_schema:
    field_1:
      type: string
      required: true
    field_2:
      type: integer
      required: false
    field_3:
      type: object
      required: false
  output_schema:
    result:
      type: object
  confirm: [yes|no]
  error_handling:
    retry: 1
    timeout: 30
    fatal_errors:
      - "auth_failure"
      - "invalid_parameters"

---

3. Input Parameter Rules

3.1 Validation Requirements

Each input must be validated for:

• Presence
• Type
• Format
• Range (if numeric)
• Allowed values (if enum)
• Safety constraints
• Domain constraints

3.2 Validation Template

validation:
  - field: field_1
    checks:
      - non_empty
      - type_string
  - field: field_2
    checks:
      - type_integer
      - range: [0, 100]
  - field: field_3
    checks:
      - type_object
      - required_fields: [subfield_a, subfield_b]

---

4. Tool Execution

4.1 Execution Pattern

validate_parameters()
apply_safety_checks()
call_tool_function()
verify_output()

4.2 Execution Rules

• Never guess parameters
• Reject hallucinated IDs, paths, or coordinates
• Require explicit values for high-risk fields
• Use blocking confirmation if confirm=yes
• Validate output strictly against schema

---

5. Output Schema Rules

Requirements

• Deterministic structure
• All fields defined
• No unexpected fields
• No null/undefined unless allowed

Output Validation Template

output_validation:
  required_fields:
    - result
  type_checks:
    result: object

---

6. Error Handling

6.1 Typed Error Policy

6.2 Error Response Template

error:
  type: [transient|soft|fatal]
  message: "..."
  details: {...}

---

7. Safety Requirements

High-Risk Tool Flags

• confirm: yes
• User must approve parameters
• Natural language safety summary required
• Abort if confirmation unclear

Safety Summary Template

You are requesting a high-risk action:
- Action: [tool_name]
- Parameters: [...]
Please confirm "yes" to proceed.

---

8. Tool Metadata (Optional)

metadata:
  owner: "team_name"
  version: "1.0.0"
  last_updated: "YYYY-MM-DD"
  changelog: "Initial release"

---

COMPLETE EXAMPLE (Generic)

set_user_permissions:
  description: "Update a user's permission level in the internal system."
  input_schema:
    user_id:
      type: string
      required: true
    new_role:
      type: string
      required: true
    reason:
      type: string
      required: false
  output_schema:
    result:
      type: object
  confirm: yes
  error_handling:
    retry: 0
    timeout: 15
    fatal_errors:
      - "auth_failure"
      - "role_not_allowed"

---

End of Template

Tool Validation Checklist

Purpose: Provide a complete, production-grade validation checklist for safe, correct, and deterministic tool use. Apply before and after any tool call.

---

When to Use

Use this checklist when:

• Creating a new tool
• Calling an existing tool
• Reviewing agent/tool behavior
• Hardening tool safety
• Debugging tool failures
• Enforcing MCP or API tool correctness

---

TEMPLATE STARTS HERE

PRE-FLIGHT VALIDATION (Before Tool Call)

1. Tool Name Validation

• [ ] Tool name matches exactly as defined
• [ ] Tool exists in available tool registry
• [ ] No hallucinated or inferred tool names

---

2. Intent → Tool Mapping

• [ ] Step requires external data OR external action
• [ ] Tool selected intentionally for the step
• [ ] Not using tool when internal reasoning suffices
• [ ] Tool chosen is the least-privileged valid option

---

3. Input Schema Validation

For each field:

• [ ] Field present if required
• [ ] Field not present if disallowed
• [ ] Type matches schema (string/int/bool/object/list)
• [ ] Format valid (e.g., email/URL/path/date)
• [ ] Numeric values within allowed range
• [ ] Enum fields match allowed values
• [ ] No guessed IDs, paths, or coordinates
• [ ] No unvalidated user free text flowing into critical fields

---

4. High-Risk Action Check

If tool is high-risk:

• [ ] Confirmation required
• [ ] Natural-language safety summary generated
• [ ] User responded with explicit “yes”
• [ ] Abort if confirmation unclear

High-Risk Categories:

• OS-level actions
• File modifications
• External system writes
• Financial/legal actions
• Irreversible operations

---

5. Safety Scan (Pre-Call)

• [ ] Input sanitized
• [ ] No prompt injection attempts
• [ ] No disallowed domain requests
• [ ] No personal or sensitive data
• [ ] No unsafe parameter combinations

---

6. Context & Dependency Validation

• [ ] Step logically follows previous steps
• [ ] Required context retrieved or prepared
• [ ] No stale values reused
• [ ] No unresolved conflicts in previous steps

---

RUNTIME VALIDATION (During Tool Call)

7. Call Execution Rules

• [ ] Tool called with validated parameters
• [ ] Retry only transient errors
• [ ] Timeout respected
• [ ] Fatal errors surfaced immediately
• [ ] All actions logged

---

POST-FLIGHT VALIDATION (After Tool Call)

8. Output Schema Validation

• [ ] Output present
• [ ] All required fields present
• [ ] Types match schema
• [ ] No unexpected fields
• [ ] No null or undefined values (unless allowed)

---

9. Output Integrity Checks

• [ ] Output grounded (not hallucinated)
• [ ] Results plausible for the domain
• [ ] No missing data that the tool guarantees
• [ ] No security violations in output
• [ ] No leaking sensitive data

---

10. Error Handling Review

If error occurred:

• [ ] Classify as transient / soft / fatal
• [ ] Retry only transient
• [ ] Request clarification only for soft
• [ ] Halt for fatal
• [ ] Produce human-readable error summary

---

11. Plan Continuation Check

• [ ] Step achieved intended effect
• [ ] Observation updated after tool call
• [ ] Next plan step depends on validated outputs
• [ ] If tool output contradicts expectations → replan

---

COMPLETE EXAMPLE (Optional)

Tool Call

tool_name: "lookup_customer"
params:
  id: "C842"

Validation Result

• Tool exists: yes
• Input valid: yes
• High-risk: no
• Safety scan: clean
• Output fields: valid
• Continue to next step: allowed

---

End of Checklist

API Contracts for Agents

Use these envelopes when exposing agents/LLMs via REST/gRPC/GraphQL.

Request Envelope

• trace_id (propagate) + request_id
• actor: user/org ids, roles/scopes, auth method
• intent: task description, system instructions
• context_refs: doc ids, vector keys, cache keys
• tools_allowed: ids + args schema; per-request allowlist
• safety: moderation level, PII policy, jailbreak guard on/off
• delivery: stream (SSE/WebSocket), async (202 + polling), callback URL + HMAC
• params: temperature, top_p, max_tokens, stop, seed

Response Envelope

• choices[]: message, role, finish_reason
• stream_delta: partial tokens/chunks when streaming
• citations[]: source_id, span, url
• tool_calls[]: name, args, status, result (if inline), latency_ms
• usage: prompt_tokens, completion_tokens, cost
• trace_id echoed; rate_limit: limit/remaining/reset

Errors (RFC 7807)

• Types: model_timeout, tool_failed, guardrail_blocked, retrieval_miss, validation_error, quota_exceeded
• Include trace_id, hint, retryable

Streaming

• SSE fields: event=delta|done|error, id, data (JSON lines)
• WebSocket: close codes documented; heartbeat/ping interval; backpressure guidance
• Keep-alives for idle connections; clear retry/backoff policy

Long-Running Jobs

• 202 Accepted + Location for status; payload includes job_id, state, eta, expires_at
• States: queued → running → succeeded | failed | cancelled
• Callbacks: signed (HMAC), replay-protected, include trace_id

Safety & Guardrails

• Pre: moderation, injection scan, scope/role checks, tool allowlist enforcement
• During: block high-risk tool calls unless approved; cap batch sizes, TTLs
• Post: PII redaction, policy filters, optional hallucination/citation checks

Observability

• Propagate traceparent/tracestate or trace_id header end-to-end
• Spans: llm_call, retrieval, tool_call, memory_op
• Logs: request envelope sans secrets, guardrail outcomes, rate-limit decisions

MCP Server Builder — Tooling for Agents

Use this when designing or implementing MCP servers for agent tools (Python or TypeScript SDKs).

Planning Checklist

• Design for workflows, not raw endpoints; consolidate related actions (e.g., schedule_event that checks availability + creates event).
• Optimize for limited context: concise defaults, optional detail flags, human-readable identifiers.
• Make errors actionable: suggest next steps and correct parameters.
• Group tools with clear prefixes and natural task names.

Research Before Coding

• Read MCP protocol spec (modelcontextprotocol.io/llms-full.txt).
• Load SDK docs (Python or TypeScript) and any target API docs (auth, rate limits, pagination, schemas).
• Define tool list, shared helpers (pagination, errors, formatting), and truncation strategy.

Implementation Patterns

• Validate inputs (Pydantic v2 or Zod .strict()); avoid any.
• Async I/O; explicit return schemas; support concise vs. detailed responses.
• Annotations: readOnlyHint, destructiveHint, idempotentHint, openWorldHint where appropriate.
• Centralize API helpers, auth, error handling, and pagination.

Review & Testing

• Create evaluation scenarios early; iterate based on agent feedback.
• Check character limits and truncation; handle rate limits and timeouts gracefully.
• Document tool usage, parameters, and error responses inside the server code.

Skill Lifecycle — Create, Validate, Share

Use this when packaging Claude skills for reuse and team distribution.

Create

• Run the skill init script (if available) to scaffold SKILL.md, scripts/, references/, assets/ with kebab-case naming and matching frontmatter.
• Write SKILL.md in imperative style; keep it lean and link to resources for depth.

Validate

• Ensure frontmatter name matches directory, and description is specific and activation-friendly.
• Check structure: required SKILL.md; optional references/, scripts/, assets/.
• Run validation tooling if present; fix any missing metadata or naming issues.

Package & Share

• Package as a zip (validation first); include all referenced files.
• Post summary to Slack via automation (Rube/Slack integration): name, description, link, and key resources.
• Keep versions discoverable; update team channels when new skills land or change materially.