Senior Prompt Engineer

Senior Prompt Engineer

Prompt engineering patterns, LLM evaluation frameworks, and agentic system design. Provides static (deterministic) analysis tools to optimize prompts, evaluate RAG retrieval and generation quality, and validate/visualize agent workflows — plus deep reference libraries of prompt patterns, evaluation metrics, and agent architectures.

Core Capabilities

• Prompt optimization — token counting and cost estimation, clarity/structure scoring, ambiguity and redundancy detection, and generation of optimized prompt versions.
• Few-shot & structured output design — extract/manage few-shot examples, design diverse example sets (simple/edge/complex/negative), and enforce reliable JSON/XML schema outputs.
• RAG evaluation — context relevance, answer faithfulness, groundedness (ROUGE-L), and retrieval metrics (Precision@K, MRR, NDCG) over pre-retrieved contexts.
• Agentic system design — validate agent configs, visualize flows (ASCII/Mermaid), estimate token cost per run, and apply ReAct / Plan-Execute / Tool-Use / multi-agent patterns.
• Pattern library — 10 prompt patterns, evaluation frameworks (A/B testing, benchmarks, human eval), and agent architectures with pseudocode.

When to Use

• Optimizing an existing prompt's performance or reducing token costs.
• Designing prompt templates, few-shot examples, or structured-output workflows.
• Evaluating LLM outputs or RAG retrieval/generation quality.
• Building or validating agentic systems and tool-calling workflows.

Tools

References

Load the reference that matches the task — keep this file lean and pull detail on demand:

• [references/tools-and-workflows.md](references/tools-and-workflows.md) — full tool usage with sample outputs, the prompt-optimization / few-shot / structured-output workflows, common-patterns and command quick references, troubleshooting table, success criteria, and complete per-script parameter/output-format reference. Read when running any tool or executing a workflow.
• [references/prompt_engineering_patterns.md](references/prompt_engineering_patterns.md) — 10 prompt patterns (zero/few-shot, CoT, role, structured output, self-consistency, ReAct, tree-of-thoughts, RAG) with example inputs and expected outputs. Read when choosing or applying a prompt technique.
• [references/llm_evaluation_frameworks.md](references/llm_evaluation_frameworks.md) — evaluation metrics, text-generation and RAG-specific scoring, human-eval frameworks, A/B testing, benchmark datasets, and pipeline design. Read when measuring quality or comparing prompts.
• [references/agentic_system_design.md](references/agentic_system_design.md) — agent architectures (ReAct, Plan-and-Execute, Tool Use, multi-agent, memory/state) and design patterns with pseudocode. Read when building agents or tool-calling systems.

Scope & Limitations

This skill covers:

• Static prompt analysis: token counting, clarity scoring, structure detection, and optimization suggestions
• RAG evaluation: context relevance, answer faithfulness, groundedness, and retrieval metrics (Precision@K, ROUGE-L, MRR, NDCG)
• Agent workflow design: configuration validation, ASCII/Mermaid visualization, and token cost estimation
• Few-shot example extraction and management from existing prompts

This skill does NOT cover:

• Live LLM calls or runtime prompt testing --- all analysis is static/deterministic (see senior-ml-engineer for LLM integration)
• Vector database setup or embedding generation --- RAG evaluator scores pre-retrieved contexts only (see senior-data-engineer for pipeline orchestration)
• Fine-tuning, RLHF, or model training workflows (see senior-ml-engineer for model deployment)
• Production monitoring, A/B test execution, or real-time drift detection (see senior-data-scientist for experiment design)

Integration Points

Agentic System Design

Agent architectures, tool use patterns, and multi-agent orchestration with pseudocode.

Architectures Index

1. ReAct Pattern 2. Plan-and-Execute 3. Tool Use / Function Calling 4. Multi-Agent Collaboration 5. Memory and State Management 6. Agent Design Patterns

---

1. ReAct Pattern

Reasoning + Acting: The agent alternates between thinking about what to do and taking actions.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        ReAct Loop                           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐ │
│   │ Thought │───▶│ Action  │───▶│  Tool   │───▶│Observat.│ │
│   └─────────┘    └─────────┘    └─────────┘    └────┬────┘ │
│        ▲                                            │       │
│        └────────────────────────────────────────────┘       │
│                         (loop until done)                   │
└─────────────────────────────────────────────────────────────┘

Pseudocode

def react_agent(query, tools, max_iterations=10):
    """
    ReAct agent implementation.

    Args:
        query: User question
        tools: Dict of available tools {name: function}
        max_iterations: Safety limit
    """
    context = f"Question: {query}\n"

    for i in range(max_iterations):
        # Generate thought and action
        response = llm.generate(
            REACT_PROMPT.format(
                tools=format_tools(tools),
                context=context
            )
        )

        # Parse response
        thought = extract_thought(response)
        action = extract_action(response)

        context += f"Thought: {thought}\n"

        # Check for final answer
        if action.name == "finish":
            return action.argument

        # Execute tool
        if action.name in tools:
            observation = tools[action.name](action.argument)
            context += f"Action: {action.name}({action.argument})\n"
            context += f"Observation: {observation}\n"
        else:
            context += f"Error: Unknown tool {action.name}\n"

    return "Max iterations reached"

Prompt Template

You are a helpful assistant that can use tools to answer questions.

Available tools:
{tools}

Answer format:
Thought: [your reasoning about what to do next]
Action: [tool_name(argument)] OR finish(final_answer)

{context}

Continue:

When to Use

---

2. Plan-and-Execute

Two-phase approach: First create a plan, then execute each step.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                     Plan-and-Execute                         │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Phase 1: Planning                                           │
│  ┌──────────┐    ┌──────────────────────────────────────┐   │
│  │  Query   │───▶│  Generate step-by-step plan          │   │
│  └──────────┘    └──────────────────────────────────────┘   │
│                              │                               │
│                              ▼                               │
│                  ┌──────────────────────┐                    │
│                  │ Plan: [S1, S2, S3]   │                    │
│                  └──────────┬───────────┘                    │
│                             │                                │
│  Phase 2: Execution         │                                │
│                  ┌──────────▼───────────┐                    │
│                  │   Execute Step 1     │                    │
│                  └──────────┬───────────┘                    │
│                             │                                │
│                  ┌──────────▼───────────┐                    │
│                  │   Execute Step 2     │──▶ Replan?         │
│                  └──────────┬───────────┘                    │
│                             │                                │
│                  ┌──────────▼───────────┐                    │
│                  │   Execute Step 3     │                    │
│                  └──────────┬───────────┘                    │
│                             │                                │
│                  ┌──────────▼───────────┐                    │
│                  │    Final Answer      │                    │
│                  └──────────────────────┘                    │
└──────────────────────────────────────────────────────────────┘

Pseudocode

def plan_and_execute(query, tools):
    """
    Plan-and-Execute agent.

    Separates planning from execution for complex tasks.
    """
    # Phase 1: Generate plan
    plan = generate_plan(query)

    results = []

    # Phase 2: Execute each step
    for i, step in enumerate(plan.steps):
        # Execute step
        result = execute_step(step, tools, results)
        results.append(result)

        # Optional: Check if replanning needed
        if should_replan(step, result, plan):
            remaining_steps = plan.steps[i+1:]
            new_plan = replan(query, results, remaining_steps)
            plan.steps = plan.steps[:i+1] + new_plan.steps

    # Synthesize final answer
    return synthesize_answer(query, results)


def generate_plan(query):
    """Generate execution plan from query."""
    prompt = f"""
    Create a step-by-step plan to answer this question:
    {query}

    Format each step as:
    Step N: [action description]

    Keep the plan concise (3-7 steps).
    """
    response = llm.generate(prompt)
    return parse_plan(response)


def execute_step(step, tools, previous_results):
    """Execute a single step using available tools."""
    prompt = f"""
    Execute this step: {step.description}

    Previous results:
    {format_results(previous_results)}

    Available tools: {format_tools(tools)}

    Provide the result of this step.
    """
    return llm.generate(prompt)

When to Use

---

3. Tool Use / Function Calling

Structured tool invocation: LLM generates structured calls that are executed externally.

Tool Definition Schema

{
  "name": "search_web",
  "description": "Search the web for current information",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "Search query"
      },
      "num_results": {
        "type": "integer",
        "default": 5,
        "description": "Number of results to return"
      }
    },
    "required": ["query"]
  }
}

Implementation Pattern

class ToolRegistry:
    """Registry for agent tools."""

    def __init__(self):
        self.tools = {}

    def register(self, name, func, schema):
        """Register a tool with its schema."""
        self.tools[name] = {
            "function": func,
            "schema": schema
        }

    def get_schemas(self):
        """Get all tool schemas for LLM."""
        return [t["schema"] for t in self.tools.values()]

    def execute(self, name, arguments):
        """Execute a tool by name."""
        if name not in self.tools:
            raise ValueError(f"Unknown tool: {name}")

        func = self.tools[name]["function"]
        return func(**arguments)


def tool_use_agent(query, registry):
    """Agent with function calling."""
    messages = [{"role": "user", "content": query}]

    while True:
        # Call LLM with tools
        response = llm.chat(
            messages=messages,
            tools=registry.get_schemas(),
            tool_choice="auto"
        )

        # Check if done
        if response.finish_reason == "stop":
            return response.content

        # Execute tool calls
        if response.tool_calls:
            for call in response.tool_calls:
                result = registry.execute(
                    call.function.name,
                    json.loads(call.function.arguments)
                )
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": str(result)
                })

Tool Design Best Practices

---

4. Multi-Agent Collaboration

Orchestration Patterns

Pattern 1: Sequential Pipeline

Agent A → Agent B → Agent C → Output

Use case: Research → Analysis → Writing

Pattern 2: Hierarchical

        ┌─────────────┐
        │ Coordinator │
        └──────┬──────┘
    ┌──────────┼──────────┐
    ▼          ▼          ▼
┌───────┐ ┌───────┐ ┌───────┐
│Agent A│ │Agent B│ │Agent C│
└───────┘ └───────┘ └───────┘

Use case: Complex task decomposition

Pattern 3: Debate/Consensus

┌───────┐     ┌───────┐
│Agent A│◄───▶│Agent B│
└───┬───┘     └───┬───┘
    │             │
    └──────┬──────┘
           ▼
    ┌─────────────┐
    │   Arbiter   │
    └─────────────┘

Use case: Critical decisions, fact-checking

Pseudocode: Hierarchical Multi-Agent

class CoordinatorAgent:
    """Coordinates multiple specialized agents."""

    def __init__(self, agents):
        self.agents = agents  # Dict[str, Agent]

    def process(self, query):
        # Decompose task
        subtasks = self.decompose(query)

        # Assign to agents
        results = {}
        for subtask in subtasks:
            agent_name = self.select_agent(subtask)
            result = self.agents[agent_name].execute(subtask)
            results[subtask.id] = result

        # Synthesize
        return self.synthesize(query, results)

    def decompose(self, query):
        """Break query into subtasks."""
        prompt = f"""
        Break this task into subtasks for specialized agents:

        Task: {query}

        Available agents:
        - researcher: Gathers information
        - analyst: Analyzes data
        - writer: Produces content

        Format:
        1. [agent]: [subtask description]
        """
        response = llm.generate(prompt)
        return parse_subtasks(response)

    def select_agent(self, subtask):
        """Select best agent for subtask."""
        return subtask.assigned_agent

    def synthesize(self, query, results):
        """Combine agent results into final answer."""
        prompt = f"""
        Combine these results to answer: {query}

        Results:
        {format_results(results)}

        Provide a coherent final answer.
        """
        return llm.generate(prompt)

Communication Protocols

---

5. Memory and State Management

Memory Types

┌─────────────────────────────────────────────────────────────┐
│                    Agent Memory System                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐  ┌─────────────────┐                  │
│  │ Working Memory  │  │  Episodic Memory │                  │
│  │ (Current task)  │  │ (Past sessions)  │                  │
│  └────────┬────────┘  └────────┬─────────┘                  │
│           │                    │                            │
│           └────────┬───────────┘                            │
│                    ▼                                        │
│  ┌─────────────────────────────────────────┐               │
│  │           Semantic Memory               │               │
│  │    (Long-term knowledge, embeddings)    │               │
│  └─────────────────────────────────────────┘               │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Implementation

class AgentMemory:
    """Memory system for conversational agents."""

    def __init__(self, embedding_model, vector_store):
        self.embedding_model = embedding_model
        self.vector_store = vector_store
        self.working_memory = []  # Current conversation
        self.buffer_size = 10     # Recent messages to keep

    def add_message(self, role, content):
        """Add message to working memory."""
        self.working_memory.append({
            "role": role,
            "content": content,
            "timestamp": datetime.now()
        })

        # Trim if too long
        if len(self.working_memory) > self.buffer_size:
            # Summarize old messages before removing
            old_messages = self.working_memory[:5]
            summary = self.summarize(old_messages)
            self.store_long_term(summary)
            self.working_memory = self.working_memory[5:]

    def store_long_term(self, content):
        """Store in semantic memory (vector store)."""
        embedding = self.embedding_model.embed(content)
        self.vector_store.add(
            embedding=embedding,
            metadata={"content": content, "type": "summary"}
        )

    def retrieve_relevant(self, query, k=5):
        """Retrieve relevant memories for context."""
        query_embedding = self.embedding_model.embed(query)
        results = self.vector_store.search(query_embedding, k=k)
        return [r.metadata["content"] for r in results]

    def get_context(self, query):
        """Build context for LLM from memories."""
        relevant = self.retrieve_relevant(query)
        recent = self.working_memory[-self.buffer_size:]

        return {
            "relevant_memories": relevant,
            "recent_conversation": recent
        }

    def summarize(self, messages):
        """Summarize messages for long-term storage."""
        content = "\n".join([
            f"{m['role']}: {m['content']}"
            for m in messages
        ])
        prompt = f"Summarize this conversation:\n{content}"
        return llm.generate(prompt)

State Persistence Patterns

---

6. Agent Design Patterns

Pattern: Reflection

Agent reviews and critiques its own output.

def reflective_agent(query, tools):
    """Agent that reflects on its answers."""
    # Initial response
    response = react_agent(query, tools)

    # Reflection
    critique = llm.generate(f"""
    Review this answer for:
    1. Accuracy - Is the information correct?
    2. Completeness - Does it fully answer the question?
    3. Clarity - Is it easy to understand?

    Question: {query}
    Answer: {response}

    Critique:
    """)

    # Check if revision needed
    if needs_revision(critique):
        revised = llm.generate(f"""
        Improve this answer based on the critique:

        Original: {response}
        Critique: {critique}

        Improved answer:
        """)
        return revised

    return response

Pattern: Self-Ask

Break complex questions into simpler sub-questions.

def self_ask_agent(query, tools):
    """Agent that asks itself follow-up questions."""
    context = []

    while True:
        prompt = f"""
        Question: {query}

        Previous Q&A:
        {format_qa(context)}

        Do you need to ask a follow-up question to answer this?
        If yes: "Follow-up: [question]"
        If no: "Final Answer: [answer]"
        """

        response = llm.generate(prompt)

        if response.startswith("Final Answer:"):
            return response.replace("Final Answer:", "").strip()

        # Answer follow-up question
        follow_up = response.replace("Follow-up:", "").strip()
        answer = simple_qa(follow_up, tools)
        context.append({"q": follow_up, "a": answer})

Pattern: Expert Routing

Route queries to specialized sub-agents.

class ExpertRouter:
    """Routes queries to expert agents."""

    def __init__(self):
        self.experts = {
            "code": CodeAgent(),
            "math": MathAgent(),
            "research": ResearchAgent(),
            "general": GeneralAgent()
        }

    def route(self, query):
        """Determine best expert for query."""
        prompt = f"""
        Classify this query into one category:
        - code: Programming questions
        - math: Mathematical calculations
        - research: Fact-finding, current events
        - general: Everything else

        Query: {query}
        Category:
        """
        category = llm.generate(prompt).strip().lower()
        return self.experts.get(category, self.experts["general"])

    def process(self, query):
        expert = self.route(query)
        return expert.execute(query)

---

Quick Reference: Pattern Selection

LLM Evaluation Frameworks

Concrete metrics, scoring methods, comparison tables, and A/B testing frameworks.

Frameworks Index

1. Evaluation Metrics Overview 2. Text Generation Metrics 3. RAG-Specific Metrics 4. Human Evaluation Frameworks 5. A/B Testing for Prompts 6. Benchmark Datasets 7. Evaluation Pipeline Design

---

1. Evaluation Metrics Overview

Metric Categories

Choosing the Right Metric

Is there a single correct answer?
├── Yes → Exact Match or F1
└── No
    └── Is there a reference output?
        ├── Yes → BLEU, ROUGE, or BERTScore
        └── No
            └── Can you define quality criteria?
                ├── Yes → Human evaluation + LLM-as-judge
                └── No → A/B testing with user metrics

---

2. Text Generation Metrics

BLEU (Bilingual Evaluation Understudy)

What it measures: N-gram overlap between generated and reference text.

Score range: 0 to 1 (higher is better)

Calculation:

BLEU = BP × exp(Σ wn × log(pn))

Where:
- BP = brevity penalty (penalizes short outputs)
- pn = precision of n-grams
- wn = weight (typically 0.25 for BLEU-4)

Interpretation:

Example:

Reference: "The quick brown fox jumps over the lazy dog"
Generated: "A fast brown fox leaps over the lazy dog"

1-gram precision: 7/9 = 0.78 (matched: brown, fox, over, the, lazy, dog)
2-gram precision: 4/8 = 0.50 (matched: brown fox, the lazy, lazy dog)
BLEU-4: ~0.35

Limitations:

• Doesn't capture meaning (synonyms penalized)
• Position-independent
• Requires reference text

---

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

What it measures: Overlap focused on recall (coverage of reference).

Variants:

Calculation:

ROUGE-N Recall = (matching n-grams) / (n-grams in reference)
ROUGE-N Precision = (matching n-grams) / (n-grams in generated)
ROUGE-N F1 = 2 × (Precision × Recall) / (Precision + Recall)

Example:

Reference: "The cat sat on the mat"
Generated: "The cat was sitting on the mat"

ROUGE-1:
  Recall: 5/6 = 0.83 (matched: the, cat, on, the, mat)
  Precision: 5/7 = 0.71
  F1: 0.77

ROUGE-2:
  Recall: 2/5 = 0.40 (matched: "the cat", "the mat")
  Precision: 2/6 = 0.33
  F1: 0.36

Best for: Summarization, text compression

---

BERTScore

What it measures: Semantic similarity using contextual embeddings.

How it works: 1. Generate BERT embeddings for each token 2. Compute cosine similarity between token pairs 3. Apply greedy matching to find best alignment 4. Aggregate into Precision, Recall, F1

Advantages over lexical metrics:

• Captures synonyms and paraphrases
• Context-aware matching
• Better correlation with human judgment

Example:

Reference: "The movie was excellent"
Generated: "The film was outstanding"

Lexical (BLEU): Low score (only "The" and "was" match)
BERTScore: High score (semantic meaning preserved)

Interpretation:

---

3. RAG-Specific Metrics

Context Relevance

What it measures: How relevant retrieved documents are to the query.

Calculation methods:

Method 1: Embedding similarity

relevance = cosine_similarity(
    embed(query),
    embed(context)
)

Method 2: LLM-as-judge

Prompt: "Rate the relevance of this context to the question.
Question: {question}
Context: {context}
Rate from 1-5 where 5 is highly relevant."

Target: > 0.8 for top-k contexts

---

Answer Faithfulness

What it measures: Whether the answer is supported by the context (no hallucination).

Evaluation prompt:

Given the context and answer, determine if every claim in the
answer is supported by the context.

Context: {context}
Answer: {answer}

For each claim in the answer:
1. Identify the claim
2. Find supporting evidence in context (or mark as unsupported)
3. Rate: Supported / Partially Supported / Not Supported

Overall faithfulness score: [0-1]

Scoring:

Faithfulness = (supported claims) / (total claims)

Target: > 0.95 for production systems

---

Retrieval Metrics

Example:

Query: "What is photosynthesis?"
Retrieved docs (k=5): [R, N, R, N, R]  (R=relevant, N=not relevant)
Total relevant in corpus: 10

Precision@5 = 3/5 = 0.6
Recall@5 = 3/10 = 0.3
MRR = 1/1 = 1.0 (first doc is relevant)

---

4. Human Evaluation Frameworks

Likert Scale Evaluation

Setup:

Rate the following response on a scale of 1-5:

Response: {generated_response}

Criteria:
- Relevance (1-5): Does it address the question?
- Accuracy (1-5): Is the information correct?
- Fluency (1-5): Is it well-written?
- Helpfulness (1-5): Would this be useful to the user?

Sample size guidance:

---

Comparative Evaluation (Side-by-Side)

Setup:

Compare these two responses to the question:

Question: {question}

Response A: {response_a}
Response B: {response_b}

Which response is better?
[ ] A is much better
[ ] A is slightly better
[ ] About the same
[ ] B is slightly better
[ ] B is much better

Why? _______________

Advantages:

• Easier for humans than absolute scoring
• Reduces calibration issues
• Clear winner for A/B decisions

Analysis:

Win rate = (A wins + 0.5 × ties) / total
Bradley-Terry model for ranking multiple variants

---

LLM-as-Judge

Setup:

You are an expert evaluator. Rate the quality of this response.

Question: {question}
Response: {response}
Reference (if available): {reference}

Evaluate on:
1. Correctness (0-10): Is the information accurate?
2. Completeness (0-10): Does it fully address the question?
3. Clarity (0-10): Is it easy to understand?
4. Conciseness (0-10): Is it appropriately brief?

Provide scores and brief justification for each.
Overall score (0-10):

Calibration techniques:

• Include reference responses with known scores
• Use chain-of-thought for reasoning
• Compare against human baseline periodically

Known biases:

---

5. A/B Testing for Prompts

Experiment Design

Hypothesis template:

H0: Prompt A and Prompt B have equal performance on [metric]
H1: Prompt B improves [metric] by at least [minimum detectable effect]

Sample size calculation:

n = 2 × ((z_α + z_β)² × σ²) / δ²

Where:
- z_α = 1.96 for 95% confidence
- z_β = 0.84 for 80% power
- σ = standard deviation of metric
- δ = minimum detectable effect

Quick reference:

---

Metrics to Track

Primary metrics:

Guardrail metrics:

---

Analysis Framework

Statistical test selection:

Is the metric binary (success/failure)?
├── Yes → Chi-squared test or Z-test for proportions
└── No
    └── Is the data normally distributed?
        ├── Yes → Two-sample t-test
        └── No → Mann-Whitney U test

Interpreting results:

p-value < 0.05: Statistically significant
Effect size (Cohen's d):
  - Small: 0.2
  - Medium: 0.5
  - Large: 0.8

Decision: Ship if p < 0.05 AND effect size meets threshold AND guardrails pass

---

6. Benchmark Datasets

General NLP Benchmarks

RAG Benchmarks

Creating Custom Benchmarks

Template:

{
  "id": "custom-001",
  "input": "What are the symptoms of diabetes?",
  "expected_output": "Common symptoms include...",
  "metadata": {
    "category": "medical",
    "difficulty": "easy",
    "source": "internal docs"
  },
  "evaluation": {
    "type": "semantic_similarity",
    "threshold": 0.85
  }
}

Best practices:

• Minimum 100 examples per category
• Include edge cases (10-20%)
• Balance difficulty levels
• Version control your benchmark
• Update quarterly

---

7. Evaluation Pipeline Design

Automated Evaluation Pipeline

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Prompt    │────▶│   LLM API   │────▶│   Output    │
│   Version   │     │             │     │   Storage   │
└─────────────┘     └─────────────┘     └──────┬──────┘
                                               │
                    ┌──────────────────────────┘
                    ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Metrics   │◀────│  Evaluator  │◀────│  Benchmark  │
│   Dashboard │     │   Service   │     │   Dataset   │
└─────────────┘     └─────────────┘     └─────────────┘

Implementation Checklist

□ Define success metrics
  □ Primary metric (what you're optimizing)
  □ Guardrail metrics (what must not regress)
  □ Monitoring metrics (operational health)

□ Create benchmark dataset
  □ Representative samples from production
  □ Edge cases and failure modes
  □ Golden answers or human labels

□ Set up evaluation infrastructure
  □ Automated scoring pipeline
  □ Version control for prompts
  □ Results tracking and comparison

□ Establish baseline
  □ Run current prompt against benchmark
  □ Document scores for all metrics
  □ Set improvement targets

□ Run experiments
  □ Test one change at a time
  □ Use statistical significance testing
  □ Check all guardrail metrics

□ Deploy and monitor
  □ Gradual rollout (canary)
  □ Real-time metric monitoring
  □ Rollback plan if regression

---

Quick Reference: Metric Selection

Prompt Engineering Patterns

Specific prompt techniques with example inputs and expected outputs.

Patterns Index

1. Zero-Shot Prompting 2. Few-Shot Prompting 3. Chain-of-Thought (CoT) 4. Role Prompting 5. Structured Output 6. Self-Consistency 7. ReAct (Reasoning + Acting) 8. Tree of Thoughts 9. Retrieval-Augmented Generation 10. Meta-Prompting

---

1. Zero-Shot Prompting

When to use: Simple, well-defined tasks where the model has sufficient training knowledge.

Pattern:

[Task instruction]
[Input]

Example:

Input:

Classify the following customer review as positive, negative, or neutral.

Review: "The shipping was fast but the product quality was disappointing."

Expected Output:

negative

Best practices:

• Be explicit about output format
• Use clear, unambiguous verbs (classify, extract, summarize)
• Specify constraints (word limits, format requirements)

When to avoid:

• Tasks requiring specific formatting the model hasn't seen
• Domain-specific tasks requiring specialized knowledge
• Tasks where consistency is critical

---

2. Few-Shot Prompting

When to use: Tasks requiring consistent formatting or domain-specific patterns.

Pattern:

[Task description]

Example 1:
Input: [example input]
Output: [example output]

Example 2:
Input: [example input]
Output: [example output]

Now process:
Input: [actual input]
Output:

Example:

Input:

Extract the company name and founding year from the text.

Example 1:
Input: "Apple Inc. was founded in 1976 by Steve Jobs."
Output: {"company": "Apple Inc.", "year": 1976}

Example 2:
Input: "Microsoft Corporation started in 1975."
Output: {"company": "Microsoft Corporation", "year": 1975}

Example 3:
Input: "Founded in 1994, Amazon has grown into a tech giant."
Output: {"company": "Amazon", "year": 1994}

Now process:
Input: "Tesla, Inc. was established in 2003 by Martin Eberhard."
Output:

Expected Output:

{"company": "Tesla, Inc.", "year": 2003}

Example selection guidelines:

Common mistakes:

• Too many examples (wastes tokens, may confuse)
• Inconsistent formatting between examples
• Examples too similar (model doesn't learn variation)

---

3. Chain-of-Thought (CoT)

When to use: Math problems, logical reasoning, multi-step analysis.

Pattern:

[Problem]

Think through this step by step:

Example:

Input:

A store sells apples for $2 each. If you buy 5 or more, you get 20% off.
How much do 7 apples cost?

Think through this step by step:

Expected Output:

Step 1: Calculate the regular price for 7 apples
7 apples × $2 = $14

Step 2: Check if discount applies
7 apples ≥ 5 apples, so 20% discount applies

Step 3: Calculate the discount amount
20% of $14 = 0.20 × $14 = $2.80

Step 4: Calculate final price
$14 - $2.80 = $11.20

The 7 apples cost $11.20

Variations:

Zero-shot CoT: Simply append "Let's think step by step." to any prompt.

---

4. Role Prompting

When to use: Tasks requiring expertise, specific perspective, or consistent persona.

Pattern:

You are a [role/expert description].

[Context about the task]

[Specific instruction]

Example:

Input:

You are a senior security engineer with 15 years of experience in web application security.

Review the following code snippet for security vulnerabilities:

def login(username, password): query = f"SELECT * FROM users WHERE username='{username}' AND password='{password}'" result = db.execute(query) return result

Identify vulnerabilities and provide specific fixes.

Expected Output:

Security Vulnerabilities Identified:

1. SQL Injection (Critical)
   - The code directly interpolates user input into SQL query
   - Attacker can input: username=' OR '1'='1' --
   - This bypasses authentication entirely

   Fix:

def login(username, password): query = "SELECT * FROM users WHERE username=? AND password=?" result = db.execute(query, (username, password)) return result

2. Plain-text Password Storage (Critical)
   - Passwords appear to be stored and compared in plain text
   - Fix: Use bcrypt or argon2 for password hashing

3. Missing Rate Limiting (Medium)
   - No protection against brute force attacks
   - Fix: Implement login attempt throttling

Effective role descriptions:

---

5. Structured Output

When to use: When you need parseable responses (JSON, XML, CSV).

Pattern:

[Task instruction]

Respond in JSON format with exactly these fields:
- field1 (type): description
- field2 (type): description

[Input]

Return ONLY valid JSON, no markdown or explanation.

Example:

Input:

Extract meeting details from this email.

Respond in JSON format with exactly these fields:
- date (string, ISO format): Meeting date
- time (string, 24h format): Meeting time
- attendees (array of strings): List of attendees
- topic (string): Meeting topic
- location (string or null): Meeting location if mentioned

Email: "Hi team, let's meet tomorrow at 2pm to discuss Q4 planning.
Sarah, Mike, and Lisa should attend. We'll use Conference Room B."

Today's date is 2024-01-15.

Return ONLY valid JSON, no markdown or explanation.

Expected Output:

{
  "date": "2024-01-16",
  "time": "14:00",
  "attendees": ["Sarah", "Mike", "Lisa"],
  "topic": "Q4 planning",
  "location": "Conference Room B"
}

Format enforcement techniques:

# Strong enforcement
"Return ONLY valid JSON. Start with { and end with }"

# Schema validation hint
"The output must be valid JSON matching this TypeScript type:
type Output = { name: string; age: number; active: boolean }"

# Negative instruction
"Do NOT include markdown code blocks. Do NOT add explanations."

---

6. Self-Consistency

When to use: Complex reasoning where multiple valid paths exist.

Pattern: 1. Generate multiple reasoning paths (temperature > 0) 2. Extract final answers from each path 3. Select most common answer (majority vote)

Example approach:

# Run this prompt 5 times with temperature=0.7

Solve this logic puzzle. Think through it step by step.

Three friends (Alice, Bob, Carol) each have a different pet (cat, dog, bird).
- Alice doesn't have the dog
- The person with the bird is not Carol
- Bob's pet has four legs

What pet does each person have?

Implementation:

# Pseudo-code for self-consistency
answers = []
for i in range(5):
    response = llm.generate(prompt, temperature=0.7)
    answer = extract_final_answer(response)
    answers.append(answer)

final_answer = majority_vote(answers)

When to use:

• Math word problems
• Logic puzzles
• Any task where you can verify consistency

---

7. ReAct (Reasoning + Acting)

When to use: Tasks requiring external tool use or information gathering.

Pattern:

Answer the question using the available tools.

Tools available:
- search(query): Search the web
- calculate(expression): Evaluate math
- lookup(term): Look up in knowledge base

Format your response as:
Thought: [reasoning about what to do]
Action: [tool_name(arguments)]
Observation: [result from tool]
... (repeat as needed)
Thought: I now have enough information
Final Answer: [answer]

Question: [user question]

Example:

Input:

Question: What is the population of France divided by the population of Belgium?

Expected Output:

Thought: I need to find the population of France and Belgium, then calculate the ratio.

Action: search("population of France 2024")
Observation: France has a population of approximately 67.75 million.

Thought: Now I need Belgium's population.

Action: search("population of Belgium 2024")
Observation: Belgium has a population of approximately 11.7 million.

Thought: Now I can calculate the ratio.

Action: calculate(67.75 / 11.7)
Observation: 5.79

Thought: I now have enough information to answer.

Final Answer: The population of France is approximately 5.79 times the population of Belgium.

---

8. Tree of Thoughts

When to use: Complex problems requiring exploration of multiple solution paths.

Pattern:

Problem: [complex problem]

Generate 3 different approaches to solve this:

Approach 1: [description]
Evaluation: [pros/cons]

Approach 2: [description]
Evaluation: [pros/cons]

Approach 3: [description]
Evaluation: [pros/cons]

Best approach: [selection with reasoning]
Solution using best approach: [detailed solution]

Example:

Input:

Problem: Design a system to handle 10,000 concurrent user sessions with sub-100ms response times.

Generate 3 different architectural approaches:

Expected Output:

Approach 1: Horizontal scaling with load balancer
- Multiple stateless app servers behind load balancer
- Session state in Redis cluster
Evaluation: Simple, proven pattern. May have Redis as bottleneck.

Approach 2: Serverless with edge computing
- AWS Lambda@Edge or Cloudflare Workers
- Session state in DynamoDB Global Tables
Evaluation: Auto-scales, but cold starts may impact latency.

Approach 3: Actor model with sticky sessions
- Akka/Orleans cluster with virtual actors
- Session state co-located with actor
Evaluation: Lowest latency, but complex to operate.

Best approach: Approach 1 (Horizontal scaling)
Reasoning: Best balance of simplicity, reliability, and performance for the given requirements.

Solution: [detailed architecture follows]

---

9. Retrieval-Augmented Generation

When to use: Tasks requiring factual accuracy or current information.

Pattern:

Use the following context to answer the question. If the answer is not in the context, say "I don't have information about that."

Context:
[retrieved documents]

Question: [user question]

Answer based only on the context provided:

Example:

Input:

Use the following context to answer the question.

Context:
Document 1: "The Acme Widget 3000 was released in March 2024. It features
a 12-hour battery life and weighs 250 grams."

Document 2: "Pricing for the Acme Widget 3000 starts at $299 for the base
model. The Pro version with extended battery is $399."

Question: How much does the Acme Widget 3000 cost and what is its battery life?

Answer based only on the context provided:

Expected Output:

The Acme Widget 3000 starts at $299 for the base model, with a Pro version
available at $399. The standard model has a 12-hour battery life.

Chunking strategies:

---

10. Meta-Prompting

When to use: Generating or improving prompts programmatically.

Pattern:

You are a prompt engineering expert.

Task: [description of what the prompt should do]
Target model: [GPT-4/Claude/etc.]
Constraints: [length limits, format requirements]

Generate an optimized prompt for this task.

Example:

Input:

You are a prompt engineering expert.

Task: Create a prompt that extracts action items from meeting notes
Target model: GPT-4
Constraints:
- Output must be valid JSON
- Each action item needs: task, owner, due_date
- Handle cases where information is missing

Generate an optimized prompt for this task.

Expected Output:

Extract action items from the following meeting notes.

For each action item, identify:
- task: The specific action to be taken
- owner: Person responsible (use "Unassigned" if not mentioned)
- due_date: Deadline in YYYY-MM-DD format (use null if not specified)

Meeting Notes:
{meeting_notes}

Respond with a JSON array. Example format:
[
  {"task": "Review proposal", "owner": "Sarah", "due_date": "2024-01-20"},
  {"task": "Send update", "owner": "Unassigned", "due_date": null}
]

Return ONLY the JSON array, no additional text.

---

Pattern Selection Guide

Tools & Workflows

Read this when running the optimizer/evaluator/orchestrator tools or executing a prompt-engineering workflow.

Quick Start

# Analyze and optimize a prompt file
python scripts/prompt_optimizer.py prompts/my_prompt.txt --analyze

# Evaluate RAG retrieval quality
python scripts/rag_evaluator.py --contexts contexts.json --questions questions.json

# Visualize agent workflow from definition
python scripts/agent_orchestrator.py agent_config.yaml --visualize

---

Tools Overview

1. Prompt Optimizer

Analyzes prompts for token efficiency, clarity, and structure. Generates optimized versions.

Input: Prompt text file or string Output: Analysis report with optimization suggestions

Usage:

# Analyze a prompt file
python scripts/prompt_optimizer.py prompt.txt --analyze

# Output:
# Token count: 847
# Estimated cost: $0.0025 (GPT-4)
# Clarity score: 72/100
# Issues found:
#   - Ambiguous instruction at line 3
#   - Missing output format specification
#   - Redundant context (lines 12-15 repeat lines 5-8)
# Suggestions:
#   1. Add explicit output format: "Respond in JSON with keys: ..."
#   2. Remove redundant context to save 89 tokens
#   3. Clarify "analyze" -> "list the top 3 issues with severity ratings"

# Generate optimized version
python scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt

# Count tokens for cost estimation
python scripts/prompt_optimizer.py prompt.txt --tokens --model gpt-4

# Extract and manage few-shot examples
python scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json

---

2. RAG Evaluator

Evaluates Retrieval-Augmented Generation quality by measuring context relevance and answer faithfulness.

Input: Retrieved contexts (JSON) and questions/answers Output: Evaluation metrics and quality report

Usage:

# Evaluate retrieval quality
python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json

# Output:
# === RAG Evaluation Report ===
# Questions evaluated: 50
#
# Retrieval Metrics:
#   Context Relevance: 0.78 (target: >0.80)
#   Retrieval Precision@5: 0.72
#   Coverage: 0.85
#
# Generation Metrics:
#   Answer Faithfulness: 0.91
#   Groundedness: 0.88
#
# Issues Found:
#   - 8 questions had no relevant context in top-5
#   - 3 answers contained information not in context
#
# Recommendations:
#   1. Improve chunking strategy for technical documents
#   2. Add metadata filtering for date-sensitive queries

# Evaluate with custom metrics
python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json \
    --metrics relevance,faithfulness,coverage

# Export detailed results
python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json \
    --output report.json --verbose

---

3. Agent Orchestrator

Parses agent definitions and visualizes execution flows. Validates tool configurations.

Input: Agent configuration (YAML/JSON) Output: Workflow visualization, validation report

Usage:

# Validate agent configuration
python scripts/agent_orchestrator.py agent.yaml --validate

# Output:
# === Agent Validation Report ===
# Agent: research_assistant
# Pattern: ReAct
#
# Tools (4 registered):
#   [OK] web_search - API key configured
#   [OK] calculator - No config needed
#   [WARN] file_reader - Missing allowed_paths
#   [OK] summarizer - Prompt template valid
#
# Flow Analysis:
#   Max depth: 5 iterations
#   Estimated tokens/run: 2,400-4,800
#   Potential infinite loop: No
#
# Recommendations:
#   1. Add allowed_paths to file_reader for security
#   2. Consider adding early exit condition for simple queries

# Visualize agent workflow (ASCII)
python scripts/agent_orchestrator.py agent.yaml --visualize

# Output:
# ┌─────────────────────────────────────────┐
# │            research_assistant           │
# │              (ReAct Pattern)            │
# └─────────────────┬───────────────────────┘
#                   │
#          ┌────────▼────────┐
#          │   User Query    │
#          └────────┬────────┘
#                   │
#          ┌────────▼────────┐
#          │     Think       │◄──────┐
#          └────────┬────────┘       │
#                   │                │
#          ┌────────▼────────┐       │
#          │   Select Tool   │       │
#          └────────┬────────┘       │
#                   │                │
#     ┌─────────────┼─────────────┐  │
#     ▼             ▼             ▼  │
# [web_search] [calculator] [file_reader]
#     │             │             │  │
#     └─────────────┼─────────────┘  │
#                   │                │
#          ┌────────▼────────┐       │
#          │    Observe      │───────┘
#          └────────┬────────┘
#                   │
#          ┌────────▼────────┐
#          │  Final Answer   │
#          └─────────────────┘

# Export workflow as Mermaid diagram
python scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid

---

Prompt Engineering Workflows

Prompt Optimization Workflow

Use when improving an existing prompt's performance or reducing token costs.

Step 1: Baseline current prompt

python scripts/prompt_optimizer.py current_prompt.txt --analyze --output baseline.json

Step 2: Identify issues Review the analysis report for:

• Token waste (redundant instructions, verbose examples)
• Ambiguous instructions (unclear output format, vague verbs)
• Missing constraints (no length limits, no format specification)

Step 3: Apply optimization patterns

Step 4: Generate optimized version

python scripts/prompt_optimizer.py current_prompt.txt --optimize --output optimized.txt

Step 5: Compare results

python scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json
# Shows: token reduction, clarity improvement, issues resolved

Step 6: Validate with test cases Run both prompts against your evaluation set and compare outputs.

---

Few-Shot Example Design Workflow

Use when creating examples for in-context learning.

Step 1: Define the task clearly

Task: Extract product entities from customer reviews
Input: Review text
Output: JSON with {product_name, sentiment, features_mentioned}

Step 2: Select diverse examples (3-5 recommended)

Step 3: Format consistently

Example 1:
Input: "Love my new iPhone 15, the camera is amazing!"
Output: {"product_name": "iPhone 15", "sentiment": "positive", "features_mentioned": ["camera"]}

Example 2:
Input: "The laptop was okay but battery life is terrible."
Output: {"product_name": "laptop", "sentiment": "mixed", "features_mentioned": ["battery life"]}

Step 4: Validate example quality

python scripts/prompt_optimizer.py prompt_with_examples.txt --validate-examples
# Checks: consistency, coverage, format alignment

Step 5: Test with held-out cases Ensure model generalizes beyond your examples.

---

Structured Output Design Workflow

Use when you need reliable JSON/XML/structured responses.

Step 1: Define schema

{
  "type": "object",
  "properties": {
    "summary": {"type": "string", "maxLength": 200},
    "sentiment": {"enum": ["positive", "negative", "neutral"]},
    "confidence": {"type": "number", "minimum": 0, "maximum": 1}
  },
  "required": ["summary", "sentiment"]
}

Step 2: Include schema in prompt

Respond with JSON matching this schema:
- summary (string, max 200 chars): Brief summary of the content
- sentiment (enum): One of "positive", "negative", "neutral"
- confidence (number 0-1): Your confidence in the sentiment

Step 3: Add format enforcement

IMPORTANT: Respond ONLY with valid JSON. No markdown, no explanation.
Start your response with { and end with }

Step 4: Validate outputs

python scripts/prompt_optimizer.py structured_prompt.txt --validate-schema schema.json

---

Common Patterns Quick Reference

---

Common Commands

# Prompt Analysis
python scripts/prompt_optimizer.py prompt.txt --analyze          # Full analysis
python scripts/prompt_optimizer.py prompt.txt --tokens           # Token count only
python scripts/prompt_optimizer.py prompt.txt --optimize         # Generate optimized version

# RAG Evaluation
python scripts/rag_evaluator.py --contexts ctx.json --questions q.json  # Evaluate
python scripts/rag_evaluator.py --contexts ctx.json --compare baseline  # Compare to baseline

# Agent Development
python scripts/agent_orchestrator.py agent.yaml --validate       # Validate config
python scripts/agent_orchestrator.py agent.yaml --visualize      # Show workflow
python scripts/agent_orchestrator.py agent.yaml --estimate-cost  # Token estimation

---

Troubleshooting

---

Success Criteria

• Prompt clarity score above 70/100 on all production prompts, measured via prompt_optimizer.py --analyze
• Token efficiency improved by 30%+ after applying optimization suggestions and removing redundant content
• RAG context relevance at or above 0.80 across evaluation sets, verified by rag_evaluator.py
• Answer faithfulness at or above 0.95 with zero unsupported claims in critical workflows
• Agent validation passes with zero errors for all deployed agent configurations
• Cost per agent run within budget --- estimated monthly spend confirmed via agent_orchestrator.py --estimate-cost
• Few-shot example coverage includes edge cases --- at least 1 simple, 1 complex, and 1 negative example per prompt template

---

Tool Reference

prompt_optimizer.py

Purpose: Static analysis tool for prompt engineering. Estimates token counts, scores clarity and structure, detects ambiguous instructions and redundant content, extracts few-shot examples, and generates optimized prompt versions.

Usage:

python scripts/prompt_optimizer.py <prompt_file> [options]

Parameters:

Example:

python scripts/prompt_optimizer.py prompt.txt --analyze --model claude-3-sonnet --json

Output Formats:

• Default (text): Human-readable report with metrics, scores, detected sections, issues, and suggestions
• JSON (`--json`): Structured PromptAnalysis object with keys: token_count, estimated_cost, model, clarity_score, structure_score, issues, suggestions, sections, has_examples, example_count, has_output_format, word_count, line_count
• Token-only (`--tokens`): Single-line token count and cost estimate
• Examples (`--extract-examples`): JSON array of {input_text, output_text, index} objects
• Optimized (`--optimize`): Cleaned prompt text with normalized whitespace

---

rag_evaluator.py

Purpose: Evaluates Retrieval-Augmented Generation quality by measuring context relevance (lexical overlap, term coverage), answer faithfulness (claim-level verification), groundedness (ROUGE-L), and retrieval metrics (Precision@K, MRR, NDCG).

Usage:

python scripts/rag_evaluator.py --contexts <contexts.json> --questions <questions.json> [options]

Parameters:

Example:

python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json --k 10 --verbose --output report.json

Output Formats:

• Default (text): Human-readable report with summary, retrieval metrics (context relevance, Precision@K), generation metrics (faithfulness, groundedness), issues, and recommendations
• JSON (`--json`): Structured RAGEvaluationReport object with keys: total_questions, avg_context_relevance, avg_faithfulness, avg_groundedness, retrieval_metrics, coverage, issues, recommendations, question_details
• Verbose (`--verbose`): Adds per-question question_details array containing individual context scores and faithfulness breakdowns

---

agent_orchestrator.py

Purpose: Parses agent configurations (YAML or JSON), validates tool registrations and flow correctness, generates ASCII or Mermaid workflow diagrams, and estimates token costs per run and monthly spend.

Usage:

python scripts/agent_orchestrator.py <config_file> [options]

Parameters:

Example:

python scripts/agent_orchestrator.py agent.yaml --validate --visualize --format mermaid --output workflow.md

Output Formats:

• Validation (text): Agent info, tool status with OK/WARN indicators, flow analysis (max iterations, token estimate, loop detection), errors, and warnings
• Validation (JSON, `--json`): Structured ValidationResult object with keys: is_valid, errors, warnings, tool_status, estimated_tokens_per_run, potential_infinite_loop, max_depth
• Visualization (`--visualize`): ASCII box-drawing diagram (default) or Mermaid flowchart (--format mermaid) showing the agent pattern flow and registered tools
• Cost estimation (`--estimate-cost`): Token range per run, cost range per run, and projected monthly cost at the specified daily run rate