Prompt Engineer Agent Prompts Evals

Prompt Engineer — Agent Prompts & Evals

When to Use

• Write or refactor system/developer prompts for agents with tools
• Design tool descriptions and parameter schemas the model must choose correctly
• Build golden datasets and scenario suites (multi-turn, failures, edge cases)
• Implement offline eval harnesses and CI regression for prompts
• Define LLM-as-judge rubrics and human calibration samples
• Set release gates when prompts or tool lists change
• Debug wrong tool, looping, or format break in agent traces

When NOT to Use

• General prompt patterns without agent/eval focus → prompt-engineer
• End-to-end RAG indexing, retrieval, serving → ai-engineer
• Jailbreak and abuse red-team engagements → ai-redteam
• Org-wide model rollout and incident ops → ai-lead-ops
• Token cost program across product → ai-token-improvement-plan-engineer
• Vertical squad management and launch PM → engineering-manager-vertical-ai-products
• Prompt/eval team management and governance → engineering-manager-agent-prompts-evals

Related skills

Core Workflows

1. Agent prompt structure

System vs developer messages, tools block, constraints, handoffs.

See `references/agent_system_prompts.md`.

2. Eval datasets

Golden sets, coverage matrix, synthetic and SME-labeled examples.

See `references/eval_dataset_design.md`.

3. Harness and metrics

Offline runs, pass/fail, tool accuracy, trajectory checks.

See `references/eval_harness_patterns.md`.

4. Judges and rubrics

LLM judge design, bias controls, human agreement.

See `references/llm_judge_rubrics.md`.

5. Versioning and regression

Prompt semver, baselines, CI gates, rollback.

See `references/prompt_versioning_regression.md`.

6. Scenario catalog

Multi-turn, tool error, refusal, escalation cases.

See `references/agent_eval_scenarios.md`.

Output standards

• Every eval case has input, expected behavior (not always exact text), tags
• Tool schema changes include eval delta in PR description
• Judge rubrics published with scoring scale and known failure modes
• No production prompt change without baseline comparison on golden set

When to load references

• Prompts → references/agent_system_prompts.md
• Data → references/eval_dataset_design.md
• Harness → references/eval_harness_patterns.md
• Judges → references/llm_judge_rubrics.md
• CI → references/prompt_versioning_regression.md
• Scenarios → references/agent_eval_scenarios.md

Agent Eval Scenarios

Scenario catalog (minimum)

Extend per product domain.

Multi-turn threads

Store as array of user turns; run harness turn-by-turn with memory as production does.

Check state leakage between turns (wrong user context).

Tool failure injection

In harness, mock tool layer:

• return_empty
• return_error_500
• delay_ms

Verify agent behavior matches rubric — not live dependency in CI.

Safety scenarios

Align tags with ai-risk-governance tier:

• Refusal without preaching
• No credential or internal URL leak
• No executing user-supplied code unless product allows

Add cases from ai-redteam findings as regression items.

Debugging failed case

1. Reproduce with saved trace (messages + tool calls) 2. Classify: prompt, tool schema, retrieval, model, harness bug 3. Fix smallest layer; add case if missing coverage 4. Re-run slice before full suite

Agent System Prompts

Message layers

Keep stable content in system; put volatile context in developer with clear delimiters.

System prompt skeleton

Role + scope (what you can/cannot do)
Tool-use rules (when to call, when to answer directly)
Output format (markdown, JSON schema, citations)
Refusal and escalation (what to decline, how)
Uncertainty (ask clarifying questions vs guess)

Tool schema quality

Models choose tools from name + description + parameter descriptions.

• One clear purpose per tool
• Negative constraints ("do not use for X")
• Required vs optional params explicit

Handoffs and subagents

Pass structured handoff payload (goal, constraints, prior tool outputs summary) — not full raw logs unless needed.

Anti-patterns

• Tool dump (20+ tools with vague descriptions)
• Instructions only in user message
• Conflicting rules between system and developer
• "Always call a tool first" without exceptions

Eval Dataset Design

Case anatomy

Expected behavior types

Avoid brittle full-string match on prose unless necessary.

Coverage matrix

Tag cases across:

Target 50–200 cases for CI; grow with every production failure.

Building the set

Version dataset with prompt (dataset v3 + prompt v1.4).

Maintenance

• Add case before fixing prompt (repro first)
• Retire flaky cases or tighten expected behavior
• Review quarterly for stale product facts

Eval Harness Patterns

Harness responsibilities

1. Load prompt version + tool definitions 2. Run agent on each case (fixed model + temperature for CI) 3. Assert expectations (tools, text, schema, trajectory) 4. Emit report (pass rate, diffs, traces)

Metrics

Track per-tag pass rate to spot weak slices.

Assertion styles

assert_tool_called(name, args_subset)
assert_no_tool()
assert_json_schema(schema_id)
assert_refusal(category)
assert_judge_score(rubric_id, min_score)

Prefer deterministic checks; use judge for nuance.

CI integration

Store traces as artifacts for failed cases.

Flakiness control

• Fixed model version in CI
• Low temperature (0–0.2) for deterministic suites
• Separate flaky quarantine folder with stricter retry policy
• Human review queue for judge disagreement

Integration with ai-engineer

Harness code lives in app repo; this skill defines what to measure and case design — implement runners per stack (LangSmith, custom, etc.).

LLM Judge Rubrics

When to use a judge

Rubric structure

Criterion: Grounded in retrieved context only
Scale: 1–5 (1 = hallucination, 5 = fully grounded with citations)
Failure examples: Invents policy not in context

Publish rubric in repo; version with prompt.

Reducing judge bias

• Blind model to prompt variant labels
• Separate judges per criterion vs one vague score
• Anchor with 2–3 scored examples in judge prompt
• Calibrate monthly against human labels (target κ or % agreement)
• Use different model family than agent under test when affordable

Judge prompt template

You are an evaluator. Score only on [criterion].
Context: ...
Agent output: ...
Rubric: ...
Respond JSON: { "score": int, "reason": string }

Human calibration

Abuse

Judges can be gamed — combine with spot human audit and deterministic safety cases.

For adversarial robustness use ai-redteam, not judge alone.

Prompt Versioning and Regression

Versioning

Store in git; tag releases aligned with app deploy.

Change workflow

1. Branch prompt change 2. Run harness vs baseline on golden set 3. Document delta: pass rate by tag, new failures, token delta 4. SME review for critical tag regressions 5. Merge with required approvers (eng + risk if Tier 1–2)

Regression policy

Allow waivers with risk sign-off — logged in release notes.

Rollback

• Keep N-1 prompt in config for instant revert
• Feature flag per prompt version
• Post-deploy monitor: judge sample + error rate 24h

PR template (minimal)

Prompt version: x.y.z
Eval dataset: ...
Pass rate: baseline → candidate (by tag table)
Notable failures: links to traces
SME sign-off: Y/N

Coordinate gates with engineering-manager-vertical-ai-products for customer-facing launches.