Engineering Manager Agent Prompts Evals

Engineering Manager, Agent Prompts & Evals

When to Use

• Build or scale a prompt + eval engineering function (central or embedded)
• Prioritize golden set, harness, and judge work with PM and risk
• Define release policy for prompt/tool changes (gates, waivers, rollback)
• Hire and level prompt engineers, eval engineers, and tech leads
• Resolve capacity conflicts between new agents and eval debt
• Report eval health (pass rate, slice regressions, judge drift) to leadership
• Partner with ai-lead-ops on production incidents tied to prompts

When NOT to Use

• Author system prompts, tool schemas, or eval cases → prompt-engineer-agent-prompts-evals
• General prompt patterns (CoT, few-shot) → prompt-engineer
• Full RAG/agent application code → ai-engineer
• Vertical AI product roadmap and GTM launches → engineering-manager-vertical-ai-products
• Adversarial red-team execution → ai-redteam
• Org-wide data or analytics management → data-manager

Related skills

Core Workflows

1. Org design

Central eval platform vs embedded prompt owners; ratios and interfaces.

See `references/team_org_prompt_eval.md`.

2. Roadmap and prioritization

Eval debt, new agent coverage, judge calibration, CI investment.

See `references/roadmap_prioritization.md`.

3. Release governance

Prompt semver gates, waivers, coordination with risk and ops.

See `references/release_governance.md`.

4. Stakeholder partnerships

PM, applied architect, risk, red-team, platform eng.

See `references/stakeholder_partnerships.md`.

5. Hiring and development

Levels for prompt/eval specialists.

See `references/hiring_development.md`.

6. Team metrics

Pass rate, coverage, time-to-golden-case, incident linkage.

See `references/team_metrics_accountability.md`.

Output standards

• Roadmap items name eval slice, prompt surface, and risk tier
• No prompt prod change without documented baseline vs candidate
• Waivers require risk sign-off for Tier 1–2 customer-facing agents
• Escalations include trade-offs (scope, date, headcount)

When to load references

• Org → references/team_org_prompt_eval.md
• Roadmap → references/roadmap_prioritization.md
• Release → references/release_governance.md
• Stakeholders → references/stakeholder_partnerships.md
• People → references/hiring_development.md
• KPIs → references/team_metrics_accountability.md

Hiring and Development

Profiles

Interview depth: prompt-engineer-agent-prompts-evals references.

Interview loop

Levels (summary)

Onboarding

Growth paths

• Deep specialist (judges, safety goldens)
• Platform eval infra
• EM or vertical AI EM pivot

Retention

• Chronic ship-without-eval culture
• Flaky CI ignored
• No path to impact beyond firefighting

Release Governance (Prompt & Eval)

Gate stack (customer-facing agents)

IC checklist detail: prompt-engineer-agent-prompts-evals → references/prompt_versioning_regression.md.

Waiver process

1. Document failing cases and business justification 2. Compensating control (human review, feature flag, limited audience) 3. Risk approver for tier 4. Expiry date and remediation ticket

Log all waivers; review monthly in leadership sync.

Rollback

• N-1 prompt in config; flag per version
• Eval must pass on N-1 before rollback deploy
• Post-rollback: add regression cases for failure mode

Internal / low-tier agents

Reduced gates — still require smoke golden set and owner sign-off.

Manager responsibilities

• Enforce no hotfix prompt without trace + case addition
• Align with vertical EM (engineering-manager-vertical-ai-products) on launch calendar
• Communicate eval status in ship/no-ship meetings

Roadmap and Prioritization

Backlog item template

Priority buckets

Reserve 25–30% capacity for P0/P1 buffer and harness maintenance.

Trade-offs (manager decisions)

Metrics for roadmap reviews

• Pass rate trend by tag (not aggregate only)
• Time-to-add-golden after production failure
• Open waiver count
• Judge–human agreement drift

Escalation

Escalate when: launch date conflicts with eval gate; risk refuses waiver without staffing; platform blocks harness access.

Bring options: reduce scope, delay GA, temporary human review layer.

Stakeholder Partnerships

Product management

Applied AI architect

• Escalate new tool surfaces and multi-agent handoffs
• Do not bypass ADRs for tenant/data boundaries

Risk and compliance

• Tier assignment before roadmap commit
• Waiver path for time pressure
• Policy updates → new refusal goldens within SLA

Red team

• Schedule before Tier 1–2 GA
• Convert findings to regression tags in golden set
• Not a substitute for continuous CI

AI lead ops

• Incidents caused by prompts: joint post-mortem
• Production judge sampling alignment
• Model version changes → re-baseline plan

IC team (prompt-engineer-agent-prompts-evals)

EM protects eval debt time; ICs own technical depth.

Status rhythm

Anti-pattern: EM rewriting prompts in prod without IC review.

Team Metrics and Accountability

Scorecard

Targets (set per org)

• No drop on safety/refusal slice without waiver
• Time-to-golden < 5 business days for Tier 1–2
• Eval debt burn: N cases/quarter per agent

Accountability

Reporting

Weekly: slice heatmap, blockers, waivers open

Quarterly: coverage map per agent, harness investment ROI (fewer incidents, faster launches)

vs ai-lead-ops

Ops owns SLOs and incidents; this function owns preventive eval quality and prompt change discipline.

Team Org — Prompt & Eval

Functions to separate

EM may own first three; partner for platform.

Models

Avoid: every squad forks harness; no shared pass-rate dashboard.

Interfaces

Staffing

• 1 eval engineer per 2–3 active agents in flight (rule of thumb)
• Dedicated judge calibration owner when >1 LLM judge in prod
• Tech lead when golden set >500 cases or multi-repo harness

Anti-patterns

• PM owns golden set without engineering versioning
• No on-call rotation for prompt-related prod regressions
• Team only reacts to fires — no eval debt budget