Ai Lead Ops

AI Lead Ops

When to Use

• Standing up AI platform operations and production service reliability
• Defining SLAs/SLOs for LLM-powered features
• Running AI incident reviews and post-mortems
• Governing model, prompt, and index rollouts with tiered gates
• Tracking AI unit economics (cost per session, tokens per feature)
• Coordinating red-team and evaluation gates before releases
• Building team rituals and cadence across engineering, research, risk, and product
• Managing AI vendor relationships, contracts, and bake-offs

When NOT to Use

• Implementing memory stores or context packing code → ai-memory-developer / ai-context-engineer
• Building RAG pipelines or agent tools → ai-engineer
• Designing corporate AI policy or regulatory mapping → ai-risk-governance
• General network penetration testing or enterprise security programs → cybersecurity
• Structured token/cost improvement roadmaps with backlog → ai-token-improvement-plan-engineer
• Commercial/enterprise AI solution architecture → applied-ai-architect-commercial-enterprise
• Vertical AI product engineering managers and squad roadmaps → engineering-manager-vertical-ai-products

Related skills

Core Workflows

1. Operating model and cadence

Define RACI: who owns model, prompt, index, eval suite, on-call.

See `references/operating_model.md` for roles and escalation.

2. Release governance

Production promotion checklist:

• [ ] Eval regression passed on golden + safety set
• [ ] Red-team sign-off for tier-2+ use cases
• [ ] Model card / change log updated
• [ ] Canary with error and cost monitors
• [ ] Rollback procedure tested (previous prompt + model version pinned)
• [ ] Comms plan for customer-visible behavior change

See `references/release_governance.md` for tiered gates and canary metrics.

3. SLOs, incidents, and observability

Example SLIs:

AI incident types: toxic output, PII leak in logs, retrieval cross-tenant leak, runaway agent loop, vendor outage.

See `references/incidents_slos.md` for severity matrix and post-incident template.

4. Cost and capacity

• Track tokens by model, feature, tenant
• Set budgets and alerts at 80/100/110%
• Optimize via routing, caching, context engineering (partner with ai-context-engineer)
• Forecast from usage growth + model price changes

See `references/cost_capacity.md` for unit economics worksheet.

5. Vendor and eval program

• Maintain scorecard: quality, latency, safety, price, data terms
• Run structured bake-offs before annual renewals
• Own central eval harness ownership and dataset hygiene

See `references/vendor_eval_program.md` for RFP topics and eval program maturity.

When to load references

• Team cadence and RACI → references/operating_model.md
• Releases and canaries → references/release_governance.md
• SLOs and incidents → references/incidents_slos.md
• Cost and capacity → references/cost_capacity.md
• Vendors and eval ops → references/vendor_eval_program.md

Cost and capacity

Table of contents

1. Unit economics 2. Optimization levers

Unit economics

cost_per_session = (input_tokens * in_price + output_tokens * out_price) / sessions

Segment by feature, model, tenant tier.

Optimization levers

Review weekly; target gross margin per product line.

Incidents and SLOs

Table of contents

1. Severity 2. Post-incident template

Severity

Post-incident template

• Timeline (UTC)
• Customer impact
• Root cause (technical + process)
• Mitigations deployed
• Eval/regression tests added
• Action items with owners

Operating model

Table of contents

1. RACI example 2. Escalation

RACI example

Escalation

L1: on-call engineer → L2: AI tech lead → L3: ops lead + risk + legal (PII/safety)

Release governance

Table of contents

1. Risk tiers 2. Canary metrics

Risk tiers

Canary metrics

Watch for 2h minimum:

• Error rate vs baseline
• p95 latency
• Cost per request
• Thumbs-down / report rate
• Safety classifier triggers

Auto-rollback if any breach SLO burn threshold.

Vendor and eval program

Table of contents

1. Vendor scorecard 2. Eval program maturity

Vendor scorecard

Dimensions 1–5: quality, latency, safety, price, contractual data rights, support.

Run bake-off on frozen golden set before renewal.

Eval program maturity

Owner: AI ops lead with eng partnership.