Ai Token Improvement Plan Engineer

AI Token Improvement Plan Engineer

When to Use

• Audit where tokens are spent (by feature, model, tenant, turn type)
• Prioritize cost-saving initiatives with estimated savings and risk
• Draft a phased improvement plan with metrics and rollback criteria
• Define eval gates so cost cuts do not regress quality or safety
• Brief leadership on trade-offs (model tier, context size, agent depth)

When NOT to Use

• Implement context packing or compression code → ai-context-engineer
• Rewrite one prompt or agent tool schema → prompt-engineer
• Build RAG ingest, chunking, or agent runtime → ai-engineer
• Weekly cost review ritual and release governance → ai-lead-ops
• Memory store architecture → ai-memory-developer
• Adversarial safety testing → ai-redteam
• AI policy and regulatory mapping → ai-risk-governance

Related skills

Core Workflows

1. Scope and baseline

Define:

• Surface area — chat, copilot, batch, agents, embeddings-only
• Time window — 7/30 days; exclude anomalies (launches, incidents)
• Segmentation — model, feature flag, tenant tier, environment

Collect baseline metrics (see references/token_audit.md).

Deliverable: current state table with top 5 cost drivers (≥70% of spend if possible).

2. Categorize spend

Bucket each driver:

Tag fixed per request vs scales with users/sessions.

3. Initiative backlog

For each idea record:

• Description and owner team
• Estimated savings (% tokens or $/month) with assumptions
• Effort (S/M/L) and dependencies
• Quality risk (low/med/high) and required evals
• Measurement — metric that proves success

Prioritize with impact × confidence ÷ effort; never ship without eval plan.

See `references/improvement_backlog.md` and `references/optimization_playbook.md`.

4. Quality and safety guardrails

Every initiative must list:

• Golden-set evals (task accuracy, format)
• Safety set (if user-facing)
• Latency check (p95)
• Rollback trigger (e.g., CSAT drop, eval regression >X%)

See `references/measurement_and_kpis.md`.

5. Phased plan

Typical phases:

Each phase: goals, initiatives, owners, exit criteria.

See `references/rollout_plan.md`.

6. Executive summary

## Situation — spend and growth rate
## Target — $ or tokens/session goal by date
## Top drivers — ranked list
## Plan — 3–5 initiatives per phase
## Risks — quality, latency, eng capacity
## Asks — budget, headcount, eval time

Hand ongoing ops to ai-lead-ops after plan approval.

When to load references

• Baseline and attribution → references/token_audit.md
• Backlog template and scoring → references/improvement_backlog.md
• Technique catalog → references/optimization_playbook.md
• KPIs and eval gates → references/measurement_and_kpis.md
• Phasing and rollback → references/rollout_plan.md

Improvement backlog

Table of contents

1. Initiative card 2. Scoring 3. Dependencies 4. Anti-patterns

Initiative card

### [ID] Title
**Category:** prompt | context | model | RAG | agent | infra
**Owner:**
**Problem:** [which driver from audit]
**Change:** [one paragraph]
**Assumptions:** e.g. 40% of input is duplicate system text
**Est. savings:** X% tokens or $Y/mo (range low–high)
**Effort:** S | M | L
**Quality risk:** Low | Med | High
**Evals required:** [golden-ids], safety subset
**Metrics:** primary KPI + guardrail KPI
**Rollback:** [trigger and action]
**Status:** proposed | in progress | shipped | reverted

Scoring

Priority score (higher = do first):

score = (impact_1to5 * confidence_1to5) / effort_1to5

Cap in-flight high-risk (quality med+) initiatives to avoid eval thrash.

Dependencies

Common chains:

• Instrumentation (phase 0) before attribution-based cuts
• Prompt version registry before A/B cost tests
• Eval harness before model downgrade
• Context compression after golden eval baseline frozen

Draw dependency graph for phase 2+ items.

Anti-patterns

• Cutting output max_tokens without format eval → broken JSON
• Removing safety instructions to save tokens → ai-redteam regression
• Optimizing staging only while prod prompts diverge
• Savings claimed without before/after measurement window

Measurement and KPIs

Table of contents

1. Primary KPIs 2. Guardrail KPIs 3. Eval gates 4. Reporting cadence

Primary KPIs

Segment all KPIs by feature and tier—global averages hide regressions.

Guardrail KPIs

Do not worsen beyond threshold:

Define thresholds with product owner before phase 1 ships.

Eval gates

Pre-merge (per initiative):

• [ ] Golden set run on old vs new (same model unless initiative is routing)
• [ ] Diff report for failures (categorize: acceptable vs blocker)
• [ ] Token/count comparison on same fixture set

Pre-prod:

• [ ] Canary 5–10% traffic or internal dogfood 48h+
• [ ] Cost dashboard shows expected delta
• [ ] No guardrail breach

Post-ship (7d):

• [ ] Sustained savings vs baseline window
• [ ] No support spike tagged to AI quality

Link safety runs to ai-redteam for tier-2+ surfaces.

Reporting cadence

Use same baseline window for before/after comparisons (avoid holiday skew).

Optimization playbook

Table of contents

1. Quick wins 2. Prompt and context 3. Model and routing 4. RAG and tools 5. Agents 6. Trade-off summary

Implement via ai-context-engineer, prompt-engineer, ai-engineer—this file is for planning only.

Quick wins

Prompt and context

Detail: ai-context-engineer references.

Model and routing

Always pair with eval matrix per route.

RAG and tools

Agents

Trade-off summary

Document per initiative:

• What we give up (quality dimension)
• Who accepts risk (product, safety, eng)
• How we detect regression (metric + threshold)

Never optimize tokens in isolation from latency and success rate.

Rollout plan

Table of contents

1. Phase template 2. Rollout patterns 3. Rollback 4. Communication

Phase template

## Phase N — [Name]
**Goal:**
**Exit criteria:**
**Initiatives:** [IDs from backlog]
**Owners:**
**Timeline:**
**Budget impact (est.):**
**Eval window:**

### Week-by-week
| Week | Deliverable |
|---|---|
| 1 | |

Phase 0 (instrumentation) is mandatory unless attribution already exists.

Rollout patterns

Pair with deployment-strategist for customer-visible behavior changes.

Rollback

Per initiative define:

Keep previous prompt + model pins for 30 days minimum.

Communication

After plan delivery, transfer steady-state to ai-lead-ops cost review ritual.

Token audit

Table of contents

1. Data sources 2. Attribution dimensions 3. Baseline worksheet 4. Driver analysis

Data sources

Reconcile billing $ to token counts monthly; flag missing instrumentation early.

Attribution dimensions

Minimum viable tags on every LLM call:

• feature or product_surface
• model_id
• environment (prod/staging)
• tenant_id or tier (if multi-tenant)
• prompt_version / agent_version
• input_tokens, output_tokens, cached_tokens (if provider supports)

Optional: turn_index, tool_name, retrieval_chunk_count.

Baseline worksheet

| Segment | Sessions/wk | Avg in tok | Avg out tok | $/session | % of total $ |
|---|---|---|---|---|---|
| Feature A | | | | | |

Compute:

cost_per_session = (in * price_in + out * price_out + cached_discount) / sessions

Note growth rate week-over-week (usage vs token inflation).

Driver analysis

For top segment by spend, sample 20–50 traces:

1. Token breakdown: system vs tools vs RAG vs history vs user 2. Longest single block (often fixable) 3. Agent loop count and failed retries 4. Model tier vs task complexity mismatch

Document findings as evidence-backed drivers, not guesses.

If traces unavailable, run shadow logging for 48h before finalizing plan.