Cost Verification Auditor

Cost Verification Auditor

Verify that token cost estimates are within ±20% of actual Claude API usage.

When to Use

✅ Use for:

• Validating token estimation systems after implementation
• Pre-deployment cost accuracy checks
• Debugging unexpected API bills
• Periodic estimation drift detection

❌ NOT for:

• Looking up model pricing (use pricing docs)
• Budget planning or forecasting
• Cost optimization strategies
• Comparing models by price

Core Audit Process

Decision Tree

Has estimator? ──No──→ Build estimator first (see Calibration Guidelines)
      │
     Yes
      ↓
Define 3+ test cases (simple/medium/complex)
      ↓
Estimate BEFORE execution (no peeking!)
      ↓
Execute against real API
      ↓
Calculate variance: (actual - estimated) / estimated
      ↓
Variance ≤ ±20%? ──Yes──→ PASS ✓
      │
     No
      ↓
Apply fixes from Anti-Patterns section
      ↓
Re-run verification

Variance Formula

const inputVariance = (actual.inputTokens - estimate.inputTokens) / estimate.inputTokens;
const outputVariance = (actual.outputTokens - estimate.outputTokens) / estimate.outputTokens;
const costVariance = (actual.totalCost - estimate.totalCost) / estimate.totalCost;

// PASS if both input AND output within ±20%
const passed = Math.abs(inputVariance) <= 0.20 && Math.abs(outputVariance) <= 0.20;

Common Anti-Patterns

Anti-Pattern: The 500-Token Overhead Myth

Novice thinking: "Claude Code adds ~500 tokens overhead, so add that to every estimate."

Reality: Direct API calls have ~10 token overhead. The 500+ overhead is ONLY when using Claude Code's full context (system prompts, tools, conversation history).

Timeline:

• Pre-2025: Many tutorials used 500+ token estimates
• 2025+: Direct API overhead is minimal (~10 tokens)

What to use instead:

How to detect: Consistent 40-90% overestimation = overhead too high.

---

Anti-Pattern: Per-Node Accuracy Obsession

Novice thinking: "Every node must be within ±20% or the estimator is broken."

Reality: LLM output length is non-deterministic. Per-node output variance of 30-50% is normal. What matters is aggregate cost accuracy.

What to use instead:

• Focus on total DAG cost variance (should be ±20%)
• Accept per-node output variance up to ±40%
• Use constrained prompts ("list exactly 3") to reduce variance

How to detect: Input estimates accurate, output varies wildly = normal LLM behavior.

---

Anti-Pattern: Peeking Before Estimating

Novice thinking: "Let me run the API call first to see what tokens we get, then build the estimator."

Reality: This produces perfectly-fitted estimates that fail on new prompts. Estimation must happen BEFORE execution.

Correct approach: 1. Estimate based on prompt length and heuristics 2. Execute API call 3. Compare variance 4. Adjust heuristics if needed

Calibration Guidelines

Input Token Estimation

// Calibrated 2026-01-30
const inputTokens = Math.ceil(prompt.length / CHARS_PER_TOKEN) + OVERHEAD;

Output Token Estimation

Always: Minimum 100 output tokens for any meaningful response.

Model Behavior

Quick Fixes

Verification Checklist

• [ ] 3+ test cases (simple, medium, complex)
• [ ] Estimates run BEFORE API calls
• [ ] Variance formula: (actual - estimated) / estimated
• [ ] Target: ±20% for input AND output
• [ ] Report includes actionable recommendations

References

See /references/calibration-data.md for detailed calibration tables and historical data.

Calibration Data

Historical calibration data from actual API verification tests.

Calibration Run: 2026-01-30

Model: claude-haiku-4-5-20251001 Target Accuracy: ±20% Result: PASS (8.6% total cost variance)

Test Results

Key Findings

1. Input estimation: Overestimated by 26% on average 2. Output estimation: Underestimated by 8% on average 3. Total cost: Within target at 8.6% variance 4. Per-node variance: Higher than aggregate (expected due to LLM non-determinism)

Calibration Parameters Used

const CHARS_PER_TOKEN = 3.5;  // Balanced for mixed text/code
const OVERHEAD_TOKENS = 10;    // Direct API call, no system prompt
const OUTPUT_MULTIPLIER = 1.2; // With 100 token minimum

Prompt Characteristics

Recommendations from This Run

• Input estimation is acceptable (within 20% aggregate)
• Output estimation benefits from constrained prompts
• Focus on total cost accuracy, not per-node accuracy
• Consider increasing output multiplier for unconstrained prompts

Historical Trends

Model Pricing (2026-01)

Next Calibration

Schedule quarterly recalibration to detect drift:

• [ ] 2026-Q2: Re-run with updated prompts
• [ ] 2026-Q3: Check against new model releases
• [ ] 2026-Q4: Annual review