Ai Llm Inference

LLMOps - Inference & Optimization - Production Skill Hub

Modern Best Practices (January 2026):

• Treat inference as a systems problem: SLOs, tail latency, retries, overload, and cache strategy.
• Use continuous batching / smart scheduling when serving many concurrent requests (Orca scheduling: https://www.usenix.org/conference/osdi22/presentation/yu).
• Use KV-cache aware serving (PagedAttention/vLLM: https://arxiv.org/abs/2309.06180) and efficient attention kernels (FlashAttention: https://arxiv.org/abs/2205.14135).
• Use speculative decoding when latency is critical and draft-model quality is acceptable (speculative decoding: https://arxiv.org/abs/2302.01318).
• Quantize only with measured quality impact and rollback plan (quantization must be validated on your eval set).

This skill provides production-ready operational patterns for optimizing LLM inference performance, cost, and reliability. It centralizes decision rules, optimization strategies, configuration templates, and operational checklists for inference workloads.

No theory. No narrative. Only what Codex can execute.

---

When to Use This Skill

Codex should activate this skill whenever the user asks for:

• Optimizing LLM inference latency or throughput
• Choosing quantization strategies (FP8/FP4/INT8/INT4)
• Configuring vLLM, TensorRT-LLM, or DeepSpeed inference
• Scaling LLM inference across GPUs (tensor/pipeline parallelism)
• Building high-throughput LLM APIs
• Improving context window performance (KV cache optimization)
• Using speculative decoding for faster generation
• Reducing cost per token
• Profiling and benchmarking inference workloads
• Planning infrastructure capacity
• CPU/edge deployment patterns
• High availability and resilience patterns

Scope Boundaries (Use These Skills for Depth)

• Prompting, tuning, datasets -> ai-llm
• RAG pipeline construction -> ai-rag
• Deployment, APIs, monitoring -> ai-mlops
• Safety, governance -> ai-mlops
• Performance monitoring -> qa-observability
• Infrastructure operations -> ops-devops-platform

---

Quick Reference

---

Decision Tree: Inference Optimization Strategy

Need to optimize LLM inference: [Optimization Path]
    │
    ├─ High throughput (>10k tok/s) OR P99 variance > 3x P50?
    │   └─ YES -> Disaggregated inference (prefill/decode separation)
    │            See references/disaggregated-inference.md
    │
    ├─ Primary constraint: Throughput?
    │   ├─ Many concurrent users? -> batching + KV-cache aware serving + admission control
    │   ├─ Chat/agents with KV reuse? -> SGLang (RadixAttention)
    │   └─ Mostly batch/offline? -> batch inference jobs + large batches + spot capacity
    │
    ├─ Primary constraint: Cost?
    │   ├─ Can accept lower quality tier? -> model tiering (small/medium/large router)
    │   └─ Must keep quality? -> caching + prompt/context reduction before quantization
    │
    ├─ Primary constraint: Latency?
    │   ├─ Draft model acceptable? -> speculative decoding
    │   └─ Long context? -> prefill optimizations + FlashAttention-3 + context budgets
    │
    ├─ Large model (>70B)?
    │   ├─ Multiple GPUs? -> Tensor parallelism (NVLink required)
    │   └─ Deep model? -> Pipeline parallelism (minimize bubbles)
    │
    ├─ Hardware selection?
    │   ├─ Memory-bound? -> more HBM, higher bandwidth
    │   ├─ Latency-bound? -> faster clocks + kernel support
    │   └─ Multi-node? -> prioritize interconnect (NVLink/RDMA) and topology
    │
    │   Notes: treat GPU/SKU advice as time-sensitive; verify with vendor docs and your own benchmarks.
    │   See references/gpu-optimization-checklists.md and references/infrastructure-tuning.md
    │
    └─ Edge deployment?
        └─ CPU + quantization -> llama.cpp/GGUF for constrained resources

---

Intake Checklist (REQUIRED)

Before recommending changes, collect (or infer) these inputs:

• Model + variant (size, context length, precision/quantization, tokenizer)
• Traffic shape (prompt/output length distributions, concurrency, QPS, streaming vs non-streaming)
• SLOs and budgets (TTFT/ITL/total latency targets, error budget, cost per request)
• Serving stack (engine/version, batching/scheduling settings, caching, parallelism, autoscaling)
• Hardware and topology (GPU type/count, VRAM, NVLink/RDMA, CPU/RAM, storage, cluster/runtime)
• Constraints (quality floor, safety requirements, rollout/rollback constraints)

Core Concepts & Practices

Core Concepts (Vendor-Agnostic)

• Latency components: queueing + prefill + decode; optimize the largest contributor first.
• Tail latency: p99 is dominated by queuing and long prompts; fix with admission control and context budgets.
• Retries: retries can multiply load; bound retries and use hedged requests only with strict budgets.
• Caching: prefix caching helps repeated system/tool scaffolds; response caching helps repeated questions (requires invalidation).
• Security & privacy: prompts/outputs can contain sensitive data; scrub logs, enforce auth/tenancy, and rate-limit abuse (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

Implementation Practices (Tooling Examples)

• Measure under load: benchmark TTFT/ITL and p95/p99 with realistic concurrency and prompt lengths.
• Separate environments: dev/stage/prod model configs; promote only after passing the inference review checklist.
• Export telemetry: request-level tokens, TTFT/ITL, queue depth, GPU memory headroom, and error classes (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).

Do / Avoid

Do

• Do enforce max_input_tokens and max_output_tokens at the API boundary.
• Do cap concurrency and queue depth; return overload errors quickly.
• Do validate quality after any quantization or kernel change.

Avoid

• Avoid unbounded retries (amplifies outages).
• Avoid unbounded context windows (OOM + latency spikes).
• Avoid benchmarking on single requests; always test with realistic concurrency.

---

Accuracy Protocol (REQUIRED)

• Treat performance ratios (for example, "2x faster") as hypotheses unless a source is cited and the workload is comparable.
• Do not recommend hardware/SKU changes without stating assumptions (model size, context length, concurrency, interconnect).
• Prefer a measured baseline + checklist-driven rollout over "best practice" claims.

---

Resources (Detailed Operational Guides)

For comprehensive guides on specific topics, see:

Infrastructure & Serving

• Disaggregated Inference - Prefill/decode separation (2025+ standard)
• Infrastructure Tuning - OS, container, Kubernetes optimization for GPU workloads
• Serving Architectures - Production serving stack patterns (vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo)
• Resilience & HA Patterns - Multi-region, failover, traffic management

Performance Optimization

• Quantization Patterns - FP8/FP4/INT8/INT4 decision trees (FP8 first, INT8 not on Blackwell)
• KV Cache Optimization - PagedAttention, FlashAttention-3, FlashInfer, RadixAttention
• Parallelism Patterns - Tensor/pipeline/expert parallelism strategies
• Optimization Strategies - Throughput, cost, memory optimization
• Batching & Scheduling - Continuous batching and throughput patterns

Deployment & Operations

• Edge & CPU Optimization - llama.cpp, GGUF, mobile/browser deployment
• GPU Optimization Checklists - Hardware-specific tuning
• Speculative Decoding Guide - Advanced generation acceleration
• Profiling & Capacity Planning - Benchmarking, SLOs, replica sizing

Cost & Routing

• Cost Optimization Patterns - Token budgets, caching economics, model tiering, cost-per-outcome tracking
• Multi-Model Routing - Router architectures, quality-cost tradeoffs, cascading strategies, A/B routing
• Streaming Patterns - SSE/WebSocket serving, token-by-token delivery, backpressure, client integration

---

Templates

Inference Configs

Production-ready configuration templates for leading inference engines:

• vLLM Configuration - Continuous batching, PagedAttention setup
• TensorRT-LLM Configuration - NVIDIA kernel optimizations
• DeepSpeed Inference - PyTorch-friendly inference

Quantization & Compression

Model compression templates for reducing memory and cost:

• GPTQ Quantization - GPU post-training quantization
• AWQ Quantization - Activation-aware weight quantization
• GGUF Format - CPU/edge optimized formats

Serving Pipelines

High-throughput serving architectures:

• LLM API Server - FastAPI + vLLM production setup
• High-Throughput Setup - Multi-replica scaling patterns

Caching & Batching

Performance optimization templates:

• Prefix Caching - KV cache reuse strategies
• Batching Configuration - Continuous batching tuning

Benchmarking

Performance measurement and validation:

• Latency & Throughput Testing - Load testing framework

Checklists

• Inference Performance Review Checklist - Baseline, bottlenecks, rollout readiness

Navigation

Resources

• references/disaggregated-inference.md
• references/serving-architectures.md
• references/profiling-and-capacity-planning.md
• references/gpu-optimization-checklists.md
• references/speculative-decoding-guide.md
• references/resilience-ha-patterns.md
• references/optimization-strategies.md
• references/kv-cache-optimization.md
• references/batching-and-scheduling.md
• references/quantization-patterns.md
• references/parallelism-patterns.md
• references/edge-cpu-optimization.md
• references/infrastructure-tuning.md
• references/cost-optimization-patterns.md
• references/multi-model-routing.md
• references/streaming-patterns.md

Templates

• assets/serving/template-llm-api.md
• assets/serving/template-high-throughput-setup.md
• assets/inference/template-vllm-config.md
• assets/inference/template-tensorrtllm-config.md
• assets/inference/template-deepspeed-inference.md
• assets/quantization/template-awq.md
• assets/quantization/template-gptq.md
• assets/quantization/template-gguf.md
• assets/batching/template-batching-config.md
• assets/caching/template-prefix-caching.md
• assets/benchmarking/template-latency-throughput-test.md
• assets/checklists/inference-review-checklist.md

Data

• data/sources.json - Curated external references

---

Trend Awareness Protocol

IMPORTANT: When users ask recommendation questions about LLM inference, you MUST use WebSearch to check current trends before answering.

Trigger Conditions

• "What's the best inference engine for [use case]?"
• "What should I use for [serving/quantization/batching]?"
• "What's the latest in LLM inference optimization?"
• "Current best practices for [vLLM/TensorRT/quantization]?"
• "Is [inference tool] still relevant in 2026?"
• "[vLLM] vs [TensorRT-LLM] vs [SGLang]?"
• "Best quantization method for [model size]?"
• "What GPU should I use for inference?"

Required Searches

1. Search: "LLM inference optimization best practices 2026" 2. Search: "[vLLM/TensorRT-LLM/SGLang] comparison 2026" 3. Search: "LLM quantization trends January 2026" 4. Search: "LLM serving new releases 2026"

What to Report

After searching, provide:

• Current landscape: What serving engines are popular NOW (not 6 months ago)
• Emerging trends: New inference optimizations gaining traction
• Deprecated/declining: Techniques or tools losing relevance
• Recommendation: Based on fresh data, not just static knowledge

Example Topics (verify with fresh search)

• Inference engines (vLLM 0.7+, TensorRT-LLM, SGLang, llama.cpp)
• Quantization methods (FP8, AWQ, GPTQ, GGUF, bitsandbytes)
• Attention kernels (FlashAttention-3, FlashInfer, xFormers)
• Speculative decoding advances
• KV cache optimization techniques
• New GPU architectures (H200, Blackwell) and their optimizations

---

Related Skills

This skill focuses on inference-time performance. For related workflows:

• See "Scope Boundaries" above.

---

External Resources

See data/sources.json for:

• Serving frameworks (vLLM, TensorRT-LLM, DeepSpeed-MII)
• Quantization libraries (GPTQ, AWQ, bitsandbytes, LLM Compressor)
• FlashAttention, FlashInfer, xFormers
• GPU hardware guides and optimization docs
• Benchmarking frameworks and tools

---

Use this skill whenever the user needs LLM inference performance, cost reduction, or serving architecture guidance.

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Batching Configuration Template

Tune batch sizes and batching behavior for optimized throughput.

---

1. Batching Mode

mode: "continuous" # recommended max_batch_size: 32 max_tokens: 2048

---

2. Scheduler

scheduler: policy: "token" # token-level scheduling preemption: true # for priority traffic

---

3. Priority Classes

priority: high: ["chat", "search"] low: ["background"]

---

4. Monitoring

Track:

• Batch size distribution
• Token throughput
• Latency-per-token metrics

---

5. Checklist

• [ ] Batch size tuned
• [ ] No starvation
• [ ] Meets P95 latency requirements

Latency & Throughput Benchmark Template

Defines a reproducible load-testing process for LLM inference.

---

1. Test Parameters

model: "<model_id>" concurrency_levels: [1, 4, 8, 16, 32] request_rate: "<req/s>" prompt_length: <tokens> max_output_tokens: <tokens>

---

2. Test Scenarios

• Short prompts (≤50 tokens)
• Medium prompts (100–500 tokens)
• Long-context prompts (2k–10k tokens)
• RAG prompts with large context blocks

---

3. Metrics to Collect

Latency

• Mean
• P50, P95, P99

Throughput

• req/s
• tokens/s

GPU Metrics

• GPU utilization
• Memory usage
• Kernel time

---

4. Pass Criteria

• P95 latency < <threshold>
• No OOM
• Throughput ≥ baseline
• No format regressions (JSON tasks)

---

5. Checklist

• [ ] Warmup period excluded
• [ ] Stable iteration count
• [ ] Metrics logged

Prefix and Prompt Caching Template

Caches repeated prefixes or prompts to reduce compute pressure.

---

1. Cache Types

Prefix Cache

• Cache prefill for repeated prompt beginnings
• Great for many users sharing similar instructions

Response Cache (Semantic)

• Cache full responses for identical or semantically close queries

---

2. Cache Config

cache: enabled: true type: "prefix" max_entries: 5000 eviction: "lru"

---

3. Cache Keys

• Hash of normalized prompt prefix
• Model version
• Context window settings

---

4. Validation Checklist

• [ ] Prefix detection correct
• [ ] No stale cache entries
• [ ] Cache hit-rate tracked

Inference Performance Review Checklist

Purpose: Verify production readiness, identify optimizations, document baseline performance.

---

Template Contract

Goals

• Establish a reproducible performance baseline under realistic load.
• Identify bottlenecks and select safe optimizations with rollback plan.
• Ensure reliability, cost control, and observability are production-ready.

Inputs

• Model artifact + config (context limits, decoding settings, quantization).
• Traffic shape: prompt length distribution, output length, concurrency, QPS.
• Hardware and deployment topology.
• SLOs/budgets: latency, availability, cost per request.

Decisions

• Serving configuration (batching, scheduling, caching, parallelism).
• Optimization plan (quantization, kernel changes, speculative decoding).
• Capacity plan (replicas, autoscaling, queue caps) and rollback triggers.

Risks

• Tail-latency explosions from queueing and long prompts.
• Quality regressions from quantization/optimizations.
• Cascading retries and overload amplification.
• OOM and instability due to unbounded context or concurrency.

Metrics

• TTFT/ITL and total latency p50/p95/p99 under load.
• Throughput (req/s, tok/s), GPU/CPU utilization, memory headroom.
• Error rate by class and cost per request.

1. Configuration

Model Details

• Model name: _______________
• Model size: ___B parameters
• Original precision: [ ] FP32 [ ] FP16 [ ] BF16 [ ] FP8

Hardware

• GPU type: _______________
• GPU count: ___
• GPU memory: ___GB each
• Total VRAM: ___GB

Framework

• Inference engine: _______________ (example: open-source server, managed endpoint)
• Version: _______________

Quantization

• Current: [ ] None [ ] FP8 [ ] INT8 [ ] INT4 [ ] AWQ [ ] GPTQ [ ] GGUF
• KV cache dtype: [ ] FP16 [ ] FP8 [ ] INT8

---

2. Performance Baseline

Latency (measure with realistic load)

Throughput

Resource Utilization

---

3. Optimization Checklist

Quantization

Accuracy validation post-quantization: [ ] Completed [ ] Pending

Batching

Attention Optimization

Caching

Parallelism

Advanced

---

4. Cost Analysis

Cost Optimization Actions

• [ ] _______________
• [ ] _______________
• [ ] _______________

---

5. Reliability

---

6. Monitoring

---

7. Anti-Patterns Check

---

8. Decisions & Actions

Optimizations to Implement

1. _______________ 2. _______________ 3. _______________

Re-benchmark Date

• Next review: _______________
• Owner: _______________

Sign-Off

• [ ] ML Engineer: _______________ Date: ___
• [ ] Platform Engineer: _______________ Date: ___

DeepSpeed Inference Template

Configuration for serving LLMs with DeepSpeed-MII or DeepSpeed-Inference.

---

1. Environment

ds_version: "0.16+" torch_version: "2.9+"

---

2. Model Settings

model: name: "<model_path>" dtype: "bf16" tensor_parallel: size: <num_gpus> quantization: enable: false

---

3. Runtime Settings

runtime: max_context: <max_tokens> enable_kv_cache: true kv_cache_quantization: "int8" use_cuda_graph: true

---

4. Launch Config

launch: hostfile: "./hostfile" deepspeed_port: 29500 num_nodes: <nodes>

---

5. Checklist

• [ ] CUDA graph enabled
• [ ] Multi-GPU tested
• [ ] KV offloading configured when needed

TensorRT-LLM Inference Config Template

Optimized configuration for high-performance, low-latency inference on NVIDIA GPUs.

---

1. Model

model: "<hf_model_path>" precision: "fp8" # or fp16/int8 tensor_parallel_size: <num_gpus> pipeline_parallel_size: 1

---

2. Engine Settings

engine: max_batch_size: 32 max_input_len: <len> max_output_len: <len> enable_refit: false use_parallel_embeddings: true

---

3. Attention Optimization

attention: type: "flash" # flash | paged | default kv_cache_quant: "int8"

---

4. Kernel Optimization

kernels: enable_fused_mlp: true enable_sparsity: false

---

5. Serving

server: grpc_port: 9000 http_port: 8000 num_workers: 4

---

6. Checklist

• [ ] Engine built successfully
• [ ] Precision validated
• [ ] Latency measured on representative inputs

vLLM Inference Configuration Template

Complete config for deploying an LLM using vLLM with continuous batching.

---

1. Model

model: "<model_name_or_path>" dtype: "bfloat16" tokenizer: "<tokenizer_path>" trust_remote_code: false tensor_parallel_size: <num_gpus>

---

2. Runtime

runtime: max_num_seqs: 512 max_model_len: <max_context> # e.g. 4096, 8192, 32768 gpu_memory_utilization: 0.90 disable_log_requests: true

---

3. Batching (Continuous)

batching: enable: true max_tokens: <value> # tune dynamically scheduler_policy: "token"

---

4. Logging

logging: level: "info" log_metrics: true log_requests: false

---

5. Server

server: host: "0.0.0.0" port: 8000 enable_cors: true

---

6. Health Checks

• Startup: ensure model loads
• Liveness: GPU accessible
• Readiness: warm cache populated

---

7. Checklist

• [ ] vLLM installed
• [ ] GPU memory fits model
• [ ] Batching load-tested
• [ ] Token throughput measured

AWQ Quantization Template (Activation-Aware)

AWQ delivers high-quality weight-only quantization with low accuracy loss.

---

1. Calibration

calibration_set: size: 256 source: "<path>"

---

2. AWQ Parameters

awq: alpha: 0.5 clip: true symmetric: false quant_group_size: 128

---

3. Output

output: path: "<target_dir>"

---

4. Checklist

• [ ] No NaNs after quantization
• [ ] KV cache tested
• [ ] Matching tokenizer used

AWQ Quantization Template (Activation-Aware)

AWQ delivers high-quality weight-only quantization with low accuracy loss.

---

1. Calibration

calibration_set: size: 256 source: "<path>"

---

2. AWQ Parameters

awq: alpha: 0.5 clip: true symmetric: false quant_group_size: 128

---

3. Output

output: path: "<target_dir>"

---

4. Checklist

• [ ] No NaNs after quantization
• [ ] KV cache tested
• [ ] Matching tokenizer used

GPTQ Quantization Template (4-bit)

Use GPTQ to quantize large LLMs for GPU inference.

---

1. Calibration Data

calibration: num_samples: 256 max_seq_length: 2048 dataset_path: "<path/to/calibration/text>"

---

2. GPTQ Settings

gptq: bits: 4 damp_percent: 0.01 desc_act: true blocksize: 128 groupsize: 128 act_order: true

---

3. Output

output: quantized_model_path: "<output_dir>"

---

4. Validation Checklist

• [ ] Quantized model loads
• [ ] Per-layer error acceptable
• [ ] Output quality tested on eval set

High Throughput Serving Setup (vLLM/TensorRT-LLM)

Configuration template for large-scale production serving.

---

1. Infra Setup

• GPU class: <A100/H100/L40>
• Replicas: <N>
• Autoscaling rules:
• CPU/GPU > 70%
• Queue length > threshold

---

2. Runtime Settings

runtime: batching: continuous kv_cache: "fp16" max_context: <max_tokens> prefill_optimization: true

---

3. Load Balancing

• Sticky routing optional
• Prefer weighted round-robin
• Health checks every 5 seconds

---

4. Observability

Dashboards must include:

• Token throughput
• Latency percentiles
• GPU utilization
• Request queue depth
• KV cache hit ratio

---

5. Failure Handling

• Fallback to smaller model
• Reject long inputs
• Autoscale aggressively during surges

LLM API Template

A production-ready API for LLM inference.

---

1. Endpoint

POST /v1/generate

---

2. Request Body

{ "prompt": "<text>", "max_tokens": 256, "temperature": 0.4, "top_p": 0.9, "model": "<model_id>" }

---

3. Response Body

{ "output": "<generated_text>", "model_version": "<vX.Y>", "tokens": { "input": <count>, "output": <count> }, "latency_ms": <value> }

---

4. Reliability Mechanisms

• Timeout
• Circuit breaker
• Request batching
• Static/dynamic routing

---

5. Observability

• Log request_id
• Track token usage
• Emit latency histograms

---

6. Security

• API key required
• Rate limiting
• Input sanitization

Batching & Scheduling for LLM Inference

Batching is the #1 source of throughput improvement. These patterns ensure efficient GPU utilization.

---

1. Continuous Batching (vLLM Pattern)

Behavior

• Incoming requests join ongoing batch
• No batch boundary waits
• Token-level scheduling

Benefits

• Massive QPS gains
• Low latency growth
• Excellent for chatbots and APIs

Checklist

• [ ] Continuous batching enabled
• [ ] Max batch size tuned
• [ ] Scheduling policy tested

---

2. Static Batching

Use for predictable batch workloads such as:

• Offline summarization
• Large dataset scoring
• Periodic batch jobs

Notes:

• Largest gains when input length similar across items
• Use fixed batch sizes (e.g., 16, 32, 64)

---

3. Priority Scheduling

Use when

• Mixture of real-time and low-priority jobs

Strategy

• Assign request classes
• Allow preemption
• Ensure latency-sensitive jobs bypass batch queue

Checklist

• [ ] Priority classes defined
• [ ] Preemption rules tested
• [ ] No starvation of low-priority requests

---

4. Token-by-Token Scheduling

Enabled by vLLM and specialized runtimes.

Benefits

• Tiny idle gaps between tokens
• Better time slicing
• Smooth multi-request performance

---

5. Batching Tuning Guidelines

• Start with batch size 8 → scale until GPU saturates
• Monitor:
• GPU utilization
• Token throughput (tok/s)
• Latency at P95

Checklist

• [ ] Throughput measured
• [ ] Latency budget respected
• [ ] Batch size auto-tuning scripted

Inference Cost Optimization Patterns

Operational reference for managing LLM inference costs — token economics, prompt caching, model routing, GPU strategies, cost monitoring, and FinOps practices for AI workloads.

Freshness anchor: January 2026 — covers OpenAI Batch API, Anthropic prompt caching, Google context caching, AWS Bedrock pricing, and current token pricing across all major providers.

---

Token Pricing Quick Reference (January 2026)

API Providers

Self-Hosted (GPU Cost Basis)

---

Cost Optimization Decision Tree

Monthly inference spend analysis
│
├── >80% of cost is one use case?
│   ├── YES → Optimize that use case specifically
│   │   ├── Can a smaller model handle it? → Model downgrade
│   │   ├── Are prompts repetitive? → Prompt caching
│   │   ├── Is it latency-insensitive? → Batch API
│   │   └── High volume? → Self-hosted or reserved capacity
│   └── NO → Broad optimization
│       ├── Implement model routing (simple→small, complex→large)
│       ├── Add prompt caching across all endpoints
│       └── Audit for wasted tokens (verbose prompts, unused context)
│
├── Cost growing faster than usage?
│   ├── Context window bloat → Implement conversation summarization
│   ├── Retry storms → Fix error handling, add circuit breakers
│   ├── Unnecessary re-processing → Add response caching
│   └── Image/multimodal costs → Resize images, batch processing
│
└── Need to hit specific cost target?
    ├── Calculate current $/request
    ├── Identify cheapest model that meets quality bar
    ├── Apply caching (typically 30-50% reduction)
    ├── Apply routing (typically 40-60% reduction)
    └── Consider self-hosted at >$10K/month

---

Prompt Caching ROI

Provider Caching Features

Anthropic Prompt Caching Implementation

import anthropic

client = anthropic.Anthropic()

# Mark the system prompt for caching (one-time write cost, then 90% off)
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_SYSTEM_PROMPT,  # Must be >1024 tokens
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_message}]
)

# Check caching effectiveness
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
print(f"Cache write tokens: {response.usage.cache_creation_input_tokens}")

Caching ROI Calculator

When Caching Is Worth It

---

Model Routing for Cost

Simple Routing Strategy

class CostOptimizedRouter:
    """Route requests to cheapest model that meets quality requirements."""

    MODELS = {
        "simple": {
            "model": "gpt-4o-mini",
            "cost_per_1k_input": 0.00015,
            "cost_per_1k_output": 0.0006
        },
        "standard": {
            "model": "gpt-4o",
            "cost_per_1k_input": 0.0025,
            "cost_per_1k_output": 0.01
        },
        "complex": {
            "model": "o3-mini",
            "cost_per_1k_input": 0.0011,
            "cost_per_1k_output": 0.0044
        }
    }

    def route(self, request: str, task_type: str = None) -> str:
        if task_type:
            return self._route_by_task(task_type)
        return self._route_by_complexity(request)

    def _route_by_task(self, task_type: str) -> str:
        TASK_ROUTING = {
            "classification": "simple",
            "extraction": "simple",
            "summarization": "standard",
            "analysis": "standard",
            "reasoning": "complex",
            "code_generation": "standard",
            "creative_writing": "standard",
        }
        tier = TASK_ROUTING.get(task_type, "standard")
        return self.MODELS[tier]["model"]

    def _route_by_complexity(self, request: str) -> str:
        # Quick heuristics
        token_count = len(request.split())
        if token_count < 50:
            return self.MODELS["simple"]["model"]
        elif token_count < 500:
            return self.MODELS["standard"]["model"]
        else:
            return self.MODELS["complex"]["model"]

---

Batch vs Real-Time Cost Tradeoffs

OpenAI Batch API

Batch API Implementation

import json
from openai import OpenAI

client = OpenAI()

# Step 1: Create JSONL batch file
requests = []
for i, item in enumerate(data_to_process):
    requests.append({
        "custom_id": f"request-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "system", "content": "Extract entities."},
                {"role": "user", "content": item["text"]}
            ],
            "max_tokens": 500
        }
    })

# Write to JSONL
with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# Step 2: Upload and create batch
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=batch_file.id, endpoint="/v1/chat/completions", completion_window="24h")

# Step 3: Poll for completion
# batch.status: "validating" → "in_progress" → "completed"

When to Use Batch

---

GPU Strategy: Spot vs On-Demand vs Reserved

Decision Matrix

Spot Instance Strategy for Inference

Spot Checklist:
- [ ] Implement graceful shutdown (save state on 2-min warning)
- [ ] Use spot fleet across multiple instance types (A100, H100, A10G)
- [ ] Implement request queuing (drain queue before shutdown)
- [ ] Set up automatic failover to on-demand
- [ ] Use in regions with lower spot prices (us-east-2, eu-west-1)
- [ ] Never use spot for latency-sensitive, user-facing inference
- [ ] Ideal for: batch processing, eval runs, fine-tuning, dev/staging

---

Cost Monitoring and Alerting

Dashboard Metrics

Budget Alert Implementation

class CostMonitor:
    def __init__(self, daily_budget: float, alert_callback):
        self.daily_budget = daily_budget
        self.alert_callback = alert_callback
        self.daily_spend = 0.0
        self.alert_thresholds = [0.5, 0.8, 0.95, 1.0]
        self.alerts_sent = set()

    def record_cost(self, cost: float, metadata: dict):
        self.daily_spend += cost

        for threshold in self.alert_thresholds:
            if self.daily_spend >= self.daily_budget * threshold:
                if threshold not in self.alerts_sent:
                    self.alerts_sent.add(threshold)
                    self.alert_callback(
                        level="warning" if threshold < 1.0 else "critical",
                        message=f"Daily budget {threshold*100:.0f}% consumed: "
                                f"${self.daily_spend:.2f} / ${self.daily_budget:.2f}",
                        metadata=metadata
                    )

        # Hard stop at 120% of budget
        if self.daily_spend >= self.daily_budget * 1.2:
            raise BudgetExceededError(f"Hard budget limit reached: ${self.daily_spend:.2f}")

---

Token Reduction Techniques

Conversation Summarization Pattern

async def manage_conversation_context(messages: list, max_tokens: int = 4000):
    """Summarize old messages to stay within token budget."""
    total_tokens = count_tokens(messages)

    if total_tokens <= max_tokens:
        return messages

    # Keep system message and last 4 turns
    system = messages[0]
    recent = messages[-8:]  # last 4 turns (user + assistant)
    old = messages[1:-8]

    if not old:
        return messages

    # Summarize old messages
    summary = await llm.generate(
        model="gpt-4o-mini",  # cheap model for summarization
        messages=[{
            "role": "user",
            "content": f"Summarize this conversation concisely:\n{format_messages(old)}"
        }]
    )

    return [
        system,
        {"role": "user", "content": f"[Previous conversation summary: {summary}]"},
        {"role": "assistant", "content": "Understood, I have the context."},
        *recent
    ]

---

Anti-Patterns

---

Cross-References

• multi-model-routing.md — detailed routing architectures for cost optimization
• streaming-patterns.md — streaming to reduce perceived latency (not cost, but UX)
• ../ai-llm/references/model-migration-guide.md — migrating to cheaper models
• ../ai-llm/references/structured-output-patterns.md — reduce retries with structured output
• ../ai-prompt-engineering/references/prompt-testing-ci-cd.md — evaluate quality after cost optimization

Disaggregated Inference (Prefill/Decode Separation)

Production architecture for separating prefill and decode phases onto specialized compute pools. Now the dominant pattern in high-throughput LLM serving (2025+).

---

Why Disaggregation

LLM inference has two fundamentally different phases:

Problem: Colocating both phases causes interference:

• Prefill jobs starve decode latency
• Decode jobs leave compute underutilized during prefill
• P99 latency explodes under load

Solution: Separate prefill and decode onto dedicated GPU pools, scale independently.

---

Architecture Patterns

Basic P/D Separation

┌─────────────────┐     KV Cache      ┌─────────────────┐
│  Prefill Pool   │ ───────────────→  │  Decode Pool    │
│  (Compute-opt)  │    Transfer       │  (Memory-opt)   │
└─────────────────┘                   └─────────────────┘
        ↑                                     │
        │         ┌─────────────┐             │
        └─────────│   Router    │←────────────┘
                  └─────────────┘
                        ↑
                   User Requests

Production Implementation (vLLM)

vLLM implements disaggregation with two instances connected via KV cache connector:

# Prefill instance
vllm serve model --prefill-only \
    --kv-connector redis://kv-store:6379

# Decode instance
vllm serve model --decode-only \
    --kv-connector redis://kv-store:6379

NVIDIA Dynamo (GTC 2025)

Data center scale disaggregation framework supporting vLLM, SGLang, TensorRT-LLM:

# dynamo-config.yaml
prefill:
  replicas: 3
  gpu_type: h100
  optimization: compute
decode:
  replicas: 9
  gpu_type: h100
  optimization: memory_bandwidth
kv_transfer:
  protocol: rdma
  compression: none

---

Decision Tree: When to Disaggregate

Should you disaggregate prefill/decode?
│
├─ Throughput > 10k tokens/sec required?
│   └─ YES → Disaggregate
│
├─ P99 latency variance > 3x P50?
│   └─ YES → Disaggregate (queuing interference)
│
├─ Mixed workloads (short + long prompts)?
│   └─ YES → Disaggregate
│
├─ Single-user or batch-only workload?
│   └─ NO → Colocated may be simpler
│
├─ GPU count < 4?
│   └─ NO → Colocated (overhead not worth it)
│
└─ Fast interconnect available (NVLink, RDMA)?
    └─ NO → Colocated (KV transfer bottleneck)

---

Performance Benchmarks (2025)

SGLang on H100 (96 GPUs)

Configuration: 3 nodes (24 GPUs) prefill + 9 nodes (72 GPUs) decode

SGLang on GB200 NVL72 (September 2025)

vLLM with llm-d 0.3 (October 2025)

---

KV Cache Transfer Strategies

Transfer Protocols

Perplexity Implementation Pattern

┌──────────────┐
│ KV Messenger │ ← Orchestrates transfers
└──────┬───────┘
       │
  ┌────┴────┐
  ↓         ↓
Prefiller  Decoder
  │           │
  └───────────┘
    Transfer blocks
    until complete

Key: Decoder blocks scheduling until KV transfer completes.

---

When NOT to Disaggregate

Avoid disaggregation when:

1. Small workloads: <4 GPUs, overhead exceeds benefit 2. Batch-only inference: No latency sensitivity 3. Slow interconnect: KV transfer becomes bottleneck 4. Simple deployment: Operational complexity not justified

Performance degradation risk: 20-30% if misconfigured (overhead exceeds gains).

---

Configuration Templates

vLLM Disaggregated Setup

# Start KV store
redis-server --port 6379

# Prefill worker (compute-optimized)
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Llama-3-70B \
    --prefill-only \
    --tensor-parallel-size 2 \
    --max-num-seqs 256 \
    --kv-connector redis://localhost:6379

# Decode worker (memory-optimized)
CUDA_VISIBLE_DEVICES=2,3,4,5 vllm serve meta-llama/Llama-3-70B \
    --decode-only \
    --tensor-parallel-size 4 \
    --max-num-seqs 64 \
    --kv-connector redis://localhost:6379

SGLang Disaggregated Setup

# Prefill cluster
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3-70B \
    --tp 8 --dp 3 \
    --disagg-mode prefill \
    --kv-transfer-config kv_config.json

# Decode cluster
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3-70B \
    --tp 8 --dp 9 \
    --disagg-mode decode \
    --kv-transfer-config kv_config.json

---

Monitoring Disaggregated Systems

Key Metrics

Prometheus Queries

# KV transfer latency P99
histogram_quantile(0.99,
  rate(kv_transfer_latency_seconds_bucket[5m]))

# Prefill/decode throughput ratio
rate(prefill_tokens_total[5m]) / rate(decode_tokens_total[5m])

# Queue depth imbalance
prefill_queue_depth / decode_queue_depth

---

Framework Support Matrix

---

References

• DistServe: Disaggregating Prefill and Decoding (OSDI 2024)
• Hao AI Lab: Disaggregated Inference 18 Months Later
• vLLM Disaggregated Prefilling Docs
• Ray Serve Prefill/Decode Disaggregation
• Perplexity: Disaggregated Prefill and Decode

Edge & CPU Optimization for LLM Inference

Production patterns for running LLM inference on CPU-only environments, edge devices, and resource-constrained deployments.

Overview

When to Use CPU Inference:

• No GPU available (edge devices, embedded systems)
• Cost-sensitive deployments (CPU hours cheaper than GPU)
• Low-throughput workloads (< 10 QPS)
• Privacy-critical applications (on-device inference)

Performance Expectations:

• CPU inference: 5-50x slower than GPU (depends on model size)
• Quantization critical: INT4/INT8 required for acceptable performance
• Optimal for: <13B parameter models
• Not recommended for: Real-time, high-throughput, or large models (>30B)

---

CPU-Optimized Stacks

llama.cpp

Best for: General-purpose CPU inference, widest model support

Key features:

• GGUF format (optimized for CPU)
• Apple Silicon optimizations (Metal)
• SIMD acceleration (AVX2, AVX512, NEON)
• Minimal dependencies

Installation:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_OPENBLAS=1  # With OpenBLAS for faster matmul

Running inference:

./main \
  -m models/llama-2-7b.Q4_K_M.gguf \
  -n 256 \
  -t 8 \
  --temp 0.7 \
  --top-p 0.9 \
  -p "Write a poem about AI"

Python bindings:

from llama_cpp import Llama

llm = Llama(
    model_path="models/llama-2-7b.Q4_K_M.gguf",
    n_ctx=2048,  # Context window
    n_threads=8,  # CPU threads
    n_gpu_layers=0  # CPU only
)

output = llm(
    "Write a poem about AI",
    max_tokens=256,
    temperature=0.7
)
print(output['choices'][0]['text'])

GGML & GGUF Formats

GGUF = GGML Universal File Format (current standard as of 2024)

Quantization types (ordered by quality/size trade-off):

Download pre-quantized models:

# From HuggingFace
huggingface-cli download \
  TheBloke/Llama-2-7B-GGUF \
  llama-2-7b.Q4_K_M.gguf \
  --local-dir ./models

Quantize your own model:

# Convert HuggingFace → GGUF
python convert.py models/llama-2-7b-hf \
  --outtype q4_K_M \
  --outfile models/llama-2-7b.Q4_K_M.gguf

OpenVINO (Intel CPUs)

Best for: Intel Xeon, Core processors (optimized for Intel architecture)

Key features:

• Intel-specific optimizations (AVX512, AMX, VNNI)
• Model compression (INT8, INT4)
• Graph optimizations (fusion, constant folding)

Installation:

pip install openvino openvino-dev

Model optimization:

# Optimize HuggingFace model for CPU
optimum-cli export openvino \
  --model meta-llama/Llama-2-7b-hf \
  --task text-generation \
  --weight-format int4 \
  llama-2-7b-openvino

Running inference:

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model = OVModelForCausalLM.from_pretrained(
    "llama-2-7b-openvino",
    device="CPU"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

inputs = tokenizer("Write a poem", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

ONNX Runtime

Best for: Cross-platform deployment, cloud/edge consistency

Key features:

• Cross-platform (Windows, Linux, macOS, ARM)
• Hardware-agnostic optimizations
• Quantization support (INT8, INT4)

Model conversion:

# HuggingFace → ONNX
optimum-cli export onnx \
  --model meta-llama/Llama-2-7b-hf \
  --task text-generation-with-past \
  llama-2-7b-onnx

Quantization:

from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    "llama-2-7b-onnx/model.onnx",
    "llama-2-7b-onnx/model_int8.onnx",
    weight_type=QuantType.QInt8
)

Inference:

from optimum.onnxruntime import ORTModelForCausalLM

model = ORTModelForCausalLM.from_pretrained(
    "llama-2-7b-onnx",
    provider="CPUExecutionProvider"
)

MLC-LLM (Mobile/Edge)

Best for: Mobile devices (iOS, Android), WebAssembly

Key features:

• Mobile GPU support (Metal, Vulkan)
• WebGPU for browser deployment
• Extreme quantization (3-bit, 4-bit)

Installation:

pip install mlc-llm mlc-ai-nightly

Compile model for mobile:

mlc_llm compile \
  meta-llama/Llama-2-7b-hf \
  --target iphone \
  --quantization q4f16_1 \
  --output llama-2-7b-mobile

---

CPU Performance Optimization

Threading Configuration

Optimal thread count:

import os
import multiprocessing

# Rule of thumb: physical cores (not hyperthreads)
num_threads = multiprocessing.cpu_count() // 2

# llama.cpp
os.environ['OMP_NUM_THREADS'] = str(num_threads)

# ONNX Runtime
session_options = onnxruntime.SessionOptions()
session_options.intra_op_num_threads = num_threads
session_options.inter_op_num_threads = 1

Avoid over-threading:

• Using all threads can cause contention
• Physical cores > hyperthreads for inference
• Test different thread counts for your workload

SIMD Acceleration

Check CPU capabilities:

# Linux
lscpu | grep -i flags

# macOS
sysctl -a | grep machdep.cpu.features

Enable optimizations:

• AVX2: Standard on modern Intel/AMD (2013+)
• AVX512: Intel Xeon Scalable, Ice Lake+
• AMX: Intel Sapphire Rapids (4th gen Xeon)
• NEON: ARM processors

Build llama.cpp with optimizations:

# Auto-detect SIMD
make LLAMA_NATIVE=1

# Explicit AVX512
make LLAMA_AVX512=1

# Apple Silicon (Metal)
make LLAMA_METAL=1

BLAS Libraries (Matrix Multiplication)

OpenBLAS (general purpose):

make LLAMA_OPENBLAS=1

Intel MKL (Intel CPUs):

make LLAMA_MKL=1

Apple Accelerate (macOS):

make LLAMA_ACCELERATE=1

Performance impact: 2-5x speedup for matmul operations

Memory Bandwidth Optimization

Use mlock() to prevent swapping:

# llama.cpp
llm = Llama(
    model_path="model.gguf",
    use_mlock=True  # Lock model in RAM
)

Enable huge pages:

# Linux
echo 1024 > /proc/sys/vm/nr_hugepages

Batch size tuning:

# Larger batches = better throughput (up to memory limit)
llm = Llama(
    model_path="model.gguf",
    n_batch=512  # Default: 512, try 1024-2048
)

---

Edge Device Deployment

Raspberry Pi (ARM Cortex-A)

Recommended models:

• Llama-2-7B-Q3_K_M (3.6 GB) - minimum viable
• TinyLlama-1.1B-Q4_K_M (0.8 GB) - recommended
• Phi-2-Q4_K_M (1.6 GB) - best quality/size

Configuration:

# Build llama.cpp for ARM
make LLAMA_NO_ACCELERATE=1

# Run with limited resources
./main \
  -m TinyLlama-1.1B.Q4_K_M.gguf \
  -n 128 \
  -t 4 \
  --temp 0.7 \
  -c 1024  # Reduce context to save RAM

Performance: ~5-10 tokens/sec (TinyLlama on Pi 5)

Mobile Devices (iOS/Android)

iOS (Swift):

import MLCChat

let model = MLCEngine(
    modelPath: "llama-2-7b-q4f16",
    device: .metal  // Use GPU if available
)

model.generate(
    prompt: "Write a poem",
    maxTokens: 256
) { token in
    print(token, terminator: "")
}

Android (Kotlin):

import ai.mlc.mlcllm

val engine = MLCEngine(
    modelPath = "llama-2-7b-q4f16",
    device = Device.VULKAN
)

engine.generate("Write a poem", maxTokens = 256)

Browser (WebAssembly + WebGPU)

Using Transformers.js:

<script type="module">
  import { pipeline } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers';

  const generator = await pipeline(
    'text-generation',
    'Xenova/Llama-2-7b-chat-q4',
    { device: 'webgpu' }  // Falls back to WASM
  );

  const output = await generator('Write a poem', {
    max_new_tokens: 256
  });

  console.log(output[0].generated_text);
</script>

---

Benchmarking & Validation

Performance Testing

llama.cpp benchmark:

./main \
  -m model.gguf \
  -p "Test prompt" \
  -n 128 \
  -t 8 \
  --log-disable

# Output includes:
# - Prompt eval time (prefill)
# - Token generation time (decode)
# - Tokens per second

Expected performance (7B model, Q4_K_M):

Quality Validation

Test quantization impact:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Original model
model_fp16 = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16
)

# Test prompts
prompts = [
    "What is the capital of France?",
    "Write a Python function to calculate fibonacci",
    "Explain quantum computing in simple terms"
]

# Compare outputs (original vs quantized)
# Run both models and check:
# 1. Factual correctness
# 2. Coherence
# 3. Instruction following

Validation Checklist

• [ ] Quantization format selected (Q4_K_M recommended)
• [ ] Threading tuned (num_threads = physical cores)
• [ ] SIMD acceleration enabled (check compilation flags)
• [ ] BLAS library selected (OpenBLAS/MKL/Accelerate)
• [ ] Batch size optimized (test 512, 1024, 2048)
• [ ] Memory locked (use_mlock=True)
• [ ] Performance benchmarked (tokens/sec measured)
• [ ] Quality validated (compare vs FP16 baseline)
• [ ] Context window sized appropriately (reduce if memory-constrained)

---

Production Deployment Patterns

Serverless CPU Inference

AWS Lambda example:

# Lambda handler with llama.cpp
import ctypes
from llama_cpp import Llama

# Load model at cold start (outside handler)
llm = Llama(
    model_path="/opt/model.gguf",  # Lambda layer
    n_ctx=1024,
    n_threads=2,  # Lambda CPU limit
    use_mmap=True,
    use_mlock=False  # Lambda doesn't support mlock
)

def handler(event, context):
    prompt = event['prompt']
    output = llm(prompt, max_tokens=128)
    return {
        'statusCode': 200,
        'body': output['choices'][0]['text']
    }

Considerations:

• Cold start: 10-30s (model loading)
• Memory limit: 10GB max (use Q3/Q4 quantization)
• CPU: 2-6 vCPUs (limited parallelism)
• Cost: $0.0000166667/GB-sec

Docker Container

Dockerfile:

FROM ubuntu:22.04

# Install dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    libopenblas-dev \
    wget

# Build llama.cpp
WORKDIR /app
RUN git clone https://github.com/ggerganov/llama.cpp && \
    cd llama.cpp && \
    make LLAMA_OPENBLAS=1

# Copy model
COPY model.Q4_K_M.gguf /app/model.gguf

# Run server
CMD ["./llama.cpp/server", "-m", "/app/model.gguf", "-c", "2048", "-t", "8"]

Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-cpu-inference
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: llama
        image: llm-cpu-inference:latest
        resources:
          requests:
            cpu: "8000m"
            memory: "16Gi"
          limits:
            cpu: "8000m"
            memory: "16Gi"

---

Common Issues & Troubleshooting

Problem: Slow token generation (< 5 tok/s)

• Check: Thread count too high or too low
• Check: SIMD not enabled (recompile with LLAMA_NATIVE=1)
• Fix: Use smaller model or better quantization

Problem: Model doesn't fit in RAM

• Check: Model size vs available memory
• Fix: Use more aggressive quantization (Q4 → Q3)
• Fix: Reduce context window (n_ctx)

Problem: Poor quality with Q3/Q4

• Check: Quantization type (try Q4_K_M instead of Q4_0)
• Fix: Use Q5_K_M or Q6_K for critical applications

Problem: Crashes on ARM devices

• Check: NEON support in binary (recompile if needed)
• Fix: Reduce batch size and context window

---

References

• llama.cpp: https://github.com/ggerganov/llama.cpp
• GGUF Specification: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
• OpenVINO: https://docs.openvino.ai/latest/home.html
• ONNX Runtime: https://onnxruntime.ai/docs/
• MLC-LLM: https://llm.mlc.ai/
• TheBloke GGUF Models: https://huggingface.co/TheBloke

GPU Optimization Checklists

Actionable diagnostics for maximizing GPU throughput, token speed, and resource efficiency.

Updated January 2026: Added Blackwell (B200, GB200) benchmarks and H200 comparisons.

---

0. GPU Hardware Comparison (2026)

Performance Benchmarks

Blackwell-Specific Benchmarks (B200/GB200)

DeepSeek-R1 on GB200 NVL72:

• 26k input tokens/sec per GPU (prefill)
• 13k output tokens/sec per GPU (decode)
• 4x performance vs H100/H200

Key Blackwell Optimizations:

• FP8 attention (required - INT8 not supported)
• NVFP4 MoE quantization
• 5th gen tensor cores
• NVLink 5 (faster GPU-to-GPU)
• TMA (Tensor Memory Accelerator)

H200 vs H100

Recommendation: H200 for production serving today. B200/GB200 for maximum performance when available.

Software Requirements by GPU

---

1. GPU Profiling Checklist

• [ ] GPU utilization > 70% for steady-state workloads
• [ ] No long idle gaps between tokens
• [ ] NVLink / PCIe bandwidth not saturated
• [ ] Memory fragmentation low (<10%)
• [ ] Kernel launches efficient (no micro-kernel spam)

---

2. Memory Optimization Checklist

• [ ] Enable FlashAttention / xFormers
• [ ] Enable fused kernels where available
• [ ] Quantize model to INT8 or INT4
• [ ] Reduce KV cache memory (8-bit KV cache)
• [ ] Reduce max sequence length if possible

---

3. Latency Optimization Checklist

• [ ] Use TensorRT-LLM kernels
• [ ] Disable Python overhead with C++ runtime when needed
• [ ] Use pinned memory for CPU→GPU transfers
• [ ] Preload model and warm caches

---

4. Throughput Optimization Checklist

• [ ] Enable continuous batching
• [ ] Increase batch size incrementally
• [ ] Use multiple replicas behind LB
• [ ] Use multi-GPU tensor parallel if model too large
• [ ] Avoid blocking CPU workloads on GPU thread

---

5. Cost Optimization Checklist

• [ ] Right-size GPU type (L40, A100, H100)
• [ ] Use quantized models
• [ ] Cap output tokens
• [ ] Cache high-frequency prompts/responses
• [ ] Use spot/preemptible instances when safe

---

6. Production GPU Health Monitoring

Monitor:

• Utilization
• Memory usage
• Temperature
• Kernel exec time
• Queue backlog
• Token throughput per replica

Alerts

• [ ] GPU OOM
• [ ] Sustained low utilization
• [ ] Temperature > 85°C

Infrastructure Tuning for GPU-Heavy LLM Inference

Production-ready infrastructure optimization patterns for OS, containers, and Kubernetes environments running GPU-accelerated LLM inference workloads.

OS & CPU Optimization

NUMA Pinning & CPU Affinity

• Pin GPU worker processes to NUMA nodes closest to PCIe root complex
• Use numactl --cpunodebind=<node> --membind=<node> for worker processes
• Avoid cross-NUMA memory access (2-3x latency penalty)

Power Management

• Disable unnecessary C-states: cpupower frequency-set -g performance
• Keep GPU persistence mode enabled: nvidia-smi -pm 1
• Prevent GPU frequency throttling with consistent power profiles

Memory Configuration

• Enable huge pages (2MB/1GB): improves TLB hit rates

  echo 1024 > /proc/sys/vm/nr_hugepages  # 2GB of 2MB pages

• Reserve huge pages at boot for predictable allocation
• Use mlock() for critical inference code paths

GPU Driver & Runtime

CUDA & Driver Compatibility

• Match CUDA toolkit version to framework requirements
• Keep driver at recommended version (check framework docs)
• Test new drivers in staging before production rollout

Multi-Process Service (MPS)

• Enable when running multiple small concurrent workloads
• Improves GPU utilization for sub-saturating jobs
• Configure context limits based on memory constraints

  nvidia-cuda-mps-control -d  # Start MPS daemon
  export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps

Multi-Instance GPU (MIG)

• Use for GPU sharing with hardware isolation
• Configure MIG slices based on workload requirements

  nvidia-smi mig -cgi 9,9,9 -C  # Create 3x 1g.5gb instances

• Ideal for multi-tenant inference serving

Container Optimization

Image Optimization

• Use slim base images (CUDA runtime, not devel)
• Minimize overlay FS layers (combine RUN commands)
• Pin CUDA compatibility layer version explicitly

Storage Configuration

• Avoid swap entirely: set --memory-swap=0
• Use --shm-size for IPC between processes
• Mount model cache on fast local NVMe (not network FS)

Resource Limits

# Docker example
docker run \
  --gpus all \
  --shm-size=8g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --memory=32g \
  --cpuset-cpus="0-15" \
  your-llm-image

Kubernetes Best Practices

Topology-Aware Scheduling

• Use node labels for GPU types: nvidia.com/gpu.product=A100-SXM4-80GB
• Enable topology manager for NUMA awareness
• Request NVLink-connected GPUs for multi-GPU jobs

  resources:
    limits:
      nvidia.com/gpu: 2
  nodeSelector:
    nvidia.com/gpu.nvlink: "true"

Resource Guarantees

• Set requests == limits for QoS=Guaranteed
• Prevent CPU throttling with generous CPU requests
• Reserve GPU memory headroom (95% utilization target)

Reduce Orchestration Jitter

• Use priorityClassName: system-cluster-critical
• Set pod anti-affinity to avoid noisy neighbors
• Configure liveness/readiness probes with appropriate timeouts

  livenessProbe:
    httpGet:
      path: /health
      port: 8080
    initialDelaySeconds: 120
    periodSeconds: 30
    timeoutSeconds: 10

Local Storage for Cache/Spill

• Prefer node-local NVMe over network storage
• Use local-path-provisioner or hostPath volumes
• Mount paths:
• Model weights: /var/lib/models (read-only)
• KV cache spill: /var/lib/kv-cache (read-write, fast)

Networking Optimization

RDMA Configuration

• Enable RDMA for multi-node tensor parallel jobs
• Use RoCE v2 or InfiniBand where available
• Configure lossless networking (PFC/ECN)

gRPC & HTTP Tuning

• Set keepalive intervals: grpc.keepalive_time_ms=10000
• Enable HTTP/2 multiplexing for concurrent requests
• Tune connection pool sizes based on concurrency

Bandwidth Provisioning

• Ensure 10Gbps+ for KV cache offload paths
• Monitor network saturation (should stay <70%)
• Use jumbo frames (MTU=9000) for high-throughput workloads

Validation Checklist

• [ ] NUMA topology verified: numactl --hardware
• [ ] GPU persistence mode enabled: nvidia-smi -q -d PERFORMANCE
• [ ] Huge pages allocated: cat /proc/meminfo | grep Huge
• [ ] Container overlay FS optimized (minimal layers)
• [ ] Kubernetes topology-aware scheduling configured
• [ ] Node-local NVMe mounted for cache/spill
• [ ] Network bandwidth tested: iperf3 between nodes
• [ ] GPU health checks passing in pod readiness probes
• [ ] Resource requests/limits set for QoS=Guaranteed
• [ ] No swap enabled: swapon -s (should be empty)

Common Issues

Problem: GPU underutilization despite workload

• Check: CPU bottleneck (use nvidia-smi dmon + htop)
• Check: Memory bandwidth saturation (nvidia-smi dmon -s m)
• Fix: Increase batch size or enable MPS

Problem: OOM despite sufficient GPU memory

• Check: KV cache size exceeding allocation
• Check: Memory fragmentation (nvidia-smi --query-gpu=memory.free,memory.used)
• Fix: Reduce max batch size or enable PagedAttention

Problem: High latency variance

• Check: CPU throttling (cat /sys/fs/cgroup/cpu/cpu.stat)
• Check: NUMA cross-socket access (numastat)
• Fix: Pin workers to NUMA nodes, set QoS=Guaranteed

References

• NVIDIA GPU Best Practices: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/
• Kubernetes Device Plugins: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
• NUMA Deep Dive: https://www.kernel.org/doc/html/latest/vm/numa.html

KV Cache Optimization

Production strategies for optimizing key-value cache in LLM inference - the #1 latency and memory bottleneck for long-context workloads.

Overview

What is KV Cache?

• Cached key and value tensors from attention layers
• Prevents recomputing attention for previous tokens
• Essential for efficient autoregressive generation

Why it's critical:

• KV cache = largest memory consumer (often > model weights)
• Memory bandwidth bottleneck for long contexts (>8k tokens)
• Directly impacts: latency, throughput, cost, max batch size

Key challenges:

• Memory grows linearly with sequence length
• Fragmentation from variable-length sequences
• Bandwidth saturation on GPU → CPU transfers

---

KV Cache Memory Analysis

Size Calculation

Formula:

KV cache size = 2 (K+V) × num_layers × batch_size × seq_len × hidden_size × dtype_bytes

Example: Llama-2-13B:

def calculate_kv_cache_size(
    num_layers=40,
    batch_size=32,
    seq_len=4096,
    hidden_size=5120,
    dtype_bytes=2  # FP16
):
    size_bytes = 2 * num_layers * batch_size * seq_len * hidden_size * dtype_bytes
    size_gb = size_bytes / (1024**3)
    return size_gb

# Result: 80GB for 32 sequences @ 4k context
# This is MORE than the model weights (26GB for 13B FP16)

Memory breakdown (Llama-2-13B, FP16):

• Model weights: 26GB
• KV cache (batch=32, ctx=4096): 80GB
• Activations: ~4GB
• Total: 110GB (vs 26GB without batching/context)

Memory vs Context Length

Takeaway: KV cache dominates memory budget for long contexts

---

Optimization Strategies

1. PagedAttention (vLLM)

What it is: Dynamically allocate KV cache in fixed-size blocks

Benefits:

• Eliminates fragmentation (variable-length sequences)
• Enables massive batching (100+ concurrent requests)
• Memory sharing across requests (prefix caching)

How it works:

Traditional: Allocate max_seq_len for every request upfront
  → Wastes memory for short sequences
  → Fragmentation prevents optimal batching

PagedAttention: Allocate blocks on-demand as sequence grows
  → Only use memory actually needed
  → Reuse freed blocks immediately

Configuration (vLLM):

from vllm import LLM

model = LLM(
    model="meta-llama/Llama-2-13b-hf",
    max_model_len=8192,
    block_size=16,  # KV cache block size (tokens per block)
    gpu_memory_utilization=0.95,  # Use 95% of GPU memory
    enable_prefix_caching=True  # Reuse KV cache for common prefixes
)

Performance impact:

• 2-4x higher throughput vs static allocation
• 80-90% memory utilization (vs 50-60% without paging)

Validation:

# Check block utilization
from vllm import EngineArgs

engine_args = EngineArgs(...)
engine = LLMEngine.from_engine_args(engine_args)

# Monitor block usage
stats = engine.get_stats()
print(f"Blocks used: {stats['num_blocks_used']} / {stats['num_blocks_total']}")
print(f"Block utilization: {stats['num_blocks_used'] / stats['num_blocks_total'] * 100:.1f}%")

---

2. FlashAttention-3 and FlashInfer (2025+ Standard)

What it is: Memory-efficient attention algorithms with kernel-level optimization

FlashAttention Evolution:

FlashAttention-3 Breakthrough (Hopper GPUs):

• 85% utilization on H100 (vs 35% for FA-2)
• 1.5-2x speedup over FA-2 with BF16 (840 TFLOPs/s)
• FP8 support: 1.3 PFLOPs/s on H100
• Exploits NVIDIA Hopper features: async TMA, warp specialization

FlashInfer (MLSys 2025 Best Paper):

• NVIDIA's new kernel library for LLM inference
• Unified API for attention, GEMM, MoE operations
• Multiple backends: FlashAttention-2/3, cuDNN, CUTLASS, TensorRT-LLM
• Integrated into vLLM and SGLang

Why FlashInfer Matters

• NVIDIA is releasing optimized kernels through FlashInfer (not just TensorRT-LLM)
• JIT compilation for custom attention patterns
• Supports RadixAttention (SGLang's KV reuse pattern)
• 29-69% inter-token-latency reduction vs compiler backends
• 28-30% latency reduction for long-context inference

SGLang RadixAttention

What it is: Keep user prompts in KV cache for reuse across requests

Benefits:

• 6.4x higher throughput on structured workloads
• 3.7x lower latency vs baseline systems
• Excellent for chat, RAG, and few-shot scenarios

How it works:

Request 1: [System] + [Few-shot examples] + [User query A]
Request 2: [System] + [Few-shot examples] + [User query B]

RadixAttention: Cache [System] + [Few-shot examples] separately
  → Only compute [User query] for each new request
  → Massive savings for repetitive prompt structures

Configuration (Transformers):

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-13b-hf",
    attn_implementation="flash_attention_2",  # FA-3 auto-selected on Hopper
    torch_dtype=torch.float16
)

Configuration (vLLM with FlashInfer):

# FlashInfer is now the default attention backend in vLLM
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

# vLLM auto-detects and uses FlashInfer/FA-3 if available
vllm serve meta-llama/Llama-3-70B --enable-flashinfer

Configuration (SGLang with RadixAttention):

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3-70B \
    --enable-radix-attention

Performance comparison:

Benchmark: Llama-2-13B, batch=32, seq_len=4096

Standard Attention:
- Latency: 2.5s per batch
- Memory: 160GB
- Throughput: 800 tok/s

FlashAttention-2:
- Latency: 0.9s per batch (2.8x faster)
- Memory: 90GB (1.8x reduction)
- Throughput: 2200 tok/s (2.75x higher)

FlashAttention-3 (H100):
- Latency: 0.45s per batch (5.5x faster)
- Memory: 85GB
- Throughput: 4400 tok/s (5.5x higher)
- GPU utilization: 85% (vs 35% FA-2)

---

3. KV Cache Quantization

What it is: Store cached keys/values in lower precision

Benefits:

• 2-4x memory reduction
• Minimal quality loss (<1% accuracy)
• Enables larger batches or longer contexts

Precision options:

• FP16 → FP8: 2x compression, ~0.5% quality loss
• FP16 → INT8: 2x compression, ~1% quality loss
• FP16 → INT4: 4x compression, ~2-3% quality loss (experimental)

Configuration (vLLM):

from vllm import LLM

model = LLM(
    model="meta-llama/Llama-2-13b-hf",
    kv_cache_dtype="fp8",  # or "auto" for automatic selection
    quantization="fp8"  # Also quantize model weights
)

Memory savings example (Llama-2-13B, batch=32, ctx=4096):

• FP16 KV cache: 80GB
• FP8 KV cache: 40GB (2x reduction)
• INT8 KV cache: 40GB (2x reduction)

Quality validation:

import numpy as np

def measure_quality_impact(prompts, model_fp16, model_fp8):
    """Compare outputs with FP16 vs FP8 KV cache"""
    results = []

    for prompt in prompts:
        output_fp16 = model_fp16.generate(prompt)
        output_fp8 = model_fp8.generate(prompt)

        # Compare token-by-token accuracy
        tokens_fp16 = tokenizer.encode(output_fp16)
        tokens_fp8 = tokenizer.encode(output_fp8)

        # Calculate token match rate
        min_len = min(len(tokens_fp16), len(tokens_fp8))
        matches = sum(t1 == t2 for t1, t2 in zip(tokens_fp16[:min_len], tokens_fp8[:min_len]))
        match_rate = matches / min_len

        results.append(match_rate)

    avg_match_rate = np.mean(results)
    print(f"Token match rate: {avg_match_rate * 100:.2f}%")
    # Expected: 98-99% for FP8, 96-98% for INT8

    return avg_match_rate

---

4. KV Cache Offloading

What it is: Move KV cache to CPU memory or disk when GPU memory full

When to use:

• Very long contexts (>32k tokens)
• Memory-constrained scenarios
• Offline/batch processing (latency less critical)

Trade-offs:

• Frees GPU memory for more batching
• Adds latency (PCIe transfer overhead)
• Requires high CPU ↔ GPU bandwidth

Implementation (DeepSpeed):

from deepspeed.inference import init_inference

model = init_inference(
    model,
    mp_size=1,
    dtype=torch.float16,
    checkpoint=None,
    enable_cuda_graph=False,
    replace_with_kernel_inject=True,
    kv_cache_dtype=torch.float16,
    offload_kv_cache=True  # Enable CPU offloading
)

Performance impact:

Without offloading:
- Max batch size: 16 (GPU memory limit)
- Latency: 1.2s
- Throughput: 512 tok/s

With CPU offloading:
- Max batch size: 64 (4x larger)
- Latency: 2.5s (2x slower due to transfers)
- Throughput: 1280 tok/s (2.5x higher)

When offloading makes sense:

• Batch size increase > latency penalty
• Offline workloads (batch prediction)
• Not suitable for real-time APIs (<1s latency requirement)

---

5. Prefix Caching / Prompt Caching

What it is: Reuse KV cache for common prompt prefixes

Use cases:

• System prompts (same for every request)
• Few-shot examples (repeated in every prompt)
• Conversation history (multi-turn chat)

Example:

Prompt 1: [System prompt] + [User: Hello]
Prompt 2: [System prompt] + [User: How are you?]
Prompt 3: [System prompt] + [User: Tell me a joke]

Without prefix caching: Recompute [System prompt] 3 times
With prefix caching: Compute [System prompt] once, reuse 3 times

Configuration (vLLM):

from vllm import LLM

model = LLM(
    model="meta-llama/Llama-2-13b-hf",
    enable_prefix_caching=True
)

# Prompts with common prefix automatically benefit
system_prompt = "You are a helpful AI assistant. You are friendly and concise."

prompts = [
    system_prompt + "\n\nUser: Hello",
    system_prompt + "\n\nUser: How are you?",
    system_prompt + "\n\nUser: Tell me a joke"
]

# First request: computes full KV cache
# Next 2 requests: reuse cached system_prompt KV, only compute user message
outputs = model.generate(prompts)

Performance impact:

Scenario: System prompt = 500 tokens, user message = 50 tokens

Without prefix caching:
- Time per request: 550 tokens × 10ms = 5.5s

With prefix caching:
- First request: 550 tokens × 10ms = 5.5s
- Subsequent: 50 tokens × 10ms = 0.5s (11x faster!)

Implementation tips:

• Structure prompts with common prefixes first
• Use deterministic ordering (cache hit depends on exact match)
• Monitor cache hit rate

---

6. Grouped Prefill

What it is: Process multiple prefills together in single batch

Benefits:

• Better GPU utilization during prefill phase
• Reduced latency for concurrent requests
• Improves throughput for bursty traffic

How it works:

Traditional: Process each prefill sequentially
  Request 1 prefill → Request 1 decode → Request 2 prefill → Request 2 decode → ...

Grouped prefill: Batch prefills together
  [Request 1, 2, 3 prefills] → [Request 1, 2, 3 decode] → ...

Configuration (vLLM):

# Enabled by default in vLLM's continuous batching
# No explicit configuration needed

# Monitor prefill batch sizes
from vllm import LLM

model = LLM(
    model="meta-llama/Llama-2-13b-hf",
    max_num_batched_tokens=8192,  # Max tokens in prefill batch
    max_num_seqs=256  # Max sequences in batch
)

---

7. Sequence Parallelism

What it is: Split sequence dimension across GPUs

Benefits:

• Reduces per-GPU memory for KV cache
• Enables longer sequences on same hardware
• Complements tensor parallelism

When to use:

• Very long contexts (>16k tokens)
• Multi-GPU setups
• Combined with tensor parallelism

Implementation (Megatron-LM):

# Megatron-LM sequence parallelism
args = {
    'tensor_model_parallel_size': 4,  # TP across 4 GPUs
    'sequence_parallel': True,  # Enable sequence parallelism
    'use_flash_attn': True
}

Performance:

Without sequence parallelism:
- Max context: 8k tokens (per-GPU memory limit)
- 4 GPUs × 8k = 32k total capacity (wasted)

With sequence parallelism:
- Max context: 32k tokens (split across 4 GPUs)
- 4 GPUs × 32k = 32k total capacity (fully utilized)

---

Configuration Recommendations

By Use Case

1. High-throughput API (short contexts):

LLM(
    model="meta-llama/Llama-2-13b-hf",
    max_model_len=2048,  # Limit context
    enable_prefix_caching=True,  # Cache system prompts
    kv_cache_dtype="auto",  # Auto FP8 if supported
    gpu_memory_utilization=0.95
)

2. Long-context workloads (<16k):

LLM(
    model="meta-llama/Llama-2-13b-hf",
    max_model_len=16384,
    kv_cache_dtype="fp8",  # Reduce memory
    enable_prefix_caching=True,
    tensor_parallel_size=2  # Split across GPUs
)

3. Ultra-long context (>32k):

LLM(
    model="meta-llama/Llama-2-13b-hf",
    max_model_len=65536,
    kv_cache_dtype="fp8",
    tensor_parallel_size=4,
    # Consider CPU offloading for very long sequences
)

4. Memory-constrained (single GPU, large model):

LLM(
    model="meta-llama/Llama-2-70b-hf",
    quantization="awq",  # INT4 model weights
    kv_cache_dtype="fp8",  # FP8 KV cache
    max_model_len=4096,  # Limit context
    gpu_memory_utilization=0.95
)

---

Monitoring & Debugging

Key Metrics

Monitor these KV cache metrics:

# vLLM engine stats
stats = engine.get_stats()

print(f"KV cache blocks used: {stats['num_blocks_used']}")
print(f"KV cache blocks total: {stats['num_blocks_total']}")
print(f"KV cache utilization: {stats['num_blocks_used'] / stats['num_blocks_total'] * 100:.1f}%")
print(f"Prefix cache hit rate: {stats.get('prefix_cache_hit_rate', 0) * 100:.1f}%")

GPU memory breakdown:

import torch

print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
print(f"Max allocated: {torch.cuda.max_memory_allocated() / 1e9:.2f} GB")

Common Issues

Problem: OOM during prefill

• Cause: Batch size too large for prompt length
• Fix: Reduce max_num_batched_tokens or enable KV cache quantization

Problem: OOM during decode

• Cause: KV cache grows beyond allocation
• Fix: Reduce max_model_len or use FP8 KV cache

Problem: Low cache hit rate (<20%)

• Cause: Variable prompt structure
• Fix: Standardize prompt templates, move common prefixes to start

Problem: High fragmentation (utilization <70%)

• Cause: Variable sequence lengths with static allocation
• Fix: Use PagedAttention (vLLM)

---

Validation Checklist

• [ ] PagedAttention enabled (vLLM)
• [ ] FlashAttention enabled (check logs/config)
• [ ] KV cache quantization configured (FP8 for >4k context)
• [ ] Cache size appropriate for max batch × max context
• [ ] Prefix caching enabled for common prompts
• [ ] Offloading strategy chosen if memory-bound
• [ ] Cache hit rate > 50% (for workloads with common prefixes)
• [ ] Memory utilization > 80% (not fragmented)
• [ ] No OOM errors under max load

---

References

• PagedAttention Paper: https://arxiv.org/abs/2309.06180
• FlashAttention-2: https://arxiv.org/abs/2307.08691
• FlashAttention-3: https://pytorch.org/blog/flashattention-3/
• FlashInfer (MLSys 2025): https://arxiv.org/abs/2501.01005
• FlashInfer GitHub: https://github.com/flashinfer-ai/flashinfer
• SGLang RadixAttention: https://lmsys.org/blog/2024-07-25-sglang-llama3/
• vLLM Documentation: https://docs.vllm.ai/
• DeepSpeed Inference: https://www.deepspeed.ai/tutorials/inference-tutorial/
• NVIDIA Attention Optimizations: https://docs.nvidia.com/deeplearning/performance/

Multi-Model Routing

Operational reference for routing LLM requests to different models based on complexity, cost, and latency requirements — classifier-based routing, cascade patterns, A/B routing, router architectures, and evaluation of routing effectiveness.

Freshness anchor: January 2026 — covers Martian router, OpenRouter, RouteLLM, Unify.ai, and custom routing with LiteLLM.

---

Routing Strategy Decision Tree

Incoming LLM request
│
├── Is the task type known at request time?
│   ├── YES → Rule-Based Routing
│   │   ├── Classification/extraction → Small model
│   │   ├── Summarization → Medium model
│   │   ├── Complex reasoning → Large model
│   │   └── Code generation → Code-specialized model
│   │
│   └── NO → Need to classify first
│       │
│       ├── Can you classify cheaply? (<1ms overhead)
│       │   ├── YES → Classifier-Based Routing
│       │   │   ├── Lightweight classifier (regex, keyword, logistic regression)
│       │   │   ├── Small LLM classifier (GPT-4o-mini)
│       │   │   └── Embedding similarity to task clusters
│       │   │
│       │   └── NO → Cascade Pattern
│       │       ├── Try small model first
│       │       ├── Check confidence/quality
│       │       └── Escalate to larger model if needed
│       │
│       └── Is latency critical?
│           ├── YES → Route to fastest model that meets quality bar
│           └── NO → Route to cheapest model that meets quality bar
│
├── Do you need guaranteed quality?
│   ├── YES → Always use best model (no routing)
│   └── NO → Routing can save 40-70% cost
│
└── Are you A/B testing models?
    ├── YES → Percentage-based routing with eval
    └── NO → Deterministic routing

---

Routing Patterns

Pattern 1: Rule-Based Routing

Use when: task types are known, limited categories, need zero latency overhead

class RuleBasedRouter:
    RULES = {
        # Task type → model
        "greeting": "gpt-4o-mini",
        "faq": "gpt-4o-mini",
        "classification": "gpt-4o-mini",
        "extraction": "gpt-4o-mini",
        "summarization": "gpt-4o",
        "analysis": "gpt-4o",
        "creative": "gpt-4o",
        "reasoning": "o3-mini",
        "math": "o3-mini",
        "code": "claude-3-5-sonnet-20241022",
    }

    def route(self, task_type: str, constraints: dict = None) -> str:
        model = self.RULES.get(task_type, "gpt-4o")  # default to standard

        # Apply constraints
        if constraints:
            if constraints.get("max_cost_per_request", float("inf")) < 0.01:
                model = self._downgrade(model)
            if constraints.get("max_latency_ms", float("inf")) < 1000:
                model = self._fastest_alternative(model)

        return model

Pattern 2: Classifier-Based Routing

Use when: task types are NOT known upfront, need dynamic classification
Overhead: 5-50ms for lightweight classifier, 200-500ms for LLM classifier

from sklearn.linear_model import LogisticRegression
from sentence_transformers import SentenceTransformer

class ClassifierRouter:
    """Classify request complexity, route to appropriate model."""

    COMPLEXITY_TIERS = {
        0: {"model": "gpt-4o-mini", "label": "simple"},
        1: {"model": "gpt-4o", "label": "standard"},
        2: {"model": "o3-mini", "label": "complex"},
    }

    def __init__(self):
        self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
        self.classifier = LogisticRegression()

    def train(self, labeled_examples: list[dict]):
        """Train on historical requests with labels."""
        texts = [ex["text"] for ex in labeled_examples]
        labels = [ex["complexity_tier"] for ex in labeled_examples]

        embeddings = self.embedder.encode(texts)
        self.classifier.fit(embeddings, labels)

    def route(self, request: str) -> dict:
        embedding = self.embedder.encode([request])
        tier = self.classifier.predict(embedding)[0]
        confidence = max(self.classifier.predict_proba(embedding)[0])

        result = self.COMPLEXITY_TIERS[tier].copy()
        result["confidence"] = confidence

        # Low confidence → route to standard model as safety net
        if confidence < 0.7:
            result["model"] = self.COMPLEXITY_TIERS[1]["model"]
            result["reason"] = "low_confidence_fallback"

        return result

Pattern 3: Cascade (Small → Large Fallback)

Use when: want to minimize cost while maintaining quality, latency tolerance exists
Overhead: 1-2x latency on escalated requests, net cost savings 40-60%

class CascadeRouter:
    """Try small model first, escalate to larger if quality is insufficient."""

    def __init__(self):
        self.models = [
            {"name": "gpt-4o-mini", "cost_tier": 1},
            {"name": "gpt-4o", "cost_tier": 2},
            {"name": "o3-mini", "cost_tier": 3},
        ]
        self.quality_checker = QualityChecker()

    async def generate(self, messages: list, quality_threshold: float = 0.8) -> dict:
        for i, model in enumerate(self.models):
            response = await llm_call(model["name"], messages)

            # Last model: return regardless
            if i == len(self.models) - 1:
                return {"response": response, "model": model["name"], "escalations": i}

            # Check quality
            quality_score = await self.quality_checker.score(
                input_messages=messages,
                output=response.text
            )

            if quality_score >= quality_threshold:
                return {
                    "response": response,
                    "model": model["name"],
                    "escalations": i,
                    "quality_score": quality_score
                }

            # Escalate to next model
            continue

class QualityChecker:
    """Lightweight quality check for cascade decisions."""

    async def score(self, input_messages, output) -> float:
        checks = {
            "not_empty": len(output.strip()) > 10,
            "not_refusal": not output.startswith("I cannot"),
            "reasonable_length": 50 < len(output) < 10000,
            "no_repetition": self._check_no_repetition(output),
            "addresses_question": await self._relevance_check(input_messages, output),
        }
        return sum(checks.values()) / len(checks)

Pattern 4: A/B Routing

Use when: evaluating new models before full migration, measuring quality impact

import hashlib
import random

class ABRouter:
    """Route traffic between models for evaluation."""

    def __init__(self, config: dict):
        self.config = config
        # Example: {"control": {"model": "gpt-4o", "weight": 0.8},
        #           "treatment": {"model": "claude-3-5-sonnet", "weight": 0.2}}

    def route(self, session_id: str) -> dict:
        # Deterministic routing based on session (same user always gets same model)
        hash_value = int(hashlib.md5(session_id.encode()).hexdigest(), 16)
        normalized = (hash_value % 1000) / 1000

        cumulative = 0
        for variant_name, variant_config in self.config.items():
            cumulative += variant_config["weight"]
            if normalized < cumulative:
                return {
                    "model": variant_config["model"],
                    "variant": variant_name,
                    "session_id": session_id
                }

        # Fallback
        return {"model": list(self.config.values())[0]["model"], "variant": "control"}

---

Router Architecture Comparison

LiteLLM Router Configuration

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "general",
            "litellm_params": {
                "model": "gpt-4o-mini",
                "api_key": "sk-...",
            },
            "model_info": {"id": "gpt-4o-mini-1"}
        },
        {
            "model_name": "general",
            "litellm_params": {
                "model": "claude-3-haiku-20240307",
                "api_key": "sk-ant-...",
            },
            "model_info": {"id": "claude-haiku-1"}
        },
    ],
    routing_strategy="least-busy",  # or "simple-shuffle", "latency-based-routing"
    num_retries=2,
    fallbacks=[{"gpt-4o-mini": ["claude-3-haiku-20240307"]}],
    set_verbose=False
)

response = await router.acompletion(
    model="general",
    messages=[{"role": "user", "content": "Hello"}]
)

---

Quality-Cost Tradeoff Framework

Calculating Optimal Routing Split

Routing Effectiveness Dashboard

Track these metrics daily:

1. Traffic distribution by model
   - % of requests per model
   - Trend over time (is routing stable?)

2. Quality by route
   - Eval score per model tier
   - Failure rate per model tier

3. Cost by route
   - $/request per model tier
   - Total cost vs "all-best-model" counterfactual

4. Escalation metrics (cascade only)
   - Escalation rate
   - Quality of escalated vs non-escalated
   - Latency impact of escalation

5. Routing decision quality
   - Were requests correctly classified?
   - Sample review of routing decisions

---

Fallback and Resilience Patterns

Fallback Chain Configuration

FALLBACK_CHAINS = {
    "openai": {
        "primary": "gpt-4o",
        "fallbacks": [
            {"model": "gpt-4o-mini", "condition": "rate_limit"},
            {"model": "claude-3-5-sonnet-20241022", "condition": "any_error"},
            {"model": "gemini-2.0-flash", "condition": "any_error"},
        ]
    },
    "anthropic": {
        "primary": "claude-3-5-sonnet-20241022",
        "fallbacks": [
            {"model": "claude-3-haiku-20240307", "condition": "rate_limit"},
            {"model": "gpt-4o", "condition": "any_error"},
        ]
    }
}

Health Check and Circuit Breaker

---

Anti-Patterns

---

Cross-References

• cost-optimization-patterns.md — cost strategies that complement routing
• streaming-patterns.md — streaming across routed models
• ../ai-llm/references/model-migration-guide.md — evaluating models for routing tiers
• ../ai-llm/references/structured-output-patterns.md — provider-specific output differences
• ../ai-prompt-engineering/references/prompt-testing-ci-cd.md — eval infrastructure for routing

Optimization Strategies: Latency, Throughput, Cost (Dec 2025)

Production patterns for optimizing LLM inference. This is intentionally vendor-agnostic and avoids benchmark numbers without context.

Core references:

• vLLM / PagedAttention (https://arxiv.org/abs/2309.06180)
• Orca scheduling (https://www.usenix.org/conference/osdi22/presentation/yu)
• FlashAttention (https://arxiv.org/abs/2205.14135)
• Speculative decoding (https://arxiv.org/abs/2302.01318)

---

0. Operating Principles (REQUIRED)

• Measure under representative load; single-request latency is not predictive.
• Optimize the largest contributor first (queueing vs prefill vs decode vs post-processing).
• Set and enforce budgets at the boundary: max_input_tokens, max_output_tokens, max_concurrency, max_queue_depth, max_retries.
• Treat every “optimization” as a change: validate quality, add rollback, and re-run the inference review checklist.

---

1. Measurement Protocol (REQUIRED)

Define traffic shape

• Prompt length distribution (p50/p95/p99)
• Output length distribution
• Concurrency and QPS targets
• Streaming vs non-streaming and typical client behavior (cancellations, disconnects)

Measure the right metrics

• TTFT (time to first token) p50/p95/p99
• ITL / tok/s (inter-token latency / token throughput)
• Total latency p50/p95/p99
• Error rate by class (timeout, overload, upstream, OOM, validation)
• Saturation: GPU memory headroom, GPU utilization, queue depth

Tag every measurement run

• Model id + revision hash
• Serving config hash
• Hardware + driver/runtime versions
• Token limits + decoding settings

---

2. Latency Optimization (TTFT and Total)

If TTFT is high

• Reduce prefill work:
• Shrink prompt scaffolding and retrieved context (context budgets).
• Use chunked prefill where supported.
• Reduce queueing:
• Admission control: cap queue depth and return overload fast.
• Autoscale on queue depth and p95 latency (not average utilization).

If decode is slow (high ITL)

• Use more efficient kernels/runtime (FlashAttention-style kernels: https://arxiv.org/abs/2205.14135).
• Consider speculative decoding if your use case tolerates draft-model errors and you validate quality (https://arxiv.org/abs/2302.01318).
• Reduce output length: enforce max_output_tokens, add stop sequences, and prefer concise outputs where possible.

---

3. Throughput Optimization (QPS / tok/s)

Scheduling and batching

• Use continuous batching / smart scheduling for high concurrency (Orca: https://www.usenix.org/conference/osdi22/presentation/yu).
• Use KV-cache aware serving and paging to avoid fragmentation and OOMs under load (vLLM/PagedAttention: https://arxiv.org/abs/2309.06180).

Parallelism

• Choose parallelism based on bottleneck:
• Tensor parallelism when the model does not fit one device.
• Pipeline parallelism when depth dominates and interconnect is constrained.
• Data parallelism (replicas) for throughput and availability.

---

4. Cost Optimization (Budget and Unit Economics)

• Model tiering: route easy traffic to smaller/cheaper models; reserve large models for hard cases.
• Caching with invalidation:
• Prefix caching for repeated scaffolds (system/tool instructions).
• Response caching only when you can invalidate safely (freshness/ACL correctness).
• Quantization/compression:
• Treat as an optimization candidate only after you have a baseline and eval set.
• Validate on: task success, refusal correctness, and long-context cases.
• Enforce budgets at the API boundary; do not rely on prompt text to cap spend.

---

5. Decision Tree: What to Optimize First?

Latency/throughput regression: [Find the bottleneck]
    ├─ P99 spikes during load?
    │   ├─ Queue depth growing? → admission control + queue cap + autoscale
    │   └─ OOMs / evictions? → reduce context + improve KV management + adjust batching
    │
    ├─ TTFT too high?
    │   ├─ Long inputs? → context budgets + chunked prefill + retrieval trimming
    │   └─ Queueing? → concurrency caps + scale out
    │
    ├─ Decode slow (high ITL)?
    │   ├─ Kernel/runtime inefficiency? → attention kernel upgrades + runtime tuning
    │   └─ Output too long? → max_output_tokens + stop sequences
    │
    └─ Cost too high?
        ├─ High token usage? → prompt/context reduction + output caps
        ├─ Too-large model? → tiering/router + fallback strategy
        └─ Inefficient serving? → batching + caching + validated quantization

---

6. Anti-Patterns (AVOID)

• Unbounded context windows (latency and OOM explosions).
• Unbounded retries (cascading failures).
• Benchmarking without concurrency (false confidence).
• Optimizing throughput while violating TTFT SLOs (bad UX for interactive apps).
• Caching without invalidation and ACL enforcement (security/correctness failures).

---

7. Baseline Checklist (Before Shipping)

• [ ] Traffic model documented (prompt/output distributions, concurrency, streaming).
• [ ] Latency/cost budgets enforced at API boundary.
• [ ] Inference review checklist completed: ../assets/checklists/inference-review-checklist.md.
• [ ] Telemetry exported for tokens/latency/errors/queue depth (OpenTelemetry GenAI semconv: https://opentelemetry.io/docs/specs/semconv/gen-ai/).
• [ ] Rollback plan tested (previous model/config still deployable).

Parallelism Patterns for Large Model Inference

Production strategies for distributing LLM inference across multiple GPUs, optimizing for different model architectures and hardware configurations.

Overview

Choose parallelism type based on:

• Model architecture (wide vs deep vs sparse MoE)
• GPU interconnect bandwidth (NVLink, PCIe, InfiniBand)
• Inference latency requirements
• Memory constraints per GPU

Tensor Parallelism (TP)

What it is: Split individual layers across multiple GPUs (horizontal sharding)

Best for:

• Wide models with large hidden dimensions
• High-bandwidth interconnects (NVLink required)
• Low-latency inference (minimal communication overhead)

How it works:

• Split weight matrices column-wise or row-wise
• Distribute computation within each layer
• Synchronize activations after each layer

Configuration example (vLLM):

from vllm import LLM

model = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,  # Split across 4 GPUs
    dtype="float16"
)

Performance characteristics:

• Communication: High (all-reduce after every layer)
• Latency impact: Low (NVLink bandwidth ~600 GB/s)
• Scaling efficiency: 80-95% with NVLink

Hardware requirements:

• NVLink or NVSwitch for multi-GPU nodes
• 600+ GB/s interconnect bandwidth recommended
• GPUs on same physical node preferred

Validation checklist:

• [ ] NVLink topology verified: nvidia-smi topo -m
• [ ] TP degree divides model dimensions evenly
• [ ] Communication overhead < 15% of compute time
• [ ] Memory balanced across GPUs (±5%)

Pipeline Parallelism (PP)

What it is: Distribute different layers to different GPUs (vertical sharding)

Best for:

• Deep models with many layers
• Cross-node inference (lower bandwidth OK)
• Memory-constrained scenarios

How it works:

• Split model into stages (consecutive layer groups)
• Process micro-batches in pipeline fashion
• Overlap computation and communication

Configuration example (DeepSpeed):

ds_config = {
    "pipeline": {
        "stages": 8,  # 8 pipeline stages
        "micro_batch_size": 4,
        "partition_method": "uniform"
    }
}

Pipeline bubble mitigation:

• Use multiple micro-batches per batch
• Overlap forward and backward passes (for training)
• Schedule: GPipe, 1F1B (one-forward-one-backward)

Performance characteristics:

• Communication: Low (only at stage boundaries)
• Latency impact: Medium (pipeline fill/drain overhead)
• Scaling efficiency: 60-75% (depends on bubble size)

Bubble calculation:

Bubble time = (num_stages - 1) / num_microbatches * stage_time

Validation checklist:

• [ ] Micro-batch count ≥ 4 × num_stages
• [ ] Stage compute times balanced (±10%)
• [ ] Pipeline bubble < 20% of total time
• [ ] Memory usage balanced across stages

Expert Parallelism (EP) - Mixture of Experts

What it is: Distribute expert networks across GPUs

Best for:

• MoE models (Mixtral, GPT-4 style architectures)
• Sparse activation patterns
• Large model capacity with controlled compute

How it works:

• Each GPU hosts subset of experts
• Route tokens to appropriate expert GPUs
• Experts process in parallel

Configuration example (Megatron-LM):

moe_config = {
    "expert_parallel_size": 8,  # 8 GPUs for experts
    "num_experts": 64,
    "top_k": 2,  # Activate top-2 experts per token
    "capacity_factor": 1.25
}

Performance characteristics:

• Communication: Variable (depends on routing)
• Load balancing: Critical (use auxiliary losses)
• Scaling efficiency: 70-90% (depends on load balance)

Load balancing strategies:

• Auxiliary load balancing loss during fine-tuning
• Dynamic capacity factors
• Expert choice routing (instead of token choice)

Validation checklist:

• [ ] Expert utilization balanced (coefficient of variation < 0.3)
• [ ] Capacity factor prevents token dropping
• [ ] Routing overhead < 10% of expert compute
• [ ] Top-k selection appropriate for task

Data Parallelism (DP)

What it is: Replicate model on multiple GPUs, process different batches

Best for:

• High-throughput serving (many concurrent requests)
• Independent requests (no shared KV cache)
• Horizontal scaling

How it works:

• Each GPU has full model copy
• Load balancer distributes requests
• No inter-GPU communication during inference

Configuration example:

# Kubernetes deployment
replicas: 4  # 4 independent model replicas
resources:
  limits:
    nvidia.com/gpu: 1

Performance characteristics:

• Communication: None (fully independent)
• Latency impact: None (per-replica)
• Scaling efficiency: ~100% (linear)

Validation checklist:

• [ ] Load balancer configured (round-robin or least-connections)
• [ ] Health checks on all replicas
• [ ] Autoscaling policies defined
• [ ] Model weights synced across replicas

Fully Sharded Data Parallel (FSDP)

What it is: Shard model parameters, optimizer states, and gradients

Best for:

• Training large models (less common for inference)
• Inference on modified/adapters
• Memory-constrained multi-GPU setups

How it works:

• Shard model parameters across GPUs
• All-gather for computation
• Reduce-scatter for gradients (training)

Configuration example (PyTorch):

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP

model = FSDP(
    model,
    sharding_strategy="FULL_SHARD",
    cpu_offload=False
)

Validation checklist:

• [ ] Sharding strategy matches use case
• [ ] All-gather bandwidth adequate
• [ ] Memory reduction verified

Hybrid Parallelism Strategies

TP + PP (Most Common)

Pattern: Tensor parallel within nodes, pipeline parallel across nodes

Example: 70B model on 16 GPUs (4 nodes × 4 GPUs)

• TP=4 (within each node via NVLink)
• PP=4 (across 4 nodes via InfiniBand)

Configuration (vLLM):

model = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,
    pipeline_parallel_size=4
)

TP + EP (MoE Models)

Pattern: Tensor parallel for dense layers, expert parallel for MoE layers

Example: Mixtral 8x7B on 8 GPUs

• TP=2 for dense layers
• EP=8 for expert layers (each GPU hosts 8 experts)

3D Parallelism: TP + PP + DP

Pattern: Combine all three for massive scale

Example: 175B model on 64 GPUs

• TP=8 (within node)
• PP=4 (across nodes)
• DP=2 (replicas for throughput)

Decision Matrix

Communication Overhead Analysis

Measure communication cost:

import torch.distributed as dist

# Profile all-reduce time
start = time.time()
dist.all_reduce(tensor)
torch.cuda.synchronize()
comm_time = time.time() - start

# Compare to compute time
compute_ratio = compute_time / comm_time
# Target: ratio > 10 for efficient parallelism

Optimization strategies:

• Overlap communication with computation
• Use gradient accumulation to reduce sync frequency
• Compress gradients (FP16/BF16)
• Use NCCL for multi-GPU communication

Troubleshooting

Problem: Poor TP scaling (< 80% efficiency)

• Check: NVLink bandwidth saturation
• Check: Unbalanced layer sizes
• Fix: Reduce TP degree or use PP instead

Problem: High PP bubble overhead

• Check: Micro-batch count too low
• Check: Stage compute times unbalanced
• Fix: Increase micro-batches or rebalance stages

Problem: Expert load imbalance in MoE

• Check: Routing distribution (expert_counts)
• Fix: Add load balancing loss or adjust capacity factor

Problem: FSDP slow all-gather

• Check: Network bandwidth between nodes
• Fix: Use faster interconnect or reduce sharding degree

Performance Validation

# Benchmark script
import time
import torch

def benchmark_parallelism(model, inputs, warmup=10, iterations=100):
    # Warmup
    for _ in range(warmup):
        _ = model(inputs)

    torch.cuda.synchronize()
    start = time.time()

    for _ in range(iterations):
        _ = model(inputs)

    torch.cuda.synchronize()
    end = time.time()

    throughput = iterations / (end - start)
    return throughput

# Expected results
# Single GPU: 10 tok/s
# TP=4: 35-40 tok/s (ideal: 40, efficiency: 87.5-100%)
# TP+PP: varies by configuration

References

• Megatron-LM Parallelism: https://arxiv.org/abs/2104.04473
• vLLM Multi-GPU: https://docs.vllm.ai/en/latest/serving/distributed_serving.html
• DeepSpeed Pipeline: https://www.deepspeed.ai/tutorials/pipeline/
• NCCL Performance: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html