Model Serving

Model Serving

Purpose

Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications.

When to Use

• Deploying LLMs for production (self-hosted Llama, Mistral, Qwen)
• Building AI APIs with streaming responses
• Serving traditional ML models (scikit-learn, XGBoost, PyTorch)
• Implementing RAG pipelines with vector databases
• Optimizing inference throughput and latency
• Integrating LLM serving with frontend chat interfaces

Model Serving Selection

LLM Serving Engines

vLLM (Recommended Primary)

• PagedAttention memory management (20-30x throughput improvement)
• Continuous batching for dynamic request handling
• OpenAI-compatible API endpoints
• Use for: Most self-hosted LLM deployments

TensorRT-LLM

• Maximum GPU efficiency (2-8x faster than vLLM)
• Requires model conversion and optimization
• Use for: Production workloads needing absolute maximum throughput

Ollama

• Local development without GPUs
• Simple CLI interface
• Use for: Prototyping, laptop development, educational purposes

Decision Framework:

Self-hosted LLM deployment needed?
├─ Yes, need maximum throughput → vLLM
├─ Yes, need absolute max GPU efficiency → TensorRT-LLM
├─ Yes, local development only → Ollama
└─ No, use managed API (OpenAI, Anthropic) → No serving layer needed

ML Model Serving (Non-LLM)

BentoML (Recommended)

• Python-native, easy deployment
• Adaptive batching for throughput
• Multi-framework support (scikit-learn, PyTorch, XGBoost)
• Use for: Most traditional ML model deployments

Triton Inference Server

• Multi-model serving on same GPU
• Model ensembles (chain multiple models)
• Use for: NVIDIA GPU optimization, serving 10+ models

LLM Orchestration

LangChain

• General-purpose workflows, agents, RAG
• 100+ integrations (LLMs, vector DBs, tools)
• Use for: Most RAG and agent applications

LlamaIndex

• RAG-focused with advanced retrieval strategies
• 100+ data connectors (PDF, Notion, web)
• Use for: RAG is primary use case

Quick Start Examples

vLLM Server Setup

# Install
pip install vllm

# Serve a model (OpenAI-compatible API)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype auto \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --port 8000

Key Parameters:

• --dtype: Model precision (auto, float16, bfloat16)
• --max-model-len: Context window size
• --gpu-memory-utilization: GPU memory fraction (0.8-0.95)
• --tensor-parallel-size: Number of GPUs for model parallelism

Streaming Responses (SSE Pattern)

Backend (FastAPI):

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import OpenAI
import json

app = FastAPI()
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

@app.post("/chat/stream")
async def chat_stream(message: str):
    async def generate():
        stream = client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            messages=[{"role": "user", "content": message}],
            stream=True,
            max_tokens=512
        )

        for chunk in stream:
            if chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                yield f"data: {json.dumps({'token': token})}\n\n"

        yield f"data: {json.dumps({'done': True})}\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache"}
    )

Frontend (React):

// Integration with ai-chat skill
const sendMessage = async (message: string) => {
  const response = await fetch('/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message })
  })

  const reader = response.body!.getReader()
  const decoder = new TextDecoder()

  while (true) {
    const { done, value } = await reader.read()
    if (done) break

    const chunk = decoder.decode(value)
    const lines = chunk.split('\n\n')

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const data = JSON.parse(line.slice(6))
        if (data.token) {
          setResponse(prev => prev + data.token)
        }
      }
    }
  }
}

BentoML Service

import bentoml
from bentoml.io import JSON
import numpy as np

@bentoml.service(
    resources={"cpu": "2", "memory": "4Gi"},
    traffic={"timeout": 10}
)
class IrisClassifier:
    model_ref = bentoml.models.get("iris_classifier:latest")

    def __init__(self):
        self.model = bentoml.sklearn.load_model(self.model_ref)

    @bentoml.api(batchable=True, max_batch_size=32)
    def classify(self, features: list[dict]) -> list[str]:
        X = np.array([[f['sepal_length'], f['sepal_width'],
                       f['petal_length'], f['petal_width']] for f in features])
        predictions = self.model.predict(X)
        return ['setosa', 'versicolor', 'virginica'][predictions]

LangChain RAG Pipeline

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Qdrant
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load and chunk documents
text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)

# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = Qdrant.from_documents(
    chunks,
    embeddings,
    url="http://localhost:6333",
    collection_name="docs"
)

# Create retrieval chain
llm = ChatOpenAI(model="gpt-4o")
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    return_source_documents=True
)

# Query
result = qa_chain({"query": "What is PagedAttention?"})

Performance Optimization

GPU Memory Estimation

Rule of thumb for LLMs:

GPU Memory (GB) = Model Parameters (B) × Precision (bytes) × 1.2

Examples:

• Llama-3.1-8B (FP16): 8B × 2 bytes × 1.2 = 19.2 GB
• Llama-3.1-70B (FP16): 70B × 2 bytes × 1.2 = 168 GB (requires 2-4 A100s)

Quantization reduces memory:

• FP16: 2 bytes per parameter
• INT8: 1 byte per parameter (2x memory reduction)
• INT4: 0.5 bytes per parameter (4x memory reduction)

vLLM Optimization

# Enable quantization (AWQ for 4-bit)
vllm serve TheBloke/Llama-3.1-8B-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.9

# Multi-GPU deployment (tensor parallelism)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9

Batching Strategies

Continuous batching (vLLM default):

• Dynamically adds/removes requests from batch
• Higher throughput than static batching
• No configuration needed

Adaptive batching (BentoML):

@bentoml.api(
    batchable=True,
    max_batch_size=32,
    max_latency_ms=1000  # Wait max 1s to fill batch
)
def predict(self, inputs: list[np.ndarray]) -> list[float]:
    # BentoML automatically batches requests
    return self.model.predict(np.array(inputs))

Production Deployment

Kubernetes Deployment

See examples/k8s-vllm-deployment/ for complete YAML manifests.

Key considerations:

• GPU resource requests: nvidia.com/gpu: 1
• Health checks: /health endpoint
• Horizontal Pod Autoscaling based on queue depth
• Persistent volume for model caching

API Gateway Pattern

For production, add rate limiting, authentication, and monitoring:

Kong Configuration:

services:
  - name: vllm-service
    url: http://vllm-llama-8b:8000
    plugins:
      - name: rate-limiting
        config:
          minute: 60  # 60 requests per minute per API key
      - name: key-auth
      - name: prometheus

Monitoring Metrics

Essential LLM metrics:

• Tokens per second (throughput)
• Time to first token (TTFT)
• Inter-token latency
• GPU utilization and memory
• Queue depth

Prometheus instrumentation:

from prometheus_client import Counter, Histogram

requests_total = Counter('llm_requests_total', 'Total requests')
tokens_generated = Counter('llm_tokens_generated', 'Total tokens')
request_duration = Histogram('llm_request_duration_seconds', 'Request duration')

@app.post("/chat")
async def chat(request):
    requests_total.inc()
    start = time.time()
    response = await generate(request)
    tokens_generated.inc(len(response.tokens))
    request_duration.observe(time.time() - start)
    return response

Integration Patterns

Frontend (ai-chat) Integration

This skill provides the backend serving layer for the ai-chat skill.

Flow:

Frontend (React) → API Gateway → vLLM Server → GPU Inference
     ↑                                                  ↓
     └─────────── SSE Stream (tokens) ─────────────────┘

See references/streaming-sse.md for complete implementation patterns.

RAG with Vector Databases

Architecture:

User Query → LangChain
              ├─> Vector DB (Qdrant) for retrieval
              ├─> Combine context + query
              └─> LLM (vLLM) for generation

See references/langchain-orchestration.md and examples/langchain-rag-qdrant/ for complete patterns.

Async Inference Queue

For batch processing or non-real-time inference:

Client → API → Message Queue (Celery) → Workers (vLLM) → Results DB

Useful for:

• Batch document processing
• Background summarization
• Non-interactive workflows

Benchmarking

Use scripts/benchmark_inference.py to measure the deployment:

python scripts/benchmark_inference.py \
  --endpoint http://localhost:8000/v1/chat/completions \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --concurrency 32 \
  --requests 1000

Outputs:

• Requests per second
• P50/P95/P99 latency
• Tokens per second
• GPU memory usage

Bundled Resources

Detailed Guides:

• references/vllm.md - vLLM setup, PagedAttention, optimization
• references/tgi.md - Text Generation Inference patterns
• references/bentoml.md - BentoML deployment patterns
• references/langchain-orchestration.md - LangChain RAG and agents
• references/inference-optimization.md - Quantization, batching, GPU tuning

Working Examples:

• examples/vllm-serving/ - Complete vLLM + FastAPI streaming setup
• examples/ollama-local/ - Local development with Ollama
• examples/langchain-agents/ - LangChain agent patterns

Utility Scripts:

• scripts/benchmark_inference.py - Throughput and latency benchmarking
• scripts/validate_model_config.py - Validate deployment configurations

Common Patterns

Migration from OpenAI API

vLLM provides OpenAI-compatible endpoints for easy migration:

# Before (OpenAI)
from openai import OpenAI
client = OpenAI(api_key="sk-...")

# After (vLLM)
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

# Same API calls work!
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}]
)

Multi-Model Serving

Route requests to different models based on task:

MODEL_ROUTING = {
    "small": "meta-llama/Llama-3.1-8B-Instruct",  # Fast, cheap
    "large": "meta-llama/Llama-3.1-70B-Instruct", # Accurate, expensive
    "code": "codellama/CodeLlama-34b-Instruct"    # Code-specific
}

@app.post("/chat")
async def chat(message: str, task: str = "small"):
    model = MODEL_ROUTING[task]
    # Route to appropriate vLLM instance

Cost Optimization

Track token usage:

import tiktoken

def estimate_cost(text: str, model: str, price_per_1k: float):
    encoding = tiktoken.encoding_for_model(model)
    tokens = len(encoding.encode(text))
    return (tokens / 1000) * price_per_1k

# Compare costs
openai_cost = estimate_cost(text, "gpt-4o", 0.005)  # $5 per 1M tokens
self_hosted_cost = 0  # Fixed GPU cost, unlimited tokens

Troubleshooting

Out of GPU memory:

• Reduce --max-model-len
• Lower --gpu-memory-utilization (try 0.8)
• Enable quantization (--quantization awq)
• Use smaller model variant

Low throughput:

• Increase --gpu-memory-utilization (try 0.95)
• Enable continuous batching (vLLM default)
• Check GPU utilization (should be >80%)
• Consider tensor parallelism for multi-GPU

High latency:

• Reduce batch size if using static batching
• Check network latency to GPU server
• Profile with scripts/benchmark_inference.py

Next Steps

1. Local Development: Start with examples/ollama-local/ for GPU-free testing 2. Production Setup: Deploy vLLM with examples/vllm-serving/ 3. RAG Integration: Add vector DB with examples/langchain-rag-qdrant/ 4. Kubernetes: Scale with examples/k8s-vllm-deployment/ 5. Monitoring: Add metrics with Prometheus and Grafana

vLLM Kubernetes Deployment

Complete Kubernetes deployment for vLLM model serving with autoscaling, GPU support, and production best practices.

Files

• deployment.yaml - vLLM deployment with GPU resources
• service.yaml - LoadBalancer service
• hpa.yaml - Horizontal Pod Autoscaler
• configmap.yaml - Model configuration
• ingress.yaml - Ingress with TLS

Quick Start

# 1. Create namespace
kubectl create namespace model-serving

# 2. Apply all manifests
kubectl apply -f . -n model-serving

# 3. Check deployment
kubectl get pods -n model-serving
kubectl logs -f deployment/vllm-llama -n model-serving

# 4. Test endpoint
kubectl port-forward service/vllm-llama 8000:8000 -n model-serving
curl http://localhost:8000/v1/models

Requirements

• Kubernetes cluster with GPU nodes (NVIDIA)
• nvidia-device-plugin installed
• 1x A100 (40GB) or 2x A10 GPUs per pod
• 50GB persistent volume for model cache

Scaling

# Manual scale
kubectl scale deployment vllm-llama --replicas=3 -n model-serving

# Autoscaling (HPA configured)
# Scales based on request queue depth

See individual YAML files for detailed configuration.

LangChain Agent Example

Demonstrates building ReAct agents with tools for document search, calculations, and API integration.

Features

• ReAct (Reasoning + Acting) agent pattern
• Tool integration (document search, calculator, weather)
• Multi-step reasoning
• Interactive mode

Prerequisites

1. Python 3.8+ 2. OpenAI API key (or vLLM server) 3. Qdrant vector database (optional, for document search)

Installation

# Install dependencies
pip install -r requirements.txt

# Set OpenAI API key
export OPENAI_API_KEY=your_key_here

# Or use vLLM (modify main.py to use vLLM endpoint)

Setup

Option 1: With Qdrant (Full Features)

# Start Qdrant
docker run -p 6333:6333 qdrant/qdrant

# Index documents (see ../langchain-rag-qdrant/ example)
python index_documents.py

# Run agent
python main.py

Option 2: Without Qdrant (Calculator + Weather Only)

# Run agent (document search will be disabled)
python main.py

Usage

Run Examples

python main.py

This will run several example queries: 1. Simple calculation 2. Document search 3. Multi-step reasoning (search + calculate) 4. Multiple tool usage

Interactive Mode

After examples, enter interactive mode:

Your query: What is PagedAttention and calculate 8B × 2 bytes

Example Agent Reasoning

Query: "What is PagedAttention? Then calculate GPU memory for 8B params × 2 bytes"

Agent Execution:

Thought: I need to first search for PagedAttention information
Action: SearchDocs
Action Input: PagedAttention

Observation: [Document search results about PagedAttention]

Thought: Now I need to calculate GPU memory
Action: Calculator
Action Input: 8000000000 * 2 / (1024**3)

Observation: 14.90 GB

Thought: I have all information needed
Final Answer: PagedAttention is a memory optimization technique...
GPU memory needed: approximately 14.9 GB.

Custom Tools

Add Your Own Tool

from langchain.tools import Tool

def my_custom_tool(input_str: str) -> str:
    """Your custom tool logic."""
    # Process input
    result = process(input_str)
    return result

tools.append(
    Tool(
        name="MyTool",
        func=my_custom_tool,
        description="Description of when to use this tool. Be specific!"
    )
)

Example: Database Query Tool

import sqlite3

def query_database(query: str) -> str:
    """Query SQLite database for user information."""
    conn = sqlite3.connect("users.db")
    cursor = conn.cursor()

    try:
        cursor.execute(query)
        results = cursor.fetchall()
        return str(results)
    except Exception as e:
        return f"Database error: {e}"
    finally:
        conn.close()

tools.append(
    Tool(
        name="DatabaseQuery",
        func=query_database,
        description="Query the user database. Input should be a valid SQL SELECT statement."
    )
)

Example: API Integration

import requests

def search_web(query: str) -> str:
    """Search the web using external API."""
    response = requests.get(
        "https://api.example.com/search",
        params={"q": query}
    )
    return response.json()["results"]

tools.append(
    Tool(
        name="WebSearch",
        func=search_web,
        description="Search the web for current information. Input should be a search query."
    )
)

Using with vLLM

Replace OpenAI with vLLM endpoint:

from langchain_openai import ChatOpenAI

# Instead of: llm = ChatOpenAI(model="gpt-4o")
llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
    model="meta-llama/Llama-3.1-8B-Instruct"
)

Agent Types

ReAct (Used in Example)

Best for:

• General-purpose reasoning
• Multi-step tasks
• Combining search + actions

How it works: 1. Thought: Reason about what to do 2. Action: Choose tool to use 3. Observation: See tool result 4. Repeat until answer found

Structured Tool Calling

For models with native tool calling (GPT-4o, Claude 3.5):

from langchain.agents import create_tool_calling_agent

agent = create_tool_calling_agent(llm, tools, prompt)

Advantages:

• More reliable tool selection
• Better error handling
• Faster execution

Plan-and-Execute

For complex multi-step tasks:

from langchain.agents import create_plan_and_execute_agent

agent = create_plan_and_execute_agent(llm, tools)

How it works: 1. Plan: Create step-by-step plan 2. Execute: Run each step with tools 3. Re-plan: Adjust based on results

Best Practices

1. Write Clear Tool Descriptions:

• Be specific about inputs and outputs
• Include examples in description
• Explain when to use the tool

2. Handle Errors in Tools:

   def my_tool(input_str: str) -> str:
       try:
           result = process(input_str)
           return result
       except Exception as e:
           return f"Error: {e}. Try a different approach."

3. Limit Agent Iterations:

   agent_executor = AgentExecutor(
       agent=agent,
       tools=tools,
       max_iterations=5,  # Prevent infinite loops
       handle_parsing_errors=True
   )

4. Monitor Token Usage:

   from langchain.callbacks import get_openai_callback

   with get_openai_callback() as cb:
       result = agent_executor.invoke({"input": query})
       print(f"Tokens: {cb.total_tokens}, Cost: ${cb.total_cost:.4f}")

Troubleshooting

Agent gets stuck in loop:

• Reduce max_iterations
• Improve tool descriptions
• Add explicit stop conditions

Tool selection is wrong:

• Make tool descriptions more specific
• Use structured tool calling
• Try different prompts

Out of context window:

• Use summarization tool
• Reduce max_iterations
• Use models with larger context

Resources

• LangChain Agents: https://python.langchain.com/docs/modules/agents/
• ReAct Paper: https://arxiv.org/abs/2210.03629
• LangChain Hub: https://smith.langchain.com/hub
• Tool Examples: https://python.langchain.com/docs/modules/tools/

LangChain RAG with Qdrant

Complete RAG (Retrieval-Augmented Generation) pipeline using LangChain and Qdrant vector database.

Files

• basic_rag.py - Simple RAG chain
• streaming_rag.py - Streaming responses with SSE
• hybrid_search.py - Vector + BM25 hybrid search
• requirements.txt - Python dependencies

Setup

# 1. Install dependencies
pip install -r requirements.txt

# 2. Start Qdrant
docker run -p 6333:6333 qdrant/qdrant

# 3. Set API keys
export OPENAI_API_KEY="your-key"
export VOYAGE_API_KEY="your-key"  # Optional, for better embeddings

# 4. Run examples
python basic_rag.py
python streaming_rag.py

Architecture

User Query
    ↓
Embedding (Voyage AI / OpenAI)
    ↓
Vector Search (Qdrant)
    ↓
Context + Query → LLM (OpenAI / vLLM)
    ↓
Streaming Response

Key Features

• Chunking: 512 tokens with 50-token overlap
• Embeddings: Voyage AI voyage-3 (1024d)
• Vector DB: Qdrant with hybrid search
• LLM: OpenAI GPT-4 or self-hosted vLLM
• Streaming: Server-Sent Events (SSE)

See individual Python files for detailed implementation.

Ollama Local Development Example

Simple local LLM serving for development without GPU requirements. Perfect for prototyping AI applications on laptops.

Features

• CPU-friendly (no GPU required)
• Simple REST API
• Streaming responses
• Conversational chat
• Multiple model support

Prerequisites

1. Ollama installed (download from https://ollama.com) 2. Python 3.8+

Installation

1. Install Ollama

macOS / Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download from https://ollama.com/download

2. Install Python Dependencies

pip install -r requirements.txt

Usage

1. Pull a Model

# Llama 3.1 8B (recommended)
ollama pull llama3.1:8b

# Smaller model for testing (faster)
ollama pull llama3.1:7b

# Larger model (better quality)
ollama pull llama3.1:70b

# List available models
ollama list

2. Start Ollama Server

# Usually auto-started, but can manually start with:
ollama serve

3. Run Example

python main.py

API Examples

Simple Generation

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.1:8b",
    "prompt": "Why is the sky blue?"
})

print(response.json()["response"])

Streaming

response = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": "llama3.1:8b", "prompt": "Count to 10", "stream": True},
    stream=True
)

for line in response.iter_lines():
    if line:
        data = json.loads(line)
        if not data.get("done"):
            print(data["response"], end="", flush=True)

Chat with History

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is machine learning?"}
]

response = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.1:8b",
    "messages": messages
})

print(response.json()["message"]["content"])

CLI Usage

Ollama also provides a CLI:

# Interactive chat
ollama run llama3.1:8b

# One-off generation
echo "Explain quantum computing" | ollama run llama3.1:8b

# List models
ollama list

# Remove model
ollama rm llama3.1:8b

# Show model info
ollama show llama3.1:8b

Integration with LangChain

from langchain_community.llms import Ollama

llm = Ollama(model="llama3.1:8b")

# Simple generation
response = llm.invoke("Explain vLLM in simple terms")
print(response)

# With chains
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

template = """You are a technical expert.

Question: {question}

Answer:"""

prompt = PromptTemplate(template=template, input_variables=["question"])
chain = LLMChain(llm=llm, prompt=prompt)

response = chain.run(question="What is PagedAttention?")
print(response)

Performance Tuning

Model Selection

• 7B models: Fast, good for testing (~4GB RAM)
• 8B models: Balanced quality/speed (~6GB RAM)
• 13B models: Better quality (~8GB RAM)
• 70B models: Best quality, slow (~40GB RAM)

Parameters

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.1:8b",
    "prompt": "Your prompt",
    "options": {
        "temperature": 0.7,      # Randomness (0-1)
        "top_p": 0.9,           # Nucleus sampling
        "num_ctx": 2048,        # Context window
        "num_predict": 512,     # Max tokens to generate
    }
})

Use Cases

Perfect for:

• Local development and testing
• Prototyping AI applications
• Learning LLM concepts
• Privacy-sensitive applications (data never leaves machine)
• Offline development

Not suitable for:

• Production high-throughput serving
• Multi-user applications (use vLLM instead)
• Maximum performance (no GPU optimization)

Migration to vLLM

When ready for production, migrate to vLLM:

# Before (Ollama)
import requests
response = requests.post("http://localhost:11434/api/generate", ...)

# After (vLLM with OpenAI API)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(...)

Troubleshooting

Ollama not starting:

# Check if running
curl http://localhost:11434/api/tags

# Restart
killall ollama
ollama serve

Out of memory:

• Use smaller model (7B instead of 70B)
• Reduce num_ctx parameter
• Close other applications

Slow generation:

• Expected for CPU inference
• Use smaller model
• Consider upgrading to vLLM with GPU

Resources

• Ollama Website: https://ollama.com
• Ollama GitHub: https://github.com/ollama/ollama
• Model Library: https://ollama.com/library
• API Documentation: https://github.com/ollama/ollama/blob/main/docs/api.md

vLLM + FastAPI Streaming Example

Complete implementation of streaming LLM responses using vLLM and FastAPI with Server-Sent Events (SSE).

Features

• Server-Sent Events (SSE) streaming
• OpenAI-compatible API
• CORS support for frontend integration
• Health checks for load balancers
• Production-ready error handling
• Logging and monitoring

Prerequisites

1. GPU with CUDA support (16GB+ VRAM for Llama-3.1-8B) 2. Python 3.8+ 3. vLLM installed separately

Installation

# Install dependencies
pip install -r requirements.txt

# Install vLLM (separate installation)
pip install vllm

Usage

1. Start vLLM Server

# Basic
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

# Production
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype float16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --port 8000

2. Start FastAPI Server

# Development
python main.py

# Production (with Gunicorn)
gunicorn main:app \
  --workers 4 \
  --worker-class uvicorn.workers.UvicornWorker \
  --bind 0.0.0.0:3000

3. Test Endpoints

Health check:

curl http://localhost:3000/health

Non-streaming chat:

curl -X POST http://localhost:3000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Explain quantum computing in simple terms",
    "temperature": 0.7,
    "max_tokens": 256
  }'

Streaming chat:

curl -X POST http://localhost:3000/chat/stream \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Write a poem about AI",
    "temperature": 0.9,
    "max_tokens": 512
  }'

List models:

curl http://localhost:3000/models

Frontend Integration

React Example

import { useState } from 'react'

export function useStreamingChat() {
  const [response, setResponse] = useState('')
  const [isStreaming, setIsStreaming] = useState(false)

  const sendMessage = async (message: string) => {
    setIsStreaming(true)
    setResponse('')

    const res = await fetch('http://localhost:3000/chat/stream', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ message })
    })

    const reader = res.body!.getReader()
    const decoder = new TextDecoder()

    while (true) {
      const { done, value } = await reader.read()
      if (done) break

      const chunk = decoder.decode(value)
      const lines = chunk.split('\n\n')

      for (const line of lines) {
        if (line.startsWith('data: ')) {
          const data = JSON.parse(line.slice(6))

          if (data.done) {
            setIsStreaming(false)
          } else if (data.token) {
            setResponse(prev => prev + data.token)
          }
        }
      }
    }
  }

  return { response, isStreaming, sendMessage }
}

Next.js API Route

// app/api/chat/route.ts
export async function POST(request: Request) {
  const { message } = await request.json()

  const response = await fetch('http://localhost:3000/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message })
  })

  // Forward SSE stream
  return new Response(response.body, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive'
    }
  })
}

Production Deployment

Docker Compose

version: '3.8'

services:
  vllm:
    image: vllm/vllm-openai:latest
    command:
      - --model
      - meta-llama/Llama-3.1-8B-Instruct
      - --dtype
      - float16
      - --gpu-memory-utilization
      - "0.9"
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  api:
    build: .
    ports:
      - "3000:3000"
    environment:
      - VLLM_URL=http://vllm:8000
    depends_on:
      - vllm

Kubernetes

See ../k8s-vllm-deployment/ for complete Kubernetes manifests.

Performance Tuning

Optimize vLLM:

• Increase --gpu-memory-utilization to 0.95 for maximum throughput
• Use quantization for memory-constrained GPUs: --quantization awq
• Enable tensor parallelism for multi-GPU: --tensor-parallel-size 2

Optimize FastAPI:

• Use Gunicorn with multiple workers
• Add response caching for repeated queries
• Implement request queuing for rate limiting

Monitoring

Prometheus Metrics

Add to main.py:

from prometheus_client import Counter, Histogram, make_asgi_app

requests_total = Counter('api_requests_total', 'Total requests')
request_duration = Histogram('api_request_duration_seconds', 'Request duration')

# Mount metrics endpoint
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

Grafana Dashboard

Monitor:

• Requests per second
• P50/P95/P99 latency
• Error rate
• Active connections
• Token generation rate

Troubleshooting

vLLM connection failed:

• Verify vLLM is running: curl http://localhost:8000/health
• Check vLLM logs for errors
• Ensure correct port (default: 8000)

Streaming not working:

• Disable nginx buffering: proxy_buffering off;
• Check browser console for CORS errors
• Verify SSE format with curl

Out of memory:

• Reduce --max-model-len in vLLM
• Lower --gpu-memory-utilization to 0.85
• Use quantization (AWQ, GPTQ)

Resources

• vLLM Docs: https://docs.vllm.ai/
• FastAPI Docs: https://fastapi.tiangolo.com/
• SSE Spec: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events

BentoML - ML Model Deployment Made Easy

Table of Contents

• Overview
• Installation
• Basic Workflow
• 1. Train and Save Model
• 2. Create Service
• 3. Serve Locally
• 4. Build and Deploy
• Adaptive Batching
• How It Works
• Configuration
• Multi-Framework Support
• PyTorch
• XGBoost
• TensorFlow
• Configuration
• Service Configuration
• bentofile.yaml
• Deployment Patterns
• Docker
• Kubernetes
• AWS Lambda
• Observability
• Built-in Metrics
• Logging
• Distributed Tracing
• Testing
• Best Practices
• Resources

Overview

BentoML is a Python-native framework for building production-ready ML model serving APIs with adaptive batching, multi-framework support, and easy deployment to Kubernetes, AWS, GCP, and Azure.

Key Features:

• Python-first design (feels native for data scientists)
• Adaptive batching for throughput optimization
• Multi-framework: scikit-learn, PyTorch, TensorFlow, XGBoost, LightGBM
• Deploy anywhere: Docker, Kubernetes, AWS Lambda, GCP Cloud Run
• Built-in observability: metrics, logging, tracing

Installation

# Core BentoML
pip install bentoml

# Framework-specific
pip install bentoml[sklearn]
pip install bentoml[pytorch]
pip install bentoml[tensorflow]
pip install bentoml[xgboost]

Basic Workflow

1. Train and Save Model

import bentoml
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

# Train model
X, y = load_iris(return_X_y=True)
model = RandomForestClassifier()
model.fit(X, y)

# Save to BentoML model store
bentoml.sklearn.save_model(
    "iris_classifier",
    model,
    signatures={"predict": {"batchable": True}},
    metadata={
        "framework": "scikit-learn",
        "accuracy": 0.96,
        "created_at": "2025-12-02"
    }
)

2. Create Service

# service.py
import bentoml
import numpy as np
from pydantic import BaseModel

class IrisFeatures(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float

@bentoml.service(
    resources={"cpu": "2", "memory": "4Gi"},
    traffic={"timeout": 10}
)
class IrisClassifier:
    model_ref = bentoml.models.get("iris_classifier:latest")

    def __init__(self):
        self.model = bentoml.sklearn.load_model(self.model_ref)
        self.class_names = ['setosa', 'versicolor', 'virginica']

    @bentoml.api(batchable=True, max_batch_size=32, max_latency_ms=1000)
    def classify(self, features: list[IrisFeatures]) -> list[str]:
        # Convert to numpy array
        X = np.array([[
            f.sepal_length,
            f.sepal_width,
            f.petal_length,
            f.petal_width
        ] for f in features])

        # Predict
        predictions = self.model.predict(X)

        # Map to class names
        return [self.class_names[p] for p in predictions]

    @bentoml.api
    def predict_proba(self, features: IrisFeatures) -> dict[str, float]:
        X = np.array([[
            features.sepal_length,
            features.sepal_width,
            features.petal_length,
            features.petal_width
        ]])

        probas = self.model.predict_proba(X)[0]

        return {
            self.class_names[i]: float(probas[i])
            for i in range(len(self.class_names))
        }

3. Serve Locally

# Development server
bentoml serve service:IrisClassifier

# Production server (with workers)
bentoml serve service:IrisClassifier --production

4. Build and Deploy

# Build Bento (packaged service)
bentoml build

# Containerize
bentoml containerize iris_classifier:latest

# Deploy to Kubernetes
kubectl apply -f deployment.yaml

# Or deploy to BentoCloud (managed)
bentoml deploy iris_classifier:latest

Adaptive Batching

BentoML's killer feature: automatic request batching for throughput.

How It Works

Without Batching:
Request 1 → Model (10ms)
Request 2 → Model (10ms)  ← Wait for Request 1
Request 3 → Model (10ms)  ← Wait for Request 2
Total: 30ms for 3 requests

With Adaptive Batching:
Request 1 ─┐
Request 2 ─┼→ Batch → Model (12ms)
Request 3 ─┘
Total: 12ms for 3 requests (2.5x faster)

Configuration

@bentoml.api(
    batchable=True,
    max_batch_size=32,      # Maximum batch size
    max_latency_ms=1000     # Maximum wait time to fill batch
)
def predict(self, inputs: list[np.ndarray]) -> list[float]:
    # BentoML automatically batches requests
    batch = np.array(inputs)
    return self.model.predict(batch).tolist()

Tuning Parameters:

• max_batch_size: Larger = higher throughput, but higher latency
• max_latency_ms: Shorter = lower latency, but smaller batches

Rule of thumb:

• High throughput service: max_batch_size=64, max_latency_ms=2000
• Low latency service: max_batch_size=8, max_latency_ms=100

Multi-Framework Support

PyTorch

import bentoml
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Save PyTorch model
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
bentoml.pytorch.save_model("bert_classifier", model)

# Service
@bentoml.service
class BertClassifier:
    model_ref = bentoml.models.get("bert_classifier:latest")

    def __init__(self):
        self.model = bentoml.pytorch.load_model(self.model_ref)
        self.tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

    @bentoml.api
    def classify(self, text: str) -> dict:
        inputs = self.tokenizer(text, return_tensors="pt", padding=True)
        with torch.no_grad():
            outputs = self.model(**inputs)
        return {"logits": outputs.logits.tolist()}

XGBoost

import bentoml
import xgboost as xgb

# Save XGBoost model
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
bentoml.xgboost.save_model("fraud_detector", model)

# Service
@bentoml.service
class FraudDetector:
    model_ref = bentoml.models.get("fraud_detector:latest")

    def __init__(self):
        self.model = bentoml.xgboost.load_model(self.model_ref)

    @bentoml.api(batchable=True, max_batch_size=128)
    def predict(self, transactions: list[dict]) -> list[bool]:
        import pandas as pd
        df = pd.DataFrame(transactions)
        predictions = self.model.predict(df)
        return predictions.tolist()

TensorFlow

import bentoml
import tensorflow as tf

# Save TensorFlow model
model = tf.keras.models.Sequential([...])
bentoml.tensorflow.save_model("image_classifier", model)

# Service
@bentoml.service
class ImageClassifier:
    model_ref = bentoml.models.get("image_classifier:latest")

    def __init__(self):
        self.model = bentoml.tensorflow.load_model(self.model_ref)

    @bentoml.api
    def classify(self, image: np.ndarray) -> dict:
        predictions = self.model.predict(np.expand_dims(image, 0))
        return {"class": int(np.argmax(predictions))}

Configuration

Service Configuration

@bentoml.service(
    # Resource allocation
    resources={
        "cpu": "2",           # 2 CPU cores
        "memory": "4Gi",      # 4GB RAM
        "gpu": 1,             # 1 GPU (optional)
        "gpu_type": "nvidia-tesla-t4"
    },

    # Traffic settings
    traffic={
        "timeout": 30,        # Request timeout (seconds)
        "concurrency": 32     # Max concurrent requests
    },

    # Workers (production mode)
    workers=4,               # Number of worker processes

    # Logging
    logging={
        "access": {
            "enabled": True,
            "request_content_length": True,
            "response_content_length": True
        }
    }
)
class MyService:
    ...

bentofile.yaml

For build-time configuration:

service: "service:IrisClassifier"
include:
  - "service.py"
  - "preprocessing.py"
python:
  packages:
    - scikit-learn==1.3.0
    - pandas==2.0.0
    - numpy==1.24.0
docker:
  distro: debian
  python_version: "3.11"
  system_packages:
    - git
  env:
    MODEL_NAME: "iris_classifier"

Deployment Patterns

Docker

# Build container
bentoml containerize iris_classifier:latest -t iris:v1

# Run locally
docker run -p 3000:3000 iris:v1

# Push to registry
docker tag iris:v1 myregistry.io/iris:v1
docker push myregistry.io/iris:v1

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iris-classifier
spec:
  replicas: 3
  selector:
    matchLabels:
      app: iris-classifier
  template:
    metadata:
      labels:
        app: iris-classifier
    spec:
      containers:
      - name: iris
        image: myregistry.io/iris:v1
        ports:
        - containerPort: 3000
        resources:
          requests:
            memory: "2Gi"
            cpu: "1"
          limits:
            memory: "4Gi"
            cpu: "2"
        livenessProbe:
          httpGet:
            path: /livez
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /readyz
            port: 3000
          initialDelaySeconds: 10
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: iris-classifier
spec:
  selector:
    app: iris-classifier
  ports:
  - port: 80
    targetPort: 3000
  type: LoadBalancer

AWS Lambda

# Build for Lambda
bentoml build --containerize

# Deploy to AWS
aws ecr get-login-password | docker login --username AWS --password-stdin <account>.dkr.ecr.us-east-1.amazonaws.com
docker tag iris:latest <account>.dkr.ecr.us-east-1.amazonaws.com/iris:latest
docker push <account>.dkr.ecr.us-east-1.amazonaws.com/iris:latest

# Create Lambda function
aws lambda create-function \
  --function-name iris-classifier \
  --package-type Image \
  --code ImageUri=<account>.dkr.ecr.us-east-1.amazonaws.com/iris:latest \
  --role arn:aws:iam::<account>:role/lambda-execution-role

Observability

Built-in Metrics

BentoML exposes Prometheus metrics at /metrics:

Key Metrics:

• bentoml_request_duration_seconds - Request latency histogram
• bentoml_request_total - Total requests counter
• bentoml_request_in_progress - Current active requests
• bentoml_runner_adaptive_batch_size - Current batch size

Logging

import logging

logger = logging.getLogger(__name__)

@bentoml.service
class MyService:
    @bentoml.api
    def predict(self, data: dict):
        logger.info(f"Received request: {data}")
        result = self.model.predict(data)
        logger.info(f"Prediction: {result}")
        return result

Distributed Tracing

from opentelemetry import trace

@bentoml.service
class MyService:
    @bentoml.api
    def predict(self, data: dict):
        tracer = trace.get_tracer(__name__)
        with tracer.start_as_current_span("preprocessing"):
            processed = preprocess(data)
        with tracer.start_as_current_span("inference"):
            result = self.model.predict(processed)
        return result

Testing

import bentoml
from bentoml.testing.server import serve

def test_classifier():
    with serve("service:IrisClassifier") as client:
        # Test classify endpoint
        response = client.classify(
            features=[{
                "sepal_length": 5.1,
                "sepal_width": 3.5,
                "petal_length": 1.4,
                "petal_width": 0.2
            }]
        )
        assert response == ["setosa"]

        # Test probability endpoint
        response = client.predict_proba(
            features={
                "sepal_length": 5.1,
                "sepal_width": 3.5,
                "petal_length": 1.4,
                "petal_width": 0.2
            }
        )
        assert "setosa" in response
        assert response["setosa"] > 0.9

Best Practices

1. Use Pydantic for Input Validation:

   from pydantic import BaseModel, validator

   class Features(BaseModel):
       value: float

       @validator('value')
       def check_range(cls, v):
           if not 0 <= v <= 100:
               raise ValueError('value must be between 0 and 100')
           return v

2. Enable Adaptive Batching for High Throughput:

   @bentoml.api(batchable=True, max_batch_size=64)
   def predict(self, inputs: list[np.ndarray]) -> list[float]:
       ...

3. Set Resource Limits:

   @bentoml.service(resources={"cpu": "2", "memory": "4Gi"})

4. Add Health Checks: BentoML provides /healthz, /livez, /readyz automatically

5. Version Models:

   bentoml.sklearn.save_model("classifier", model, version="v1.2.0")

6. Monitor Metrics: Configure Prometheus scraping of /metrics endpoint

Resources

• BentoML Documentation: https://docs.bentoml.com/
• GitHub: https://github.com/bentoml/BentoML
• Examples: https://github.com/bentoml/BentoML/tree/main/examples
• BentoCloud: https://bentoml.com/ (managed deployment)

Inference Optimization Guide

Table of Contents

• Overview
• Quantization Techniques
• LLM Quantization
• AWQ (Activation-aware Weight Quantization)
• GPTQ
• FP8 Quantization (H100 GPUs)
• Batching Strategies
• Continuous Batching (vLLM)
• Adaptive Batching (BentoML)
• Manual Batching
• GPU Optimization
• Memory Management
• Tensor Parallelism
• Pipeline Parallelism
• KV Cache Optimization
• PagedAttention (vLLM)
• KV Cache Quantization
• Attention Optimization
• Flash Attention
• Multi-Query Attention (MQA)
• Kernel Optimization
• Custom CUDA Kernels (vLLM)
• TensorRT-LLM
• Caching Strategies
• Response Caching
• Prefix Caching
• Embedding Caching
• Profiling and Monitoring
• GPU Monitoring
• vLLM Metrics
• Profiling with PyTorch
• Benchmarking
• Throughput Benchmark
• Latency Benchmark
• Production Optimization Checklist
• Resources

Overview

Optimize LLM and ML model inference for production through quantization, batching, caching, and GPU tuning.

Quantization Techniques

LLM Quantization

Reduce model size and memory usage with minimal accuracy loss.

Quantization Methods:

AWQ (Activation-aware Weight Quantization)

How it works: Protects important weights, quantizes less important ones more aggressively.

Use with vLLM:

# Use pre-quantized model
vllm serve TheBloke/Llama-3.1-8B-AWQ \
  --quantization awq \
  --dtype auto

Quantize your own model:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "llama-3.1-8b-awq"

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Quantize (requires calibration data)
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}

model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

GPTQ

Similar to AWQ, different algorithm:

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_path = "meta-llama/Llama-3.1-8B-Instruct"

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False
)

model = AutoGPTQForCausalLM.from_pretrained(
    model_path,
    quantize_config=quantize_config
)

model.quantize(calibration_dataset)
model.save_quantized("llama-3.1-8b-gptq")

FP8 Quantization (H100 GPUs)

Maximum performance on NVIDIA H100:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --dtype auto

Performance: 1.8-2.5x faster than FP16 with <0.5% accuracy loss.

Batching Strategies

Continuous Batching (vLLM)

Default in vLLM. No configuration needed.

How it works:

Static Batching (old way):
Batch 1: [Req1, Req2, Req3, Req4] → Generate until ALL finish
Batch 2: [Req5, Req6, Req7, Req8] → Wait for Batch 1

Continuous Batching (vLLM):
[Req1, Req2, Req3, Req4] → Req1 finishes → Add Req5
[Req2, Req3, Req4, Req5] → Req3 finishes → Add Req6
[Req2, Req4, Req5, Req6] → ...

Throughput improvement: 2-3x vs static batching

Adaptive Batching (BentoML)

Automatically batches requests with configurable trade-offs:

@bentoml.api(
    batchable=True,
    max_batch_size=32,      # Maximum batch size
    max_latency_ms=1000     # Maximum wait time
)
def predict(self, inputs: list[np.ndarray]) -> list[float]:
    # BentoML automatically batches
    batch = np.array(inputs)
    return self.model.predict(batch).tolist()

Tuning:

• High throughput: max_batch_size=64, max_latency_ms=2000
• Low latency: max_batch_size=8, max_latency_ms=100

Manual Batching

For custom serving:

import asyncio
from collections import deque
from typing import List

class BatchProcessor:
    def __init__(self, model, max_batch_size=32, max_wait_ms=100):
        self.model = model
        self.max_batch_size = max_batch_size
        self.max_wait_ms = max_wait_ms
        self.queue = deque()

    async def add_request(self, input_data):
        future = asyncio.Future()
        self.queue.append((input_data, future))

        # Trigger batch processing
        asyncio.create_task(self.process_batch())

        return await future

    async def process_batch(self):
        # Wait for batch to fill or timeout
        await asyncio.sleep(self.max_wait_ms / 1000)

        if not self.queue:
            return

        # Collect batch
        batch_size = min(len(self.queue), self.max_batch_size)
        batch = [self.queue.popleft() for _ in range(batch_size)]

        inputs = [item[0] for item in batch]
        futures = [item[1] for item in batch]

        # Process batch
        results = self.model.predict(inputs)

        # Return results
        for future, result in zip(futures, results):
            future.set_result(result)

GPU Optimization

Memory Management

Estimate GPU memory:

def estimate_gpu_memory(num_params_billions, precision="fp16"):
    """Estimate GPU memory for LLM.

    Args:
        num_params_billions: Model parameters in billions
        precision: fp32, fp16, int8, int4
    """
    bytes_per_param = {
        "fp32": 4,
        "fp16": 2,
        "int8": 1,
        "int4": 0.5
    }

    # Model weights
    model_memory_gb = num_params_billions * bytes_per_param[precision]

    # KV cache and activations (1.2x overhead)
    total_memory_gb = model_memory_gb * 1.2

    return total_memory_gb

# Llama-3.1-8B in FP16
print(estimate_gpu_memory(8, "fp16"))  # ~19.2 GB

# Llama-3.1-70B in INT4
print(estimate_gpu_memory(70, "int4"))  # ~42 GB

vLLM GPU tuning:

# Maximum utilization (throughput)
vllm serve model \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 256

# Conservative (stability)
vllm serve model \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 128

# Multi-GPU
vllm serve model \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9

Tensor Parallelism

Split model across multiple GPUs:

# Llama-3.1-70B on 4x A100 (40GB each)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

# Or 2x A100 (80GB each)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2

When to use:

• Model doesn't fit on single GPU
• Each GPU gets 1/N of model weights
• Linear scaling up to ~8 GPUs

Pipeline Parallelism

Split model layers across GPUs:

# 70B model on 8 GPUs (4-way pipeline, 2-way tensor)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --pipeline-parallel-size 4 \
  --tensor-parallel-size 2

Trade-offs:

• Better for very large models
• Pipeline bubbles reduce efficiency
• More complex than tensor parallelism

KV Cache Optimization

PagedAttention (vLLM)

Automatically enabled. No configuration needed.

How it works:

• Stores key-value cache in paged memory blocks
• Eliminates fragmentation
• Near-zero memory waste

Impact: 2-4x memory efficiency vs traditional caching

KV Cache Quantization

Reduce cache memory usage:

# INT8 KV cache
vllm serve model --kv-cache-dtype int8

# FP8 KV cache (H100)
vllm serve model --kv-cache-dtype fp8

Memory reduction: 2x with minimal quality impact

Attention Optimization

Flash Attention

Automatically used by vLLM when available.

Benefits:

• 2-4x faster attention computation
• Lower memory usage
• Exact attention (not approximate)

Requirements:

• NVIDIA Ampere+ GPU (A100, H100, RTX 3090+)
• Installed automatically with vLLM

Multi-Query Attention (MQA)

Model architecture feature (e.g., Falcon models).

Benefits:

• Reduced KV cache size
• Faster inference
• Minimal accuracy impact

Kernel Optimization

Custom CUDA Kernels (vLLM)

vLLM includes optimized kernels for:

• PagedAttention
• Rotary position embeddings
• Layer normalization
• Activation functions

No configuration needed. Automatically used.

TensorRT-LLM

Maximum optimization (requires model conversion):

# 1. Convert model
python convert_checkpoint.py \
  --model_dir ./llama-3.1-8b \
  --output_dir ./trt-checkpoint \
  --dtype float16

# 2. Build TensorRT engine
trtllm-build \
  --checkpoint_dir ./trt-checkpoint \
  --output_dir ./trt-engine \
  --gemm_plugin float16 \
  --max_batch_size 256

# 3. Serve
tritonserver --model-repository=./model_repo

Performance: 2-8x faster than vLLM, but more complex setup.

Caching Strategies

Response Caching

Cache complete responses for repeated queries:

from functools import lru_cache

@lru_cache(maxsize=1000)
def cached_inference(prompt: str, temperature: float = 0.0):
    """Cache responses for identical prompts."""
    return model.generate(prompt, temperature=temperature)

Best for:

• Temperature = 0 (deterministic)
• Repeated identical queries
• FAQ-style applications

Prefix Caching

Cache common prompt prefixes:

# Example: System prompt is constant
SYSTEM_PROMPT = "You are a helpful assistant..."

# vLLM automatically shares KV cache for common prefixes
# across requests with same system prompt

Enabled automatically in vLLM.

Embedding Caching

Cache embeddings for RAG:

from langchain.embeddings import CacheBackedEmbeddings
from langchain.storage import LocalFileStore

store = LocalFileStore("./embeddings_cache/")
cached_embedder = CacheBackedEmbeddings.from_bytes_store(
    OpenAIEmbeddings(),
    store,
    namespace="docs-v1"
)

Profiling and Monitoring

GPU Monitoring

# Real-time GPU stats
nvidia-smi dmon -s pucvmet -d 1

# Watch specific GPU
nvidia-smi -i 0 dmon

# Log to file
nvidia-smi dmon -s pucvmet -d 5 -o T > gpu_metrics.log

Key metrics:

• GPU utilization (should be >80%)
• Memory usage
• Temperature
• Power consumption

vLLM Metrics

import requests

# Prometheus metrics
response = requests.get("http://localhost:8000/metrics")
metrics = response.text

# Parse key metrics
for line in metrics.split('\n'):
    if 'vllm:' in line and not line.startswith('#'):
        print(line)

Important metrics:

• vllm:time_to_first_token_seconds - TTFT latency
• vllm:time_per_output_token_seconds - Token generation speed
• vllm:num_requests_waiting - Queue depth
• vllm:gpu_cache_usage_perc - KV cache utilization

Profiling with PyTorch

import torch
from torch.profiler import profile, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
    with_stack=True
) as prof:
    outputs = model.generate(inputs)

# Print summary
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

# Export Chrome trace
prof.export_chrome_trace("trace.json")

Benchmarking

Throughput Benchmark

import time
import asyncio

async def benchmark_throughput(endpoint, num_requests=100):
    """Measure requests per second."""
    start = time.time()

    tasks = [
        asyncio.create_task(
            send_request(endpoint, f"Test prompt {i}")
        )
        for i in range(num_requests)
    ]

    await asyncio.gather(*tasks)

    duration = time.time() - start
    rps = num_requests / duration

    print(f"Throughput: {rps:.2f} req/s")
    print(f"Duration: {duration:.2f}s")

Latency Benchmark

import numpy as np

def benchmark_latency(endpoint, num_requests=100):
    """Measure latency percentiles."""
    latencies = []

    for i in range(num_requests):
        start = time.time()
        send_request(endpoint, f"Test prompt {i}")
        latency = time.time() - start
        latencies.append(latency)

    latencies = np.array(latencies) * 1000  # Convert to ms

    print(f"P50 latency: {np.percentile(latencies, 50):.2f}ms")
    print(f"P95 latency: {np.percentile(latencies, 95):.2f}ms")
    print(f"P99 latency: {np.percentile(latencies, 99):.2f}ms")
    print(f"Mean latency: {np.mean(latencies):.2f}ms")

Production Optimization Checklist

LLM Serving (vLLM):

• [ ] Use FP16 or quantization (AWQ, INT8)
• [ ] Set --gpu-memory-utilization 0.9
• [ ] Enable tensor parallelism for large models
• [ ] Monitor GPU utilization (>80%)
• [ ] Track queue depth (scale if >10)
• [ ] Cache common prompt prefixes

ML Model Serving (BentoML):

• [ ] Enable adaptive batching
• [ ] Tune max_batch_size and max_latency_ms
• [ ] Set resource limits (CPU, memory)
• [ ] Monitor batch sizes
• [ ] Use multiple workers

General:

• [ ] Add response caching for repeated queries
• [ ] Implement request queuing
• [ ] Set up monitoring (Prometheus, Grafana)
• [ ] Profile with PyTorch profiler
• [ ] Benchmark before deploying
• [ ] Load test at expected RPS

Resources

• vLLM Performance: https://docs.vllm.ai/en/latest/performance/
• PagedAttention Paper: https://arxiv.org/abs/2309.06180
• Flash Attention: https://github.com/Dao-AILab/flash-attention
• AWQ: https://github.com/mit-han-lab/llm-awq
• TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM

LangChain Orchestration for LLM Applications

Table of Contents

• Overview
• Installation
• Basic RAG Pipeline
• Complete Example
• Chain Types for RAG
• 1. Stuff Chain (Recommended for Most Cases)
• 2. Map-Reduce Chain
• 3. Refine Chain
• 4. Map-Rerank Chain
• Advanced Retrieval
• Hybrid Search (Keyword + Semantic)
• Re-ranking
• Multi-Query Retrieval
• Conversational RAG
• Agents
• ReAct Agent (Recommended)
• Structured Tool Calling
• Streaming Responses
• Custom Chains
• Error Handling
• Integration with vLLM
• Best Practices
• Resources

Overview

LangChain is a framework for building applications with LLMs through composable components: chains, agents, retrievers, and tools. Supports RAG (Retrieval-Augmented Generation), multi-step reasoning, and tool use.

Key Use Cases:

• RAG pipelines (document Q&A)
• Conversational agents with memory
• Multi-step reasoning (ReAct, Plan-and-Execute)
• Tool integration (search, calculators, APIs)

Installation

# Core
pip install langchain

# LLM providers
pip install langchain-openai      # OpenAI, Azure OpenAI
pip install langchain-anthropic   # Anthropic Claude
pip install langchain-google      # Google (Gemini, Vertex AI)

# Vector stores
pip install langchain-qdrant
pip install langchain-chroma
pip install langchain-pinecone

# Community integrations
pip install langchain-community

Basic RAG Pipeline

Complete Example

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Qdrant
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader, PyPDFLoader
from qdrant_client import QdrantClient

# 1. Load documents
loader = TextLoader("./data/documents.txt")
documents = loader.load()

# For PDFs
pdf_loader = PyPDFLoader("./data/manual.pdf")
pdf_docs = pdf_loader.load()

# 2. Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    length_function=len,
    separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)

# 3. Create embeddings and vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

client = QdrantClient(url="http://localhost:6333")
vectorstore = Qdrant.from_documents(
    chunks,
    embeddings,
    url="http://localhost:6333",
    collection_name="documents",
    force_recreate=True
)

# 4. Create retrieval chain
llm = ChatOpenAI(model="gpt-4o", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # stuff, map_reduce, refine, map_rerank
    retriever=vectorstore.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 3}
    ),
    return_source_documents=True
)

# 5. Query
result = qa_chain.invoke({"query": "What is PagedAttention?"})

print(f"Answer: {result['result']}")
print(f"\nSources:")
for doc in result['source_documents']:
    print(f"- {doc.metadata.get('source', 'Unknown')}: {doc.page_content[:100]}...")

Chain Types for RAG

1. Stuff Chain (Recommended for Most Cases)

Concatenates all retrieved documents into a single prompt.

Pros:

• Simple and fast
• Single LLM call
• Works well when documents fit in context

Cons:

• Limited by context window
• Fails if total text > max tokens

When to use: 3-5 documents, each <1000 tokens

2. Map-Reduce Chain

Processes documents individually, then combines results.

Pros:

• Handles large document sets
• Parallel processing possible
• No context limit

Cons:

• Multiple LLM calls (expensive)
• Slower than stuff
• May lose cross-document context

When to use: 10+ documents or documents > context window

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="map_reduce",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 10})
)

3. Refine Chain

Iteratively refines answer by processing documents sequentially.

Pros:

• Better synthesis than map-reduce
• Handles large document sets

Cons:

• Many sequential LLM calls (slow, expensive)
• Later documents may dominate

When to use: Need high-quality synthesis of many documents

4. Map-Rerank Chain

Scores each document's relevance, uses highest-scoring.

Pros:

• Good for finding specific info
• Handles irrelevant retrievals

Cons:

• May miss multi-document answers

When to use: Looking for single best answer from many candidates

Advanced Retrieval

Hybrid Search (Keyword + Semantic)

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# Semantic retriever
semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# Keyword retriever
keyword_retriever = BM25Retriever.from_documents(chunks)
keyword_retriever.k = 5

# Combine with weights
ensemble_retriever = EnsembleRetriever(
    retrievers=[semantic_retriever, keyword_retriever],
    weights=[0.7, 0.3]  # 70% semantic, 30% keyword
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=ensemble_retriever
)

Re-ranking

Improve retrieval quality by re-ranking results:

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CohereRerank

# Base retriever (gets 10 candidates)
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

# Re-ranker (selects top 3)
compressor = CohereRerank(model="rerank-english-v2.0", top_n=3)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=compression_retriever
)

Multi-Query Retrieval

Generate multiple query variations for better recall:

from langchain.retrievers.multi_query import MultiQueryRetriever

multi_query_retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(),
    llm=llm
)

# User query: "How does vLLM work?"
# LLM generates variations:
# - "Explain vLLM architecture"
# - "What are vLLM's key features?"
# - "vLLM performance optimizations"
# Retrieves for all variations and deduplicates

Conversational RAG

Add memory to maintain conversation context:

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

# Create memory
memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True,
    output_key="answer"
)

# Conversational chain
conv_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    memory=memory,
    return_source_documents=True
)

# First turn
result1 = conv_chain.invoke({"question": "What is PagedAttention?"})
print(result1["answer"])

# Follow-up (uses conversation history)
result2 = conv_chain.invoke({"question": "How does it improve throughput?"})
print(result2["answer"])  # References PagedAttention from context

Agents

Agents use LLMs to decide which tools to use and in what order.

ReAct Agent (Recommended)

from langchain.agents import create_react_agent, AgentExecutor
from langchain.tools import Tool
from langchain_openai import ChatOpenAI
from langchain import hub

# Define tools
def search_docs(query: str) -> str:
    """Search documentation for technical information."""
    results = vectorstore.similarity_search(query, k=3)
    return "\n\n".join([doc.page_content for doc in results])

def calculate(expression: str) -> str:
    """Calculate mathematical expressions. Use Python syntax."""
    try:
        return str(eval(expression))
    except Exception as e:
        return f"Error: {e}"

tools = [
    Tool(
        name="SearchDocs",
        func=search_docs,
        description="Search documentation for technical information about vLLM, PagedAttention, and model serving"
    ),
    Tool(
        name="Calculator",
        func=calculate,
        description="Calculate mathematical expressions. Input should be valid Python expression."
    )
]

# Create agent
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = hub.pull("hwchase17/react")

agent = create_react_agent(llm, tools, prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    max_iterations=5,
    handle_parsing_errors=True
)

# Run agent
result = agent_executor.invoke({
    "input": "What is PagedAttention and calculate GPU memory for Llama-3.1-8B (8B params × 2 bytes)"
})

print(result["output"])

Agent reasoning flow:

Thought: I need to search docs for PagedAttention and calculate memory
Action: SearchDocs
Action Input: PagedAttention
Observation: [retrieved doc content]

Thought: Now I need to calculate GPU memory
Action: Calculator
Action Input: 8000000000 * 2 / (1024**3)
Observation: 14.901161193847656

Thought: I have enough information to answer
Final Answer: PagedAttention is... and requires approximately 14.9 GB of GPU memory.

Structured Tool Calling

For newer models with native tool calling:

from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.tools import tool

@tool
def search_database(query: str, limit: int = 5) -> str:
    """Search the vector database for relevant documents.

    Args:
        query: Search query
        limit: Maximum number of results (default: 5)
    """
    results = vectorstore.similarity_search(query, k=limit)
    return "\n\n".join([doc.page_content for doc in results])

llm = ChatOpenAI(model="gpt-4o")
tools = [search_database]

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)

Streaming Responses

Stream tokens as they're generated:

from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

llm = ChatOpenAI(
    model="gpt-4o",
    streaming=True,
    callbacks=[StreamingStdOutCallbackHandler()]
)

# For web applications
from langchain.callbacks.base import BaseCallbackHandler

class StreamingHandler(BaseCallbackHandler):
    def __init__(self):
        self.tokens = []

    def on_llm_new_token(self, token: str, **kwargs):
        self.tokens.append(token)
        # Send token to frontend via SSE
        yield f"data: {json.dumps({'token': token})}\n\n"

handler = StreamingHandler()
llm = ChatOpenAI(model="gpt-4o", streaming=True, callbacks=[handler])

Custom Chains

Build application-specific chains:

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Custom prompt
template = """You are a technical documentation expert.

Context from documentation:
{context}

Question: {question}

Provide a detailed, accurate answer based on the context. If the context doesn't contain enough information, say so clearly.

Answer:"""

prompt = PromptTemplate(
    input_variables=["context", "question"],
    template=template
)

# Custom chain
def custom_rag(question: str) -> str:
    # 1. Retrieve
    docs = vectorstore.similarity_search(question, k=3)
    context = "\n\n".join([doc.page_content for doc in docs])

    # 2. Generate
    llm_chain = LLMChain(llm=llm, prompt=prompt)
    answer = llm_chain.run(context=context, question=question)

    return answer

Error Handling

from langchain.callbacks import get_openai_callback

try:
    with get_openai_callback() as cb:
        result = qa_chain.invoke({"query": question})

        print(f"Tokens used: {cb.total_tokens}")
        print(f"Cost: ${cb.total_cost:.4f}")
        print(f"Answer: {result['result']}")

except Exception as e:
    print(f"Error: {e}")
    # Fallback logic
    result = {"result": "Sorry, I encountered an error. Please try again."}

Integration with vLLM

Use vLLM as LLM backend:

from langchain_community.llms import VLLM

llm = VLLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    vllm_kwargs={
        "max_new_tokens": 512,
        "temperature": 0.7,
        "top_p": 0.9
    },
    trust_remote_code=True
)

# Or use OpenAI-compatible endpoint
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
    model="meta-llama/Llama-3.1-8B-Instruct"
)

Best Practices

1. Chunk Size Matters:

• Too small: Loses context
• Too large: Irrelevant info
• Sweet spot: 512-1024 tokens with 10-20% overlap

2. Use Appropriate Chain Type:

• Few docs (<5): stuff
• Many docs (>10): map_reduce
• Quality synthesis needed: refine

3. Add Metadata for Filtering:

   chunks = text_splitter.split_documents(documents)
   for chunk in chunks:
       chunk.metadata["source"] = "manual.pdf"
       chunk.metadata["date"] = "2025-12-02"

   # Filter during retrieval
   retriever = vectorstore.as_retriever(
       search_kwargs={
           "k": 5,
           "filter": {"source": "manual.pdf"}
       }
   )

4. Monitor Token Usage:

   from langchain.callbacks import get_openai_callback

   with get_openai_callback() as cb:
       result = chain.invoke({"query": question})
       print(f"Cost: ${cb.total_cost:.4f}")

5. Cache Embeddings:

   from langchain.embeddings import CacheBackedEmbeddings
   from langchain.storage import LocalFileStore

   store = LocalFileStore("./cache/")
   cached_embedder = CacheBackedEmbeddings.from_bytes_store(
       OpenAIEmbeddings(),
       store,
       namespace="openai-embeddings"
   )

Resources

• LangChain Docs: https://python.langchain.com/
• LangChain Hub (prompts): https://smith.langchain.com/hub
• LangSmith (tracing): https://smith.langchain.com/
• GitHub: https://github.com/langchain-ai/langchain

Server-Sent Events (SSE) for LLM Streaming

Complete guide for streaming LLM responses using Server-Sent Events (SSE) across multiple backends.

Table of Contents

• Why SSE for LLM Streaming
• Protocol Overview
• Python + FastAPI
• Basic Implementation
• OpenAI-Compatible vLLM Streaming
• TypeScript + Hono (Edge)
• Cloudflare Workers
• Rust + Axum
• Frontend Integration
• React + EventSource (Native)
• Fetch API (POST with SSE)
• Advanced Patterns
• Reconnection with Event ID
• Error Handling
• Production Considerations
• Nginx Configuration
• Load Balancing
• Timeouts
• Comparison with Alternatives
• Common Pitfalls
• Resources

Why SSE for LLM Streaming

Advantages over WebSockets:

• Simpler protocol (HTTP, not bidirectional)
• Automatic reconnection (browser handles it)
• Built-in event IDs for resumption
• Standard HTTP infrastructure (proxies, load balancers work)

Use case: Streaming LLM tokens progressively to frontend (ai-chat skill integration).

Protocol Overview

SSE format:

data: First chunk of text\n\n
data: Second chunk\n\n
data: {"token": "word", "done": false}\n\n
data: [DONE]\n\n

Key points:

• Each message starts with data:
• Messages end with double newline (\n\n)
• Can send JSON or plain text
• Client auto-reconnects on disconnect

---

Python + FastAPI

Basic Implementation

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from sse_starlette.sse import EventSourceResponse
import asyncio

app = FastAPI()

async def generate_tokens():
    """Simulate LLM streaming"""
    tokens = ["Hello", " there!", " How", " can", " I", " help", " you", " today", "?"]

    for token in tokens:
        yield {
            "event": "message",
            "data": token
        }
        await asyncio.sleep(0.1)  # Simulate model latency

    yield {
        "event": "done",
        "data": "[DONE]"
    }

@app.post("/chat/stream")
async def stream_chat():
    return EventSourceResponse(generate_tokens())

OpenAI-Compatible vLLM Streaming

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import OpenAI
import json

app = FastAPI()
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

@app.post("/chat/stream")
async def stream_chat(prompt: str):
    async def generate():
        stream = client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            messages=[{"role": "user", "content": prompt}],
            stream=True,
            max_tokens=512,
        )

        for chunk in stream:
            if chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                # SSE format
                yield f"data: {json.dumps({'token': token})}\n\n"

        yield f"data: {json.dumps({'done': True})}\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",  # Disable nginx buffering
        },
    )

---

TypeScript + Hono (Edge)

Cloudflare Workers

import { Hono } from 'hono';
import { streamSSE } from 'hono/streaming';

const app = new Hono();

app.post('/chat/stream', async (c) => {
  const { prompt } = await c.req.json();

  return streamSSE(c, async (stream) => {
    const tokens = prompt.split(' ');

    for (const token of tokens) {
      await stream.writeSSE({
        data: JSON.stringify({ token: token + ' ' }),
        event: 'message',
      });
      await stream.sleep(100);
    }

    await stream.writeSSE({
      data: JSON.stringify({ done: true }),
      event: 'done',
    });
  });
});

export default app;

---

Rust + Axum

use axum::{
    response::sse::{Event, Sse},
    routing::post,
    Json, Router,
};
use futures::stream::{self, Stream};
use serde::{Deserialize, Serialize};
use std::convert::Infallible;
use std::time::Duration;
use tokio::time::sleep;

#[derive(Deserialize)]
struct ChatRequest {
    prompt: String,
}

#[derive(Serialize)]
struct TokenResponse {
    token: String,
    done: bool,
}

async fn stream_chat(
    Json(payload): Json<ChatRequest>,
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
    let tokens: Vec<String> = payload.prompt.split_whitespace()
        .map(|s| s.to_string())
        .collect();

    let stream = stream::iter(tokens)
        .then(|token| async move {
            sleep(Duration::from_millis(100)).await;

            let response = TokenResponse {
                token: format!("{} ", token),
                done: false,
            };

            Ok::<_, Infallible>(
                Event::default()
                    .event("message")
                    .json_data(response)
                    .unwrap()
            )
        });

    Sse::new(stream)
        .keep_alive(axum::response::sse::KeepAlive::default())
}

#[tokio::main]
async fn main() {
    let app = Router::new().route("/chat/stream", post(stream_chat));

    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service())
        .await
        .unwrap();
}

---

Frontend Integration

React + EventSource (Native)

import { useState, useEffect } from 'react';

export function useStreamingChat() {
  const [response, setResponse] = useState('');
  const [isStreaming, setIsStreaming] = useState(false);

  const sendMessage = async (prompt: string) => {
    setResponse('');
    setIsStreaming(true);

    const es = new EventSource(`/chat/stream?prompt=${encodeURIComponent(prompt)}`);

    es.addEventListener('message', (e) => {
      const data = JSON.parse(e.data);
      if (data.token) {
        setResponse((prev) => prev + data.token);
      }
    });

    es.addEventListener('done', () => {
      setIsStreaming(false);
      es.close();
    });

    es.onerror = () => {
      setIsStreaming(false);
      es.close();
    };
  };

  return { response, isStreaming, sendMessage };
}

// Usage
function ChatComponent() {
  const { response, isStreaming, sendMessage } = useStreamingChat();

  return (
    <div>
      <button onClick={() => sendMessage('Hello AI!')}>
        Send
      </button>
      <div>{response}</div>
      {isStreaming && <span>...</span>}
    </div>
  );
}

Fetch API (POST with SSE)

async function streamChat(prompt: string) {
  const response = await fetch('/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split('\n\n');

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const data = JSON.parse(line.slice(6));

        if (data.token) {
          appendToken(data.token);
        }

        if (data.done) {
          return;
        }
      }
    }
  }
}

---

Advanced Patterns

Reconnection with Event ID

from fastapi import Request
from sse_starlette.sse import EventSourceResponse

@app.post("/chat/stream")
async def stream_chat(request: Request):
    last_event_id = request.headers.get("Last-Event-ID", "0")
    start_index = int(last_event_id)

    async def generate():
        tokens = get_all_tokens()  # Get full token list

        for i, token in enumerate(tokens[start_index:], start=start_index):
            yield {
                "id": str(i + 1),  # Event ID for resumption
                "event": "message",
                "data": token
            }

    return EventSourceResponse(generate())

Frontend auto-reconnection:

const es = new EventSource('/chat/stream');

// Browser automatically sends Last-Event-ID header on reconnect
es.addEventListener('message', (e) => {
  console.log('Last ID:', e.lastEventId);  // Automatically tracked
});

Error Handling

@app.post("/chat/stream")
async def stream_chat():
    async def generate():
        try:
            # LLM streaming logic
            for token in llm_stream:
                yield {"data": token}
        except Exception as e:
            yield {
                "event": "error",
                "data": json.dumps({"error": str(e)})
            }

    return EventSourceResponse(generate())

---

Production Considerations

Nginx Configuration

# Disable buffering for SSE
location /chat/stream {
    proxy_pass http://backend;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding off;
}

Load Balancing

Sticky sessions required:

# Kubernetes ingress annotation
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"

Timeouts

# Set appropriate timeouts
EventSourceResponse(
    generate(),
    ping_interval=15,  # Send keep-alive every 15s
)

---

Comparison with Alternatives

---

Common Pitfalls

Buffer issues:

# ❌ Don't do this (buffered)
return Response(generate(), media_type="text/event-stream")

# ✅ Do this (streaming)
return EventSourceResponse(generate())

Missing headers:

# Required headers for SSE
headers = {
    "Cache-Control": "no-cache",
    "X-Accel-Buffering": "no",  # Disable nginx buffering
    "Connection": "keep-alive",
}

---

Resources

• MDN SSE Docs: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
• sse-starlette: https://github.com/sysid/sse-starlette
• Hono Streaming: https://hono.dev/helpers/streaming

Text Generation Inference (TGI) - HuggingFace LLM Serving

Table of Contents

• Overview
• Installation
• Quick Start
• Docker Deployment
• Key Parameters
• API Usage
• HTTP Endpoints
• Python Client
• Quantization
• GPTQ (4-bit)
• AWQ (4-bit)
• bitsandbytes (8-bit/4-bit)
• Multi-GPU Deployment
• Kubernetes Deployment
• Monitoring
• Health Check
• Metrics
• Comparison: TGI vs vLLM
• Resources

Overview

Text Generation Inference (TGI) is HuggingFace's production-ready LLM serving solution with continuous batching, tensor parallelism, and optimized kernels.

When to use TGI instead of vLLM:

• Deep HuggingFace ecosystem integration required
• Want official HuggingFace support
• Prefer opinionated, simpler deployment
• Need HuggingFace Hub integration

vLLM is generally preferred for:

• Maximum throughput (PagedAttention)
• More flexible configuration
• Better community support

Installation

# Docker (recommended)
docker pull ghcr.io/huggingface/text-generation-inference:latest

# From source
cargo install --path router
cargo install --path launcher

Quick Start

Docker Deployment

# Basic deployment
docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct

# Production configuration
docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN=your_token_here \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-total-tokens 4096 \
  --max-input-length 2048 \
  --max-batch-prefill-tokens 8192 \
  --max-batch-total-tokens 16384

Key Parameters

Model Loading:

• --model-id: HuggingFace model ID or local path
• --revision: Model revision (branch, tag, or commit)
• --dtype: Data type (float16, bfloat16, auto)

Performance:

• --max-total-tokens: Maximum sequence length
• --max-input-length: Maximum input tokens
• --max-batch-prefill-tokens: Max tokens for prefill stage
• --max-batch-total-tokens: Total tokens in batch

Multi-GPU:

• --num-shard: Number of GPUs for tensor parallelism
• --quantize: Quantization method (bitsandbytes, gptq, awq)

API Usage

HTTP Endpoints

Generate (synchronous):

curl http://localhost:8080/generate \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": "What is machine learning?",
    "parameters": {
      "max_new_tokens": 256,
      "temperature": 0.7,
      "top_p": 0.9
    }
  }'

Generate Stream (SSE):

curl http://localhost:8080/generate_stream \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": "Write a poem about AI",
    "parameters": {
      "max_new_tokens": 256
    }
  }'

Chat Completions (OpenAI-compatible):

curl http://localhost:8080/v1/chat/completions \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain quantum computing"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Python Client

from huggingface_hub import InferenceClient

client = InferenceClient(
    base_url="http://localhost:8080",
    token=None  # No token for local TGI
)

# Generate
response = client.text_generation(
    "Explain PagedAttention in simple terms",
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9
)
print(response)

# Stream
for token in client.text_generation(
    "Count to 10",
    max_new_tokens=50,
    stream=True
):
    print(token, end="")

Quantization

GPTQ (4-bit)

docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id TheBloke/Llama-3.1-8B-GPTQ \
  --quantize gptq

AWQ (4-bit)

docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id TheBloke/Llama-3.1-8B-AWQ \
  --quantize awq

bitsandbytes (8-bit/4-bit)

# 8-bit
docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --quantize bitsandbytes

# 4-bit
docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --quantize bitsandbytes-nf4

Multi-GPU Deployment

# Llama-3.1-70B on 4 GPUs
docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-70B-Instruct \
  --num-shard 4

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-llama
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tgi-llama
  template:
    metadata:
      labels:
        app: tgi-llama
    spec:
      containers:
      - name: tgi
        image: ghcr.io/huggingface/text-generation-inference:latest
        args:
          - --model-id
          - meta-llama/Llama-3.1-8B-Instruct
          - --max-total-tokens
          - "4096"
        ports:
        - containerPort: 80
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 32Gi
          requests:
            nvidia.com/gpu: 1
            memory: 16Gi
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

Monitoring

Health Check

curl http://localhost:8080/health

Metrics

TGI exposes Prometheus metrics at /metrics:

Key metrics:

• tgi_request_duration_seconds - Request latency
• tgi_queue_size - Current queue depth
• tgi_request_success_total - Successful requests
• tgi_request_failure_total - Failed requests

Comparison: TGI vs vLLM

Choose TGI if:

• Want official HuggingFace support
• Prefer simpler, opinionated deployment
• Deep HuggingFace Hub integration needed

Choose vLLM if:

• Need maximum throughput
• Want flexible configuration
• PagedAttention benefits important

Resources

• TGI Documentation: https://huggingface.co/docs/text-generation-inference
• GitHub: https://github.com/huggingface/text-generation-inference
• Supported Models: https://huggingface.co/docs/text-generation-inference/supported_models

vLLM - High-Performance LLM Serving

Table of Contents

• Overview
• Installation
• PagedAttention Architecture
• The Memory Problem
• PagedAttention Solution
• Basic Usage
• Starting the Server
• Key Parameters Explained
• Advanced Configuration
• Quantization
• Multi-GPU Deployment
• Python API
• OpenAI-Compatible API
• Performance Tuning
• Maximizing Throughput
• Minimizing Latency
• Monitoring
• Built-in Metrics
• Health Checks
• Troubleshooting
• Out of Memory (OOM)
• Low Throughput
• High Latency
• Model Compatibility
• Production Best Practices
• Resources

Overview

vLLM (Versatile LLM) is a high-throughput and memory-efficient inference engine for LLMs featuring PagedAttention memory management and continuous batching.

Key Advantages:

• 20-30x higher throughput vs naive PyTorch implementation
• Eliminates memory fragmentation with PagedAttention
• OpenAI-compatible API for easy migration
• Supports 100+ HuggingFace models out-of-the-box

Installation

# Standard installation
pip install vllm

# For specific GPU architectures
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu121  # CUDA 12.1

# From source (for latest features)
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

System Requirements:

• NVIDIA GPU with compute capability 7.0+ (V100, A100, H100, RTX 3090+)
• CUDA 11.8 or 12.1+
• Python 3.8+
• 16GB+ GPU memory for 7B models

PagedAttention Architecture

The Memory Problem

Traditional LLM serving wastes GPU memory:

Traditional KV Cache (static allocation):
┌─────────────────────────────────────┐
│ Request 1 [████░░░░] 50% utilized  │ ← Fragmentation
│ Request 2 [██████░░] 75% utilized  │ ← Fragmentation
│ Request 3 [████████] 100% utilized │ ← No waste
└─────────────────────────────────────┘
Total Memory Utilization: ~50-60%

PagedAttention Solution

Inspired by OS virtual memory paging:

PagedAttention (dynamic allocation):
┌─────────────────────────────────────┐
│ Shared Pool: [████████████████████] │
│                                      │
│ Request 1 → Pages [0,1,2,3]        │
│ Request 2 → Pages [4,5,6,7,8,9]    │
│ Request 3 → Pages [10,11,12,13]    │
└─────────────────────────────────────┘
Total Memory Utilization: ~90%+

Benefits:

• Near-zero memory fragmentation
• 2-4x memory efficiency vs static allocation
• Enables larger batch sizes
• 20-30x throughput improvement

Basic Usage

Starting the Server

# Basic server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype auto \
  --port 8000

# Production configuration
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype float16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 256 \
  --host 0.0.0.0 \
  --port 8000

Key Parameters Explained

Model Loading:

• --dtype: Precision (auto, float16, bfloat16, float32)
• auto: Let vLLM choose (usually float16)
• float16: Standard half precision
• bfloat16: Better numerical stability, slightly slower

Memory Management:

• --gpu-memory-utilization: Fraction of GPU memory to use (0.8-0.95)
• Too low: Underutilized GPU
• Too high: Risk of OOM errors
• Recommended: Start with 0.9

• --max-model-len: Maximum sequence length (context window)
• Default: Model's max (e.g., 8192 for Llama-3.1-8B)
• Reduce to fit larger batches

Throughput:

• --max-num-seqs: Maximum concurrent sequences
• Higher = more throughput
• Limited by GPU memory
• Default: 256

Parallelism:

• --tensor-parallel-size: Number of GPUs for model parallelism
• Use for models that don't fit on single GPU
• Must divide model evenly

Advanced Configuration

Quantization

Reduce memory usage with quantization:

# AWQ (4-bit quantization)
vllm serve TheBloke/Llama-3.1-8B-AWQ \
  --quantization awq \
  --dtype auto

# GPTQ (another 4-bit method)
vllm serve TheBloke/Llama-3.1-8B-GPTQ \
  --quantization gptq

# FP8 quantization (H100 optimized)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --dtype auto

Quantization Comparison:

• FP16 (baseline): 2 bytes/param, best accuracy
• AWQ (4-bit): 0.5 bytes/param, 4x memory reduction, ~2% accuracy loss
• GPTQ (4-bit): 0.5 bytes/param, similar to AWQ
• FP8: 1 byte/param, 2x memory reduction, minimal accuracy loss (H100 only)

Multi-GPU Deployment

Tensor Parallelism (single model across GPUs):

# Llama-3.1-70B on 4x A100 (40GB each)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9

Pipeline Parallelism:

# Split model layers across GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --pipeline-parallel-size 2 \
  --tensor-parallel-size 2  # Total 4 GPUs

Python API

For programmatic use without HTTP server:

from vllm import LLM, SamplingParams

# Initialize model
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    dtype="auto"
)

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,    # Randomness (0 = deterministic, 1 = creative)
    top_p=0.9,         # Nucleus sampling
    max_tokens=256,    # Maximum response length
    stop=["</s>", "\n\n"]  # Stop sequences
)

# Generate (batched automatically)
prompts = [
    "Explain quantum computing",
    "Write a Python function for Fibonacci"
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Response: {output.outputs[0].text}")
    print(f"Tokens: {len(output.outputs[0].token_ids)}")

OpenAI-Compatible API

vLLM exposes OpenAI-compatible endpoints:

Endpoints:

• /v1/chat/completions - Chat format
• /v1/completions - Raw completion
• /v1/embeddings - Text embeddings (if model supports)
• /v1/models - List available models

Example:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # vLLM doesn't require auth by default
)

# Chat completion
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain PagedAttention"}
    ],
    temperature=0.7,
    max_tokens=512
)

print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Count to 10"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Performance Tuning

Maximizing Throughput

1. Tune GPU Memory Utilization:

# Conservative (safe)
--gpu-memory-utilization 0.85

# Aggressive (maximum throughput)
--gpu-memory-utilization 0.95

2. Adjust Sequence Length:

# Shorter context = more concurrent requests
--max-model-len 2048  # Instead of default 8192

3. Enable Continuous Batching: Already enabled by default in vLLM. No configuration needed.

Minimizing Latency

1. Reduce Batch Size:

--max-num-seqs 16  # Lower than default 256

2. Use Faster Precision:

--dtype float16  # Faster than bfloat16

3. Smaller Model: Use Llama-3.1-8B instead of Llama-3.1-70B for latency-sensitive applications.

Monitoring

Built-in Metrics

vLLM exposes Prometheus metrics at /metrics:

Key Metrics:

• vllm:num_requests_running - Current active requests
• vllm:num_requests_waiting - Queue depth
• vllm:gpu_cache_usage_perc - KV cache utilization
• vllm:time_to_first_token_seconds - TTFT latency
• vllm:time_per_output_token_seconds - Inter-token latency

Prometheus scrape config:

scrape_configs:
  - job_name: 'vllm'
    static_configs:
      - targets: ['localhost:8000']

Health Checks

# Health endpoint
curl http://localhost:8000/health

# Model info
curl http://localhost:8000/v1/models

Troubleshooting

Out of Memory (OOM)

Symptoms: CUDA out of memory errors

Solutions: 1. Reduce --gpu-memory-utilization to 0.85 2. Decrease --max-model-len 3. Lower --max-num-seqs 4. Enable quantization (AWQ, GPTQ) 5. Use smaller model variant

Example fix:

# Before (OOM)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95

# After (fixed)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --quantization awq  # If using quantized model

Low Throughput

Symptoms: Low requests/sec, high queue depth

Solutions: 1. Increase --gpu-memory-utilization to 0.9-0.95 2. Increase --max-num-seqs 3. Check GPU utilization (should be >80%) 4. Use tensor parallelism if memory allows

High Latency

Symptoms: Slow time to first token

Solutions: 1. Reduce batch size (--max-num-seqs) 2. Use smaller model 3. Check network latency 4. Profile with nvidia-smi during inference

Model Compatibility

vLLM supports 100+ models from HuggingFace. Common families:

LLMs:

• Llama 2, Llama 3, Llama 3.1
• Mistral, Mixtral
• Qwen 2, Qwen 2.5
• Gemma, Gemma 2
• Phi-2, Phi-3

Embedding Models:

• BERT variants
• sentence-transformers models

Check compatibility:

# List supported architectures
python -c "from vllm import ModelRegistry; print(ModelRegistry.get_supported_archs())"

Production Best Practices

1. Use float16 dtype for best performance/quality balance 2. Set gpu-memory-utilization to 0.9 for production 3. Enable monitoring with Prometheus metrics 4. Add health checks in load balancer 5. Use quantization (AWQ) for GPU memory constrained deployments 6. Deploy behind API gateway (Kong, Nginx) for auth and rate limiting 7. Monitor queue depth - scale if consistently >10

Resources

• vLLM Documentation: https://docs.vllm.ai/
• PagedAttention Paper: https://arxiv.org/abs/2309.06180
• GitHub: https://github.com/vllm-project/vllm
• Model Hub: https://huggingface.co/models?library=vllm