Resilience Patterns

Resilience Patterns Skill

Production-grade resilience patterns for distributed systems and LLM-based workflows. Covers circuit breakers, bulkheads, retry strategies, and LLM-specific resilience techniques.

Overview

• Building fault-tolerant multi-agent systems
• Implementing LLM API integrations with proper error handling
• Designing distributed workflows that need graceful degradation
• Adding observability to failure scenarios
• Protecting systems from cascade failures

Core Patterns

1. Circuit Breaker Pattern (reference: circuit-breaker.md)

Prevents cascade failures by "tripping" when a service exceeds failure thresholds.

+-------------------------------------------------------------------+
|                    Circuit Breaker States                         |
+-------------------------------------------------------------------+
|                                                                   |
|    +----------+     failures >= threshold    +----------+         |
|    |  CLOSED  | ----------------------------> |   OPEN   |        |
|    | (normal) |                              | (reject) |         |
|    +----+-----+                              +----+-----+         |
|         |                                         |               |
|         | success                    timeout      |               |
|         |                            expires      |               |
|         |         +------------+                  |               |
|         |         | HALF_OPEN  |<-----------------+               |
|         +---------+  (probe)   |                                  |
|                   +------------+                                  |
|                                                                   |
|   CLOSED:    Allow requests, count failures                       |
|   OPEN:      Reject immediately, return fallback                  |
|   HALF_OPEN: Allow probe request to test recovery                 |
|                                                                   |
+-------------------------------------------------------------------+

Key Configuration:

• failure_threshold: Failures before opening (default: 5)
• recovery_timeout: Seconds before attempting recovery (default: 30)
• half_open_requests: Probes to allow in half-open (default: 1)

2. Bulkhead Pattern (reference: bulkhead-pattern.md)

Isolates failures by partitioning resources into independent pools.

+-------------------------------------------------------------------+
|                      Bulkhead Isolation                           |
+-------------------------------------------------------------------+
|                                                                   |
|   +------------------+  +------------------+                      |
|   | TIER 1: Critical |  | TIER 2: Standard |                      |
|   |  (5 workers)     |  |  (3 workers)     |                      |
|   |  +-+ +-+ +-+     |  |  +-+ +-+ +-+     |                      |
|   |  |#| |#| | |     |  |  |#| | | | |     |                      |
|   |  +-+ +-+ +-+     |  |  +-+ +-+ +-+     |                      |
|   |  +-+ +-+         |  |                  |                      |
|   |  | | | |         |  |  Queue: 2        |                      |
|   |  +-+ +-+         |  |                  |                      |
|   |  Queue: 0        |  +------------------+                      |
|   +------------------+                                            |
|                                                                   |
|   +------------------+                                            |
|   | TIER 3: Optional |   # = Active request                       |
|   |  (2 workers)     |     = Available slot                       |
|   |  +-+ +-+         |                                            |
|   |  |#| |#| FULL!   |   Tier 1: synthesis, quality_gate          |
|   |  +-+ +-+         |   Tier 2: analysis agents                  |
|   |  Queue: 5        |   Tier 3: enrichment, optional features    |
|   +------------------+                                            |
|                                                                   |
+-------------------------------------------------------------------+

Tier Configuration (OrchestKit):

3. Retry Strategies (reference: retry-strategies.md)

Intelligent retry logic with exponential backoff and jitter.

+-------------------------------------------------------------------+
|                   Exponential Backoff + Jitter                    |
+-------------------------------------------------------------------+
|                                                                   |
|   Attempt 1:  --> X (fail)                                        |
|               wait: 1s +/- 0.5s                                   |
|                                                                   |
|   Attempt 2:  --> X (fail)                                        |
|               wait: 2s +/- 1s                                     |
|                                                                   |
|   Attempt 3:  --> X (fail)                                        |
|               wait: 4s +/- 2s                                     |
|                                                                   |
|   Attempt 4:  --> OK (success)                                    |
|                                                                   |
|   Formula: delay = min(base * 2^attempt, max_delay) * jitter      |
|   Jitter:  random(0.5, 1.5) to prevent thundering herd            |
|                                                                   |
+-------------------------------------------------------------------+

Error Classification for Retries:

RETRYABLE_ERRORS = {
    # HTTP/Network
    408, 429, 500, 502, 503, 504,  # HTTP status codes
    ConnectionError, TimeoutError,  # Network errors

    # LLM-specific
    "rate_limit_exceeded",
    "model_overloaded",
    "context_length_exceeded",  # Retry with truncation
}

NON_RETRYABLE_ERRORS = {
    400, 401, 403, 404,  # Client errors
    "invalid_api_key",
    "content_policy_violation",
    "invalid_request_error",
}

4. LLM-Specific Resilience (reference: llm-resilience.md)

Patterns specific to LLM API integrations.

+-------------------------------------------------------------------+
|                    LLM Fallback Chain                             |
+-------------------------------------------------------------------+
|                                                                   |
|   Request --> [Primary Model] --success--> Response               |
|                     |                                             |
|                   fail                                            |
|                     v                                             |
|               [Fallback Model] --success--> Response              |
|                     |                                             |
|                   fail                                            |
|                     v                                             |
|               [Cached Response] --hit--> Response                 |
|                     |                                             |
|                   miss                                            |
|                     v                                             |
|               [Default Response] --> Graceful Degradation         |
|                                                                   |
|   Example Chain:                                                  |
|   1. claude-sonnet-4-5-20251101 (primary)                         |
|   2. gpt-5.2-mini (fallback)                                      |
|   3. Semantic cache lookup                                        |
|   4. "Analysis unavailable" + partial results                     |
|                                                                   |
+-------------------------------------------------------------------+

Token Budget Management:

+-------------------------------------------------------------------+
|                     Token Budget Guard                            |
+-------------------------------------------------------------------+
|                                                                   |
|   Input: 8,000 tokens                                             |
|   +---------------------------------------------+                 |
|   |#################################            |                 |
|   +---------------------------------------------+                 |
|                                          ^                        |
|                                          |                        |
|                                    Context Limit (16K)            |
|                                                                   |
|   Strategy when approaching limit:                                |
|   1. Summarize earlier context (compress 4:1)                     |
|   2. Drop low-priority content (optional fields)                  |
|   3. Split into multiple requests                                 |
|   4. Fail fast with "content too large" error                     |
|                                                                   |
+-------------------------------------------------------------------+

Quick Reference

OrchestKit Integration Points

1. Workflow Agents: Each agent wrapped with circuit breaker + bulkhead tier 2. LLM Calls: All model invocations use fallback chain + retry logic 3. External APIs: Circuit breaker on YouTube, arXiv, GitHub APIs 4. Database Ops: Bulkhead isolation for read vs write operations

Files in This Skill

References (Conceptual Guides)

• references/circuit-breaker.md - Deep dive on circuit breaker pattern
• references/bulkhead-pattern.md - Bulkhead isolation strategies
• references/retry-strategies.md - Retry algorithms and error classification
• references/llm-resilience.md - LLM-specific patterns
• references/error-classification.md - How to categorize errors

Templates (Code Patterns)

• scripts/circuit-breaker.py - Ready-to-use circuit breaker class
• scripts/bulkhead.py - Semaphore-based bulkhead implementation
• scripts/retry-handler.py - Configurable retry decorator
• scripts/llm-fallback-chain.py - Multi-model fallback pattern
• scripts/token-budget.py - Token budget guard implementation

Examples

• examples/orchestkit-workflow-resilience.md - Full OrchestKit integration example

Checklists

• checklists/pre-deployment-resilience.md - Production readiness checklist
• checklists/circuit-breaker-setup.md - Circuit breaker configuration guide

2026 Best Practices

1. Adaptive Thresholds: Use sliding windows, not fixed counters 2. Observability First: Every circuit trip = alert + metric + trace 3. Graceful Degradation: Always have a fallback, even if partial 4. Health Endpoints: Separate health check from circuit state 5. Chaos Testing: Regularly test failure scenarios in staging

---

Related Skills

• observability-monitoring - Metrics and alerting for circuit breaker state changes
• caching-strategies - Cache as fallback layer in degradation scenarios
• error-handling-rfc9457 - Structured error responses for resilience failures
• background-jobs - Async processing with retry and failure handling

Key Decisions

---

Capability Details

circuit-breaker

Keywords: circuit breaker, failure threshold, cascade failure, trip, half-open Solves:

• Prevent cascade failures when external services fail
• Automatically recover when services come back online
• Fail fast instead of waiting for timeouts

bulkhead

Keywords: bulkhead, isolation, semaphore, thread pool, resource pool, tier Solves:

• Isolate failures to prevent entire system crashes
• Prioritize critical operations over optional ones
• Limit concurrent requests to protect resources

retry-strategies

Keywords: retry, backoff, exponential, jitter, thundering herd Solves:

• Handle transient failures automatically
• Avoid overwhelming recovering services
• Classify errors as retryable vs non-retryable

llm-resilience

Keywords: LLM, fallback, model, token budget, rate limit, context length Solves:

• Handle LLM API rate limits gracefully
• Fall back to alternative models when primary fails
• Manage token budgets to prevent context overflow

error-classification

Keywords: error, retryable, transient, permanent, classification Solves:

• Determine which errors should be retried
• Categorize errors by severity and recoverability
• Map HTTP status codes to resilience actions

Circuit Breaker Setup Guide

Step-by-step guide for adding circuit breakers to a service.

Step 1: Identify Services

List all external dependencies that need circuit breakers:

┌─────────────────────────────────────────────────────────────┐
│ Service Inventory                                            │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│ External APIs:                                               │
│ □ OpenAI API (LLM)                                          │
│ □ Anthropic API (LLM)                                       │
│ □ YouTube Data API                                          │
│ □ GitHub API                                                │
│ □ arXiv API                                                 │
│                                                              │
│ Internal Services:                                           │
│ □ Embedding service                                         │
│ □ Database (PostgreSQL)                                     │
│ □ Redis cache                                               │
│ □ Semantic search                                           │
│                                                              │
│ For each, answer:                                            │
│ 1. What's the expected failure rate?                        │
│ 2. How long does recovery typically take?                   │
│ 3. What's the fallback behavior?                            │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Step 2: Configure Thresholds

Failure Threshold

Recovery Timeout

Slow Call Threshold

Step 3: Implement Circuit Breaker

Basic Implementation

from resilience import CircuitBreaker, CircuitBreakerFactory

# Option 1: Use factory for common patterns
openai_breaker = CircuitBreakerFactory.for_llm_api("openai")
db_breaker = CircuitBreakerFactory.for_database("postgres")

# Option 2: Custom configuration
custom_breaker = CircuitBreaker(
    name="my-service",
    failure_threshold=5,
    success_threshold=2,
    recovery_timeout=30.0,
    slow_call_threshold=10.0,
)

Wrap Service Calls

# Method 1: Decorator
@openai_breaker
async def call_openai(prompt: str) -> str:
    return await openai_client.complete(prompt)

# Method 2: Explicit call
async def call_openai(prompt: str) -> str:
    return await openai_breaker.call(
        openai_client.complete,
        prompt,
    )

Step 4: Add Fallback Handling

from resilience import CircuitOpenError

async def analyze_with_fallback(content: str) -> Analysis:
    try:
        return await circuit_breaker.call(primary_analysis, content)

    except CircuitOpenError as e:
        logger.warning(
            f"Circuit open for {e.name}, using fallback",
            time_until_recovery=e.time_until_recovery,
        )

        # Fallback 1: Try cache
        cached = await cache.get(f"analysis:{hash(content)}")
        if cached:
            return Analysis.from_cache(cached, is_stale=True)

        # Fallback 2: Degraded response
        return Analysis(
            status="degraded",
            message="Full analysis temporarily unavailable",
            basic_info=extract_basic_info(content),
        )

Step 5: Add Observability

Logging

def setup_circuit_logging(breaker: CircuitBreaker):
    def on_state_change(old: str, new: str, name: str):
        logger.warning(
            "circuit_state_change",
            circuit=name,
            old_state=old,
            new_state=new,
        )

        if new == "open":
            # Send alert
            alerting.send(
                severity="warning",
                message=f"Circuit {name} opened",
                runbook="https://docs/runbooks/circuit-breaker",
            )

    breaker._on_state_change = on_state_change

Metrics

from prometheus_client import Gauge, Counter

circuit_state = Gauge(
    "circuit_breaker_state",
    "Circuit breaker state (0=closed, 1=open, 2=half_open)",
    ["service"],
)

circuit_rejections = Counter(
    "circuit_breaker_rejections_total",
    "Total requests rejected by circuit breaker",
    ["service"],
)

def update_metrics(breaker: CircuitBreaker):
    state_map = {"closed": 0, "open": 1, "half_open": 2}
    circuit_state.labels(service=breaker.name).set(
        state_map[breaker.state.value]
    )

Health Endpoint

@app.get("/health/circuits")
async def circuit_health():
    return {
        name: {
            "state": cb.state.value,
            "failure_count": cb._failure_count,
            "time_until_recovery": (
                cb._time_until_recovery()
                if cb.state == CircuitState.OPEN
                else None
            ),
        }
        for name, cb in circuit_breakers.items()
    }

Step 6: Test Circuit Behavior

Unit Tests

@pytest.mark.asyncio
async def test_circuit_opens_on_failures():
    breaker = CircuitBreaker(name="test", failure_threshold=3)

    async def failing_call():
        raise ConnectionError("Failed")

    # Fail 3 times
    for _ in range(3):
        with pytest.raises(ConnectionError):
            await breaker.call(failing_call)

    # Should be open now
    assert breaker.state == CircuitState.OPEN

    # Next call rejected
    with pytest.raises(CircuitOpenError):
        await breaker.call(failing_call)


@pytest.mark.asyncio
async def test_circuit_recovers():
    breaker = CircuitBreaker(
        name="test",
        failure_threshold=1,
        recovery_timeout=0.1,  # Fast for testing
    )

    async def failing_then_succeeding():
        if breaker._failure_count > 0:
            return "success"
        raise ConnectionError("First call fails")

    # Open circuit
    with pytest.raises(ConnectionError):
        await breaker.call(failing_then_succeeding)

    # Wait for recovery
    await asyncio.sleep(0.2)

    # Should succeed now
    result = await breaker.call(failing_then_succeeding)
    assert result == "success"
    assert breaker.state == CircuitState.CLOSED

Integration Tests

@pytest.mark.asyncio
async def test_circuit_isolates_failures():
    """Verify circuit prevents cascade failures."""
    openai_breaker = circuit_breakers["openai"]

    # Simulate OpenAI outage
    with patch("openai.complete", side_effect=ConnectionError):
        # Multiple calls should fail then trip circuit
        for _ in range(5):
            try:
                await analyze_content("test")
            except (ConnectionError, CircuitOpenError):
                pass

    # Circuit should be open
    assert openai_breaker.state == CircuitState.OPEN

    # Anthropic should still work (different circuit)
    anthropic_breaker = circuit_breakers["anthropic"]
    assert anthropic_breaker.state == CircuitState.CLOSED

Step 7: Document and Monitor

Documentation

Add to your service's README:

## Circuit Breakers

| Service | Threshold | Recovery | Fallback |
|---------|-----------|----------|----------|
| openai | 3 failures | 60s | Use gpt-5.2-mini |
| anthropic | 3 failures | 60s | Use cache |
| youtube | 5 failures | 120s | Return partial data |

### Monitoring

- Dashboard: [Grafana Circuit Breakers](...)
- Alerts: PagerDuty channel #resilience
- Runbook: [Circuit Breaker Runbook](...)

Runbook Template

## Circuit Breaker Open - {service}

### Symptoms
- Service returning 503 errors
- Alert: "Circuit {service} opened"
- Dashboard shows circuit in OPEN state

### Impact
- {describe impact on users}

### Resolution
1. Check {service} status page
2. Review logs for failure pattern
3. If transient: wait for auto-recovery
4. If persistent: {escalation steps}

### Verification
1. Circuit state returns to CLOSED
2. Service calls succeeding
3. Metrics returning to normal

Quick Reference

┌─────────────────────────────────────────────────────────────┐
│ Circuit Breaker Quick Reference                              │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│ CREATE:                                                      │
│   breaker = CircuitBreaker("name", failure_threshold=5)     │
│                                                              │
│ USE:                                                         │
│   @breaker                                                   │
│   async def my_function(): ...                              │
│                                                              │
│   result = await breaker.call(func, *args)                  │
│                                                              │
│ HANDLE:                                                      │
│   try:                                                       │
│       await breaker.call(...)                               │
│   except CircuitOpenError:                                  │
│       return fallback_response()                            │
│                                                              │
│ MONITOR:                                                     │
│   breaker.get_status()                                      │
│   breaker.state == CircuitState.OPEN                        │
│                                                              │
│ RESET (manual):                                              │
│   breaker.reset()                                           │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Pre-Deployment Resilience Checklist

Use this checklist before deploying services with resilience patterns.

Circuit Breakers

• [ ] Threshold Configuration
• [ ] Failure threshold set appropriately (not too low, not too high)
• [ ] Recovery timeout allows service to actually recover
• [ ] Sliding window size captures representative sample

• [ ] Fallback Behavior
• [ ] Every circuit breaker has a defined fallback response
• [ ] Fallbacks return meaningful partial data when possible
• [ ] Fallbacks don't call other services with closed circuits

• [ ] Observability
• [ ] State changes logged with structured logging
• [ ] Metrics exported (Prometheus/Langfuse)
• [ ] Alerts configured for OPEN state
• [ ] Dashboard shows circuit status

• [ ] Testing
• [ ] Unit tests for state transitions
• [ ] Integration tests simulate failure scenarios
• [ ] Chaos testing validates circuit behavior under load

Bulkheads

• [ ] Tier Assignment
• [ ] Critical operations in Tier 1 (highest priority)
• [ ] Standard operations in Tier 2
• [ ] Optional/background operations in Tier 3
• [ ] No critical path through Tier 3

• [ ] Capacity Planning
• [ ] Max concurrent based on downstream capacity
• [ ] Queue sizes prevent memory exhaustion
• [ ] Timeouts shorter than caller's timeout

• [ ] Rejection Handling
• [ ] Rejection policy defined per tier
• [ ] HTTP 503 returned with Retry-After header
• [ ] Rejections logged and metriced

• [ ] Testing
• [ ] Load test validates bulkhead isolation
• [ ] Tier 3 failure doesn't affect Tier 1
• [ ] Queue depth monitored under load

Retry Logic

• [ ] Error Classification
• [ ] Retryable vs non-retryable errors defined
• [ ] HTTP status codes classified correctly
• [ ] LLM API errors handled specifically

• [ ] Backoff Strategy
• [ ] Exponential backoff configured
• [ ] Jitter enabled to prevent thundering herd
• [ ] Max delay caps retry storms

• [ ] Limits
• [ ] Max attempts bounded (typically 3-5)
• [ ] Total retry time < caller's timeout
• [ ] Retry budget prevents system overload

• [ ] Testing
• [ ] Transient failures recovered automatically
• [ ] Non-retryable errors fail immediately
• [ ] Retry budget depletes under sustained failures

LLM Resilience

• [ ] Fallback Chain
• [ ] Primary model defined
• [ ] At least one fallback model configured
• [ ] Semantic cache as final fallback
• [ ] Default response for complete outage

• [ ] Token Budget
• [ ] Budget allocation per category
• [ ] Truncation strategy defined
• [ ] Output reserve prevents overflow

• [ ] Rate Limiting
• [ ] Client-side rate limiter configured
• [ ] Respects API provider limits
• [ ] Graceful handling of 429 responses

• [ ] Cost Control
• [ ] Per-request cost tracking
• [ ] Hourly/daily budget alerts
• [ ] Cost circuit breaker configured

Integration

• [ ] Pattern Composition
• [ ] Retry INSIDE circuit breaker
• [ ] Bulkhead wraps retry+circuit
• [ ] Timeout inside all patterns

• [ ] Health Endpoints
• [ ] /health returns 200 (doesn't check circuit)
• [ ] /ready reflects degraded state
• [ ] /resilience shows all pattern status

• [ ] Configuration
• [ ] All thresholds configurable via env vars
• [ ] Defaults documented
• [ ] Per-environment overrides

Observability

• [ ] Logging
• [ ] Structured logging with trace IDs
• [ ] State changes logged at WARN level
• [ ] Rejections logged with context

• [ ] Metrics
• [ ] Circuit state gauge
• [ ] Bulkhead utilization gauge
• [ ] Retry counter
• [ ] Latency histograms

• [ ] Alerting
• [ ] Alert when circuit opens
• [ ] Alert when bulkhead consistently full
• [ ] Alert when retry budget exhausted
• [ ] Runbook links in alerts

Documentation

• [ ] Architecture
• [ ] Resilience patterns documented in ADR
• [ ] Diagram shows pattern composition
• [ ] Tier assignments documented

• [ ] Operations
• [ ] Runbook for circuit open scenarios
• [ ] Runbook for bulkhead exhaustion
• [ ] Manual override procedures documented

• [ ] API Documentation
• [ ] 503 responses documented
• [ ] Retry-After header usage documented
• [ ] Degraded response format documented

Final Verification

• [ ] Load Test
• [ ] System handles expected load
• [ ] Graceful degradation under 2x load
• [ ] Recovery after load spike

• [ ] Chaos Test
• [ ] Dependency failure isolated
• [ ] Recovery automatic when dependency restored
• [ ] No cascading failures

• [ ] Security Review
• [ ] Fallback responses don't leak sensitive data
• [ ] Error messages don't expose internals
• [ ] Rate limits prevent abuse

OrchestKit Workflow Resilience Integration

This example shows how to wire resilience patterns into the OrchestKit analysis pipeline.

Current Architecture

┌────────────────────────────────────────────────────────────────────┐
│                    OrchestKit Analysis Pipeline                     │
├────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Content ─▶ [Supervisor] ─▶ [Agent Fan-Out] ─▶ [Aggregate] ─▶ ... │
│                  │              │    │    │                         │
│                  ▼              ▼    ▼    ▼                         │
│              Agent Selection   A1   A2   A3   (Parallel Analysis)  │
│                                │    │    │                         │
│                                ▼    ▼    ▼                         │
│                              [findings, findings, findings]         │
│                                       │                             │
│                                       ▼                             │
│                              [Synthesize] ─▶ [Quality Gate]        │
│                                                                     │
└────────────────────────────────────────────────────────────────────┘

Resilience Layer Integration

┌────────────────────────────────────────────────────────────────────┐
│                    With Resilience Patterns                         │
├────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Content ─▶ [Rate Limiter] ─▶ [Circuit Breaker: LLM] ─▶ ...      │
│                                         │                           │
│                  ┌──────────────────────┼──────────────────────┐   │
│                  │                      ▼                      │   │
│                  │   ┌──────────────────────────────────────┐  │   │
│                  │   │         TIER 1: CRITICAL             │  │   │
│                  │   │  [Supervisor] [Synthesis] [QualityGate]│  │   │
│                  │   │   Bulkhead: 5 concurrent, 300s timeout │  │   │
│                  │   └──────────────────────────────────────┘  │   │
│                  │                                              │   │
│                  │   ┌──────────────────────────────────────┐  │   │
│                  │   │         TIER 2: STANDARD             │  │   │
│                  │   │  [Tech Comparator] [Impl Planner]     │  │   │
│                  │   │  [Security Auditor] [Learning Synth]  │  │   │
│                  │   │   Bulkhead: 3 concurrent, 120s timeout │  │   │
│                  │   └──────────────────────────────────────┘  │   │
│                  │                                              │   │
│                  │   ┌──────────────────────────────────────┐  │   │
│                  │   │         TIER 3: OPTIONAL             │  │   │
│                  │   │  [Enrichment] [Cache Warm]            │  │   │
│                  │   │   Bulkhead: 2 concurrent, 60s timeout  │  │   │
│                  │   └──────────────────────────────────────┘  │   │
│                  │                                              │   │
│                  └──────────────────────────────────────────────┘   │
│                                                                     │
│   Each agent call wrapped with:                                     │
│   @circuit_breaker(name="agent-{agent_type}")                      │
│   @bulkhead(tier=agent.tier)                                       │
│   @retry(max_attempts=2, base_delay=1.0)                           │
│                                                                     │
└────────────────────────────────────────────────────────────────────┘

Implementation

1. Circuit Breaker Registry

# backend/app/shared/resilience/circuit_breakers.py

from app.core.circuit_breaker import CircuitBreaker

# Per-service circuit breakers
circuit_breakers = {
    # LLM APIs
    "openai": CircuitBreaker(
        name="openai",
        failure_threshold=3,
        recovery_timeout=60.0,
    ),
    "anthropic": CircuitBreaker(
        name="anthropic",
        failure_threshold=3,
        recovery_timeout=60.0,
    ),

    # External APIs
    "youtube": CircuitBreaker(
        name="youtube",
        failure_threshold=5,
        recovery_timeout=120.0,
    ),
    "arxiv": CircuitBreaker(
        name="arxiv",
        failure_threshold=5,
        recovery_timeout=60.0,
    ),
    "github": CircuitBreaker(
        name="github",
        failure_threshold=5,
        recovery_timeout=60.0,
    ),

    # Internal services
    "embedding": CircuitBreaker(
        name="embedding",
        failure_threshold=3,
        recovery_timeout=30.0,
    ),
    "database": CircuitBreaker(
        name="database",
        failure_threshold=2,
        recovery_timeout=15.0,
    ),
}

def get_circuit_breaker(service: str) -> CircuitBreaker:
    """Get or create circuit breaker for service."""
    if service not in circuit_breakers:
        circuit_breakers[service] = CircuitBreaker(
            name=service,
            failure_threshold=5,
            recovery_timeout=30.0,
        )
    return circuit_breakers[service]

2. Bulkhead Registry

# backend/app/shared/resilience/bulkheads.py

from enum import Enum
from .bulkhead import Bulkhead, Tier, BulkheadRegistry

# Agent tier assignments
AGENT_TIERS = {
    # Tier 1: Critical
    "supervisor": Tier.CRITICAL,
    "synthesis": Tier.CRITICAL,
    "quality_gate": Tier.CRITICAL,

    # Tier 2: Standard
    "tech_comparator": Tier.STANDARD,
    "implementation_planner": Tier.STANDARD,
    "security_auditor": Tier.STANDARD,
    "learning_synthesizer": Tier.STANDARD,
    "codebase_analyzer": Tier.STANDARD,
    "prerequisite_mapper": Tier.STANDARD,
    "practical_applicator": Tier.STANDARD,
    "complexity_assessor": Tier.STANDARD,

    # Tier 3: Optional
    "enrichment": Tier.OPTIONAL,
    "cache_warm": Tier.OPTIONAL,
    "metrics": Tier.OPTIONAL,
}

# Create registry
bulkhead_registry = BulkheadRegistry()

# Register bulkheads for each tier
for agent_name, tier in AGENT_TIERS.items():
    bulkhead_registry.register(agent_name, tier)

def get_agent_bulkhead(agent_type: str) -> Bulkhead:
    """Get bulkhead for agent type."""
    tier = AGENT_TIERS.get(agent_type, Tier.STANDARD)
    return bulkhead_registry.get_or_create(agent_type, tier)

3. Resilient Agent Wrapper

# backend/app/shared/resilience/agent_wrapper.py

from functools import wraps
from typing import TypeVar, Callable, Awaitable
import structlog

from .circuit_breakers import get_circuit_breaker
from .bulkheads import get_agent_bulkhead
from .retry_handler import retry, MaxRetriesExceededError

logger = structlog.get_logger()
T = TypeVar("T")

def resilient_agent(
    agent_type: str,
    llm_service: str = "anthropic",
    max_retries: int = 2,
):
    """
    Decorator to wrap agent execution with resilience patterns.

    Applies (in order):
    1. Circuit breaker for LLM service
    2. Bulkhead for concurrency control
    3. Retry for transient failures

    Example:
        @resilient_agent("tech_comparator", llm_service="anthropic")
        async def run_tech_comparator(content: str) -> AgentOutput:
            ...
    """
    def decorator(fn: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]:
        @wraps(fn)
        async def wrapper(*args, **kwargs) -> T:
            circuit = get_circuit_breaker(llm_service)
            bulkhead = get_agent_bulkhead(agent_type)

            # Track for observability
            logger.info(
                "agent_execution_start",
                agent_type=agent_type,
                circuit_state=circuit.state.value,
                bulkhead_active=bulkhead.stats.current_active,
            )

            async def execute():
                # Retry layer (innermost)
                @retry(max_attempts=max_retries, base_delay=1.0)
                async def with_retry():
                    return await fn(*args, **kwargs)

                return await with_retry()

            try:
                # Bulkhead layer
                async def with_bulkhead():
                    return await bulkhead.execute(execute)

                # Circuit breaker layer (outermost)
                result = await circuit.call(with_bulkhead)

                logger.info(
                    "agent_execution_success",
                    agent_type=agent_type,
                )

                return result

            except CircuitOpenError as e:
                logger.warning(
                    "agent_circuit_open",
                    agent_type=agent_type,
                    time_until_recovery=e.time_until_recovery,
                )
                raise

            except BulkheadFullError as e:
                logger.warning(
                    "agent_bulkhead_full",
                    agent_type=agent_type,
                    tier=e.tier.name,
                )
                raise

            except MaxRetriesExceededError as e:
                logger.error(
                    "agent_max_retries_exceeded",
                    agent_type=agent_type,
                    attempts=e.attempts,
                )
                raise

        return wrapper
    return decorator

4. Graph Builder Integration

# backend/app/domains/analysis/workflows/graph_builder.py

from app.shared.resilience.agent_wrapper import resilient_agent
from app.shared.resilience.circuit_breakers import circuit_breakers

async def build_analysis_graph() -> StateGraph:
    """Build the analysis workflow graph with resilience."""

    # Wrap each agent node with resilience
    @resilient_agent("supervisor", llm_service="anthropic")
    async def supervisor_node(state: AnalysisState) -> AnalysisState:
        # Existing supervisor logic
        ...

    @resilient_agent("tech_comparator", llm_service="anthropic")
    async def tech_comparator_node(state: AnalysisState) -> AnalysisState:
        # Existing agent logic
        ...

    # Build graph with wrapped nodes
    graph = StateGraph(AnalysisState)
    graph.add_node("supervisor", supervisor_node)
    graph.add_node("tech_comparator", tech_comparator_node)
    # ... add other nodes

    # Add health check endpoint
    @app.get("/health/resilience")
    async def resilience_health():
        return {
            "circuit_breakers": {
                name: cb.get_status()
                for name, cb in circuit_breakers.items()
            },
            "bulkheads": bulkhead_registry.get_all_status(),
        }

    return graph.compile()

5. LLM Fallback Chain Integration

# backend/app/shared/resilience/llm_chain.py

from app.shared.resilience.llm_fallback_chain import (
    LLMFallbackChain,
    LLMConfig,
)
from app.shared.services.cache.semantic_cache import SemanticCache

# Configure fallback chain for analysis
analysis_llm_chain = LLMFallbackChain(
    primary=AnthropicProvider(
        LLMConfig(
            name="primary",
            model="claude-sonnet-4-5-20251101",
            timeout=60.0,
            max_tokens=8192,
        )
    ),
    fallbacks=[
        OpenAIProvider(
            LLMConfig(
                name="fallback",
                model="gpt-5.2-mini",
                timeout=30.0,
                max_tokens=4096,
            )
        ),
    ],
    cache=SemanticCache(
        redis_client=redis_client,
        threshold=0.85,
    ),
    default_response=lambda p: json.dumps({
        "status": "degraded",
        "message": "Analysis temporarily unavailable",
        "partial_results": None,
    }),
)

6. Observability Integration

# backend/app/shared/resilience/observability.py

from langfuse import Langfuse

langfuse = Langfuse()

def on_circuit_state_change(old_state: str, new_state: str, name: str):
    """Record circuit state changes in Langfuse."""
    langfuse.event(
        name="circuit_breaker_state_change",
        metadata={
            "circuit_name": name,
            "old_state": old_state,
            "new_state": new_state,
        },
        level="WARNING" if new_state == "open" else "INFO",
    )

def on_bulkhead_rejection(name: str, tier: str):
    """Record bulkhead rejections."""
    langfuse.event(
        name="bulkhead_rejection",
        metadata={
            "bulkhead_name": name,
            "tier": tier,
        },
        level="WARNING",
    )

# Wire up callbacks
for name, cb in circuit_breakers.items():
    cb._on_state_change = on_circuit_state_change

for name, bh in bulkhead_registry._bulkheads.items():
    bh._on_rejection = on_bulkhead_rejection

Configuration

Environment Variables

# Circuit Breaker
CIRCUIT_FAILURE_THRESHOLD=5
CIRCUIT_RECOVERY_TIMEOUT=30

# Bulkhead Tier 1
BULKHEAD_TIER1_CONCURRENT=5
BULKHEAD_TIER1_QUEUE=10
BULKHEAD_TIER1_TIMEOUT=300

# Bulkhead Tier 2
BULKHEAD_TIER2_CONCURRENT=3
BULKHEAD_TIER2_QUEUE=5
BULKHEAD_TIER2_TIMEOUT=120

# Bulkhead Tier 3
BULKHEAD_TIER3_CONCURRENT=2
BULKHEAD_TIER3_QUEUE=3
BULKHEAD_TIER3_TIMEOUT=60

# Retry
RETRY_MAX_ATTEMPTS=3
RETRY_BASE_DELAY=1.0
RETRY_MAX_DELAY=30.0

Monitoring Dashboard

┌────────────────────────────────────────────────────────────────────┐
│                   OrchestKit Resilience Dashboard                   │
├────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   CIRCUIT BREAKERS                                                 │
│   ┌─────────────┬──────────┬────────────┬───────────────────┐     │
│   │ Service     │ State    │ Failures   │ Next Recovery     │     │
│   ├─────────────┼──────────┼────────────┼───────────────────┤     │
│   │ anthropic   │ ✅ CLOSED │ 0/5        │ -                 │     │
│   │ openai      │ ✅ CLOSED │ 1/5        │ -                 │     │
│   │ youtube     │ ⚠️ OPEN   │ 5/5        │ 45s               │     │
│   │ embedding   │ ✅ CLOSED │ 0/3        │ -                 │     │
│   └─────────────┴──────────┴────────────┴───────────────────┘     │
│                                                                     │
│   BULKHEADS                                                        │
│   ┌─────────────────┬────────┬─────────┬─────────┬──────────┐     │
│   │ Tier            │ Active │ Queued  │ Max     │ Rejected │     │
│   ├─────────────────┼────────┼─────────┼─────────┼──────────┤     │
│   │ 1: Critical     │ 3/5    │ 1/10    │ 5       │ 0        │     │
│   │ 2: Standard     │ 3/3 ⚠️ │ 4/5     │ 3       │ 12       │     │
│   │ 3: Optional     │ 1/2    │ 0/3     │ 2       │ 45       │     │
│   └─────────────────┴────────┴─────────┴─────────┴──────────┘     │
│                                                                     │
│   RETRY STATS (Last Hour)                                          │
│   • Total Attempts: 1,234                                          │
│   • Success Rate: 94.2%                                            │
│   • Retries Used: 187                                              │
│   • Max Retries Exceeded: 23                                       │
│                                                                     │
└────────────────────────────────────────────────────────────────────┘

Testing Resilience

# backend/tests/integration/test_resilience.py

import pytest
from unittest.mock import AsyncMock, patch

async def test_circuit_opens_after_failures():
    """Circuit should open after threshold failures."""
    breaker = get_circuit_breaker("test-service")

    # Simulate failures
    for _ in range(5):
        with pytest.raises(ConnectionError):
            await breaker.call(failing_function)

    # Next call should be rejected
    with pytest.raises(CircuitOpenError):
        await breaker.call(failing_function)

async def test_bulkhead_rejects_when_full():
    """Bulkhead should reject when queue is full."""
    bulkhead = Bulkhead("test", Tier.STANDARD, max_concurrent=1, queue_size=1)

    # Fill the bulkhead
    task1 = asyncio.create_task(bulkhead.execute(slow_function))
    task2 = asyncio.create_task(bulkhead.execute(slow_function))

    # Third should be rejected
    with pytest.raises(BulkheadFullError):
        await bulkhead.execute(slow_function)

async def test_fallback_chain_uses_fallback():
    """Chain should use fallback when primary fails."""
    chain = LLMFallbackChain(
        primary=FailingProvider(),
        fallbacks=[MockProvider()],
    )

    response = await chain.complete("test prompt")
    assert response.is_fallback
    assert response.source == ResponseSource.FALLBACK

Bulkhead Pattern

Overview

The bulkhead pattern isolates failures by partitioning system resources into independent pools. Named after ship bulkheads that prevent flooding from spreading, it ensures one failing component doesn't bring down the entire system.

Types of Bulkheads

1. Thread Pool Isolation

Dedicated thread pools per service/operation.

┌────────────────────────────────────────────────────────────┐
│                    Thread Pool Bulkhead                     │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   Service A Pool (5 threads)    Service B Pool (3 threads) │
│   ┌─┬─┬─┬─┬─┐                  ┌─┬─┬─┐                    │
│   │█│█│█│░│░│                  │█│░│░│                    │
│   └─┴─┴─┴─┴─┘                  └─┴─┴─┘                    │
│                                                             │
│   If Service A hangs, only 5 threads blocked               │
│   Service B continues with its own 3 threads               │
│                                                             │
└────────────────────────────────────────────────────────────┘

2. Semaphore Isolation

Limits concurrent executions without dedicated threads.

┌────────────────────────────────────────────────────────────┐
│                   Semaphore Bulkhead                        │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   Semaphore: permits=5, current=3                          │
│                                                             │
│   Request 1: acquire() → ✓ (permits=2)                     │
│   Request 2: acquire() → ✓ (permits=1)                     │
│   Request 3: acquire() → ✓ (permits=0)                     │
│   Request 4: acquire() → BLOCKED (queue) or REJECTED       │
│                                                             │
│   Request 1: release() → ✓ (permits=1)                     │
│   Request 4: acquire() → ✓ (permits=0)                     │
│                                                             │
└────────────────────────────────────────────────────────────┘

3. Tier-Based Bulkheads (Recommended for Multi-Agent)

Group operations by criticality.

┌────────────────────────────────────────────────────────────┐
│                    Tier-Based Bulkheads                     │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   TIER 1: CRITICAL (50% resources)                         │
│   ├── Synthesis node                                        │
│   ├── Quality gate                                          │
│   └── User-facing responses                                 │
│                                                             │
│   TIER 2: STANDARD (35% resources)                         │
│   ├── Content analysis agents                               │
│   ├── Data processing                                       │
│   └── API integrations                                      │
│                                                             │
│   TIER 3: OPTIONAL (15% resources)                         │
│   ├── Enrichment                                            │
│   ├── Caching warmup                                        │
│   └── Analytics                                             │
│                                                             │
│   When Tier 3 exhausted → operations queued/dropped         │
│   Tier 1 & 2 continue unaffected                           │
│                                                             │
└────────────────────────────────────────────────────────────┘

Configuration for OrchestKit

Agent Tier Assignment

Rejection Policies

class RejectionPolicy(Enum):
    ABORT = "abort"           # Return error immediately
    CALLER_RUNS = "caller"    # Execute in caller's context (blocking)
    DISCARD = "discard"       # Silently drop (for optional ops)
    QUEUE = "queue"           # Wait in bounded queue

# Per-tier policies
TIER_POLICIES = {
    1: RejectionPolicy.QUEUE,      # Critical: wait for slot
    2: RejectionPolicy.CALLER_RUNS, # Standard: degrade caller
    3: RejectionPolicy.DISCARD,     # Optional: skip if busy
}

Implementation Pattern (Python asyncio)

from asyncio import Semaphore, wait_for, TimeoutError
from collections import defaultdict
from enum import Enum
from typing import TypeVar, Callable, Awaitable

T = TypeVar("T")

class Tier(Enum):
    CRITICAL = 1
    STANDARD = 2
    OPTIONAL = 3

class Bulkhead:
    def __init__(self, tier: Tier, max_concurrent: int, queue_size: int, timeout: float):
        self.tier = tier
        self.semaphore = Semaphore(max_concurrent)
        self.queue_size = queue_size
        self.timeout = timeout
        self.waiting = 0
        self.active = 0

    async def execute(self, fn: Callable[[], Awaitable[T]]) -> T:
        # Check queue
        if self.waiting >= self.queue_size:
            raise BulkheadFullError(f"Tier {self.tier.name} queue full")

        self.waiting += 1
        try:
            # Acquire with timeout
            await wait_for(self.semaphore.acquire(), timeout=self.timeout)
            self.waiting -= 1
            self.active += 1

            try:
                return await wait_for(fn(), timeout=self.timeout)
            finally:
                self.active -= 1
                self.semaphore.release()
        except TimeoutError:
            self.waiting -= 1
            raise BulkheadTimeoutError(f"Tier {self.tier.name} timeout")

Best Practices (2026)

1. Size Based on Downstream Capacity

# BAD: Arbitrary numbers
bulkhead = Bulkhead(max_concurrent=100)

# GOOD: Based on downstream limits
# If OpenAI allows 60 RPM, don't have 100 concurrent
bulkhead = Bulkhead(max_concurrent=10)  # 10 concurrent * 6s avg = 60 RPM

2. Monitor Queue Depth

async def execute_with_metrics(self, fn):
    # Metric: queue depth
    metrics.gauge("bulkhead.queue_depth", self.waiting, tags={"tier": self.tier.name})

    # Metric: active requests
    metrics.gauge("bulkhead.active", self.active, tags={"tier": self.tier.name})

    # Alert when queue consistently > 80% full
    if self.waiting > self.queue_size * 0.8:
        logger.warning(f"Bulkhead queue high", tier=self.tier.name, depth=self.waiting)

    return await self.execute(fn)

3. Graceful Degradation by Tier

async def run_analysis(content: str) -> Analysis:
    results = {}

    # Tier 1: Must succeed
    results["core"] = await tier1_bulkhead.execute(
        lambda: analyze_core(content)
    )

    # Tier 2: Best effort
    try:
        results["enriched"] = await tier2_bulkhead.execute(
            lambda: enrich_analysis(content)
        )
    except BulkheadFullError:
        results["enriched"] = None  # Skip enrichment

    # Tier 3: Optional
    try:
        await tier3_bulkhead.execute(
            lambda: warm_cache(results)
        )
    except (BulkheadFullError, BulkheadTimeoutError):
        pass  # Don't even log

    return Analysis(**results)

4. Dynamic Tier Adjustment

class AdaptiveBulkhead:
    """Adjusts tier capacity based on system load."""

    def adjust_for_load(self, cpu_percent: float, memory_percent: float):
        if cpu_percent > 80 or memory_percent > 85:
            # Reduce optional tier
            self.tiers[Tier.OPTIONAL].max_concurrent = 1
            self.tiers[Tier.STANDARD].max_concurrent = 2
        elif cpu_percent < 50 and memory_percent < 60:
            # Restore capacity
            self.tiers[Tier.OPTIONAL].max_concurrent = 2
            self.tiers[Tier.STANDARD].max_concurrent = 3

Anti-Patterns

1. Too Many Bulkheads

# BAD: Bulkhead per endpoint
bulkheads = {
    "/api/v1/users": Bulkhead(5),
    "/api/v1/users/{id}": Bulkhead(5),
    "/api/v1/users/{id}/profile": Bulkhead(5),
    # ... 50 more
}
# Result: Complexity nightmare, no real isolation

# GOOD: Bulkhead per tier/dependency
bulkheads = {
    "database_read": Bulkhead(10),
    "database_write": Bulkhead(3),
    "external_api": Bulkhead(5),
}

2. Ignoring Rejection Handling

# BAD: Exception bubbles up as 500
@app.post("/analyze")
async def analyze(content: str):
    return await bulkhead.execute(lambda: do_analysis(content))
    # BulkheadFullError → 500 Internal Server Error

# GOOD: Proper error handling
@app.post("/analyze")
async def analyze(content: str):
    try:
        return await bulkhead.execute(lambda: do_analysis(content))
    except BulkheadFullError:
        raise HTTPException(
            status_code=503,
            detail="Service busy, please retry",
            headers={"Retry-After": "30"}
        )

3. No Correlation with Circuit Breaker

# BAD: Bulkhead fills up, circuit never opens
# All slots blocked on slow service

# GOOD: Combine patterns
@circuit_breaker(failure_threshold=5)
@bulkhead(tier=Tier.STANDARD)
async def call_external_service():
    ...
# Slow calls → timeouts → circuit opens → bulkhead cleared

Monitoring Dashboard

┌────────────────────────────────────────────────────────────┐
│                  Bulkhead Status Dashboard                  │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   TIER 1: CRITICAL         [████████░░] 8/10 active        │
│   Queue: 2/10              [██░░░░░░░░] 2/10 queued        │
│   Rejected (1h): 0         Timeouts (1h): 1                │
│                                                             │
│   TIER 2: STANDARD         [██████████] 3/3 active ⚠️      │
│   Queue: 5/5 FULL          [██████████] 5/5 queued ⚠️      │
│   Rejected (1h): 23        Timeouts (1h): 5                │
│                                                             │
│   TIER 3: OPTIONAL         [█░░░░░░░░░] 1/2 active         │
│   Queue: 0/3               [░░░░░░░░░░] 0/3 queued         │
│   Rejected (1h): 156       Timeouts (1h): 0                │
│                                                             │
└────────────────────────────────────────────────────────────┘

Circuit Breaker Pattern

Overview

The circuit breaker pattern prevents cascade failures by "tripping" when a downstream service exceeds failure thresholds. Named after electrical circuit breakers, it protects your system from repeated failures.

States

CLOSED (Normal Operation)

• All requests pass through
• Failures are counted within a sliding window
• Success resets failure count (or decrements in sliding window)
• Transitions to OPEN when failures >= threshold

OPEN (Failing Fast)

• All requests immediately rejected
• Returns fallback response or error
• No calls made to downstream service
• After recovery_timeout, transitions to HALF_OPEN

HALF_OPEN (Recovery Probe)

• Limited requests allowed (probe requests)
• If probe succeeds → CLOSED
• If probe fails → OPEN (reset recovery timer)

State Machine

              failures >= threshold
    CLOSED ──────────────────────────────▶ OPEN
       ▲                                      │
       │                                      │
       │ probe succeeds              timeout  │
       │                              expires │
       │         ┌─────────────┐             │
       └─────────│  HALF_OPEN  │◀────────────┘
                 └─────────────┘
                       │
                       │ probe fails
                       ▼
                     OPEN

Configuration Parameters

Best Practices (2026)

1. Use Sliding Windows, Not Fixed Counters

# BAD: Fixed counter resets on success
if failures >= 5:
    open_circuit()
if success:
    failures = 0  # One success resets everything!

# GOOD: Sliding window with time decay
window = deque(maxlen=10)
window.append(("fail", time.time()))
failure_rate = sum(1 for r, _ in window if r == "fail") / len(window)
if failure_rate >= 0.5:
    open_circuit()

2. Separate Health Checks from Circuit State

# Health endpoint should NOT check circuit state
@app.get("/health")
async def health():
    return {"status": "healthy"}  # Always returns 200

# Readiness endpoint CAN check circuit state
@app.get("/ready")
async def ready():
    if circuit.is_open:
        return {"status": "degraded", "reason": "circuit_open"}, 503
    return {"status": "ready"}

3. Include Observability

def on_state_change(from_state: str, to_state: str, service: str):
    # Metric
    metrics.increment(f"circuit_breaker.{service}.state_change",
                      tags={"from": from_state, "to": to_state})

    # Log
    logger.warning(f"Circuit breaker state change",
                   service=service, from_state=from_state, to_state=to_state)

    # Alert (only on OPEN)
    if to_state == "OPEN":
        alert_service.send(
            severity="warning",
            message=f"Circuit breaker opened for {service}",
            runbook="https://docs.internal/runbooks/circuit-breaker"
        )

4. Provide Meaningful Fallbacks

async def get_analysis_with_fallback(content: str) -> Analysis:
    try:
        return await circuit_breaker.call(analyze_content, content)
    except CircuitOpenError:
        # Fallback 1: Cached result
        cached = await cache.get(f"analysis:{hash(content)}")
        if cached:
            return Analysis.from_cache(cached, is_stale=True)

        # Fallback 2: Simplified analysis
        return Analysis(
            status="degraded",
            message="Full analysis unavailable, showing basic info",
            basic_info=extract_basic_info(content)
        )

5. Per-Service Breakers

# BAD: Single breaker for all services
global_breaker = CircuitBreaker()

# GOOD: Per-service breakers
breakers = {
    "openai": CircuitBreaker(failure_threshold=3, recovery_timeout=60),
    "anthropic": CircuitBreaker(failure_threshold=5, recovery_timeout=30),
    "youtube_api": CircuitBreaker(failure_threshold=10, recovery_timeout=120),
}

Anti-Patterns

1. Opening Too Quickly

# BAD: Opens on first failure
CircuitBreaker(failure_threshold=1)  # One blip = outage

# GOOD: Tolerates transient failures
CircuitBreaker(failure_threshold=5, sliding_window_size=10)

2. Recovery Timeout Too Short

# BAD: Hammers failing service
CircuitBreaker(recovery_timeout=5)  # Tries every 5 seconds

# GOOD: Gives service time to recover
CircuitBreaker(recovery_timeout=30)  # 30 seconds minimum

3. No Fallback

# BAD: Just throws error
async def call():
    if circuit.is_open:
        raise CircuitOpenError()  # User sees error page

# GOOD: Graceful degradation
async def call():
    if circuit.is_open:
        return await fallback_handler()  # User sees partial data

Integration with Other Patterns

Circuit Breaker + Retry

# Retry INSIDE circuit breaker
@circuit_breaker
@retry(max_attempts=3, backoff=exponential)
async def call_service():
    ...

# Circuit only sees final result after retries exhausted

Circuit Breaker + Bulkhead

# Bulkhead limits concurrency, circuit limits failures
@circuit_breaker(service="analysis")
@bulkhead(tier=2, max_concurrent=3)
async def analyze():
    ...

Circuit Breaker + Timeout

# Timeout INSIDE circuit breaker
@circuit_breaker
@timeout(seconds=30)
async def call_service():
    ...

# Timeout counts as failure toward circuit threshold

Monitoring Queries

Prometheus

# Circuit state changes per minute
rate(circuit_breaker_state_changes_total[5m])

# Percentage of time in OPEN state
avg_over_time(circuit_breaker_state{state="open"}[1h])

# Requests rejected due to open circuit
rate(circuit_breaker_rejected_total[5m])

Langfuse (LLM-specific)

# Tag traces with circuit state
trace.update(metadata={
    "circuit_state": circuit.state,
    "circuit_failure_count": circuit.failure_count,
})

Error Classification

Overview

Proper error classification is the foundation of resilience. Different errors require different handling strategies: retry, fallback, fail fast, or alert.

Error Classification Matrix

┌────────────────────────────────────────────────────────────────────┐
│                       Error Classification Matrix                   │
├────────────────────────────────────────────────────────────────────┤
│                                                                     │
│                    TRANSIENT                 PERMANENT              │
│              ┌─────────────────────┬─────────────────────┐         │
│              │                     │                     │         │
│   EXTERNAL   │  • Rate limits      │  • Invalid API key  │         │
│   (API/Net)  │  • Timeouts         │  • 403 Forbidden    │         │
│              │  • 502/503/504      │  • 404 Not Found    │         │
│              │  • Connection reset │  • 400 Bad Request  │         │
│              │                     │                     │         │
│              │  ACTION: Retry      │  ACTION: Fail Fast  │         │
│              │  with backoff       │  Log & Alert        │         │
│              │                     │                     │         │
│              ├─────────────────────┼─────────────────────┤         │
│              │                     │                     │         │
│   INTERNAL   │  • DB connection    │  • Schema error     │         │
│   (System)   │  • Memory pressure  │  • Logic bug        │         │
│              │  • Lock contention  │  • Missing config   │         │
│              │  • Resource exhaust │  • Invalid state    │         │
│              │                     │                     │         │
│              │  ACTION: Retry      │  ACTION: Fail Fast  │         │
│              │  Circuit breaker    │  Fix code, restart  │         │
│              │                     │                     │         │
│              └─────────────────────┴─────────────────────┘         │
│                                                                     │
└────────────────────────────────────────────────────────────────────┘

HTTP Status Code Classification

Retryable (Transient)

Non-Retryable (Permanent)

LLM API Error Classification

OpenAI Errors

OPENAI_RETRYABLE = {
    "rate_limit_exceeded",     # Retry with backoff
    "server_error",            # Retry with backoff
    "timeout",                 # Retry immediately
    "overloaded",              # Retry with longer backoff
}

OPENAI_NON_RETRYABLE = {
    "invalid_api_key",         # Fix config
    "invalid_request_error",   # Fix request
    "context_length_exceeded", # Reduce input (special handling)
    "content_policy_violation",# Change content
    "insufficient_quota",      # Add credits
    "model_not_found",         # Fix model name
}

Anthropic Errors

ANTHROPIC_RETRYABLE = {
    "overloaded_error",        # Retry with backoff
    "api_error",               # Retry with backoff
    "rate_limit_error",        # Retry with Retry-After
}

ANTHROPIC_NON_RETRYABLE = {
    "authentication_error",    # Fix API key
    "permission_error",        # Check permissions
    "invalid_request_error",   # Fix request format
    "not_found_error",         # Fix resource reference
}

Exception Classification Helper

from enum import Enum
from typing import Union, Type
import httpx

class ErrorCategory(Enum):
    RETRYABLE = "retryable"           # Transient, retry with backoff
    NON_RETRYABLE = "non_retryable"   # Permanent, fail fast
    CIRCUIT_TRIP = "circuit_trip"     # Count toward circuit breaker
    ALERTABLE = "alertable"           # Should trigger alert
    DEGRADABLE = "degradable"         # Can fall back to alternative

class ErrorClassifier:
    """Classify errors for resilience handling."""

    RETRYABLE_STATUS_CODES = {408, 429, 500, 502, 503, 504}
    NON_RETRYABLE_STATUS_CODES = {400, 401, 403, 404, 405, 422}
    ALERTABLE_STATUS_CODES = {401, 403, 500}

    RETRYABLE_EXCEPTIONS = {
        ConnectionError,
        TimeoutError,
        ConnectionResetError,
        ConnectionRefusedError,
        BrokenPipeError,
        httpx.ConnectError,
        httpx.ConnectTimeout,
        httpx.ReadTimeout,
    }

    def classify(self, error: Exception) -> set[ErrorCategory]:
        """Return set of applicable error categories."""
        categories = set()

        # HTTP errors
        if hasattr(error, "status_code"):
            code = error.status_code
            if code in self.RETRYABLE_STATUS_CODES:
                categories.add(ErrorCategory.RETRYABLE)
            if code in self.NON_RETRYABLE_STATUS_CODES:
                categories.add(ErrorCategory.NON_RETRYABLE)
            if code in self.ALERTABLE_STATUS_CODES:
                categories.add(ErrorCategory.ALERTABLE)
            if code >= 500:
                categories.add(ErrorCategory.CIRCUIT_TRIP)

        # Exception types
        if type(error) in self.RETRYABLE_EXCEPTIONS:
            categories.add(ErrorCategory.RETRYABLE)
            categories.add(ErrorCategory.CIRCUIT_TRIP)

        # LLM-specific errors
        if hasattr(error, "code"):
            categories.update(self._classify_llm_error(error.code))

        # Default: non-retryable if nothing matched
        if not categories:
            categories.add(ErrorCategory.NON_RETRYABLE)
            categories.add(ErrorCategory.ALERTABLE)

        return categories

    def _classify_llm_error(self, error_code: str) -> set[ErrorCategory]:
        """Classify LLM API error codes."""
        categories = set()

        retryable_codes = {
            "rate_limit_exceeded", "server_error", "timeout",
            "overloaded", "overloaded_error", "api_error",
        }

        if error_code in retryable_codes:
            categories.add(ErrorCategory.RETRYABLE)
            categories.add(ErrorCategory.CIRCUIT_TRIP)
        else:
            categories.add(ErrorCategory.NON_RETRYABLE)

        if error_code in {"context_length_exceeded"}:
            categories.add(ErrorCategory.DEGRADABLE)

        return categories

    def should_retry(self, error: Exception) -> bool:
        """Quick check if error should be retried."""
        return ErrorCategory.RETRYABLE in self.classify(error)

    def should_trip_circuit(self, error: Exception) -> bool:
        """Check if error should count toward circuit breaker."""
        return ErrorCategory.CIRCUIT_TRIP in self.classify(error)

    def should_alert(self, error: Exception) -> bool:
        """Check if error should trigger an alert."""
        return ErrorCategory.ALERTABLE in self.classify(error)

Usage in Resilience Patterns

With Retry

classifier = ErrorClassifier()

async def call_with_smart_retry(fn, *args, **kwargs):
    for attempt in range(max_attempts):
        try:
            return await fn(*args, **kwargs)
        except Exception as e:
            categories = classifier.classify(e)

            if ErrorCategory.NON_RETRYABLE in categories:
                logger.error(f"Non-retryable error: {e}")
                raise

            if ErrorCategory.ALERTABLE in categories:
                await alert_service.send(
                    severity="warning",
                    message=f"Retryable error in {fn.__name__}",
                    error=str(e),
                )

            if attempt < max_attempts - 1:
                await asyncio.sleep(backoff(attempt))

    raise MaxRetriesExceeded()

With Circuit Breaker

class SmartCircuitBreaker:
    def __init__(self, classifier: ErrorClassifier):
        self.classifier = classifier
        self.failure_count = 0

    async def call(self, fn, *args, **kwargs):
        try:
            result = await fn(*args, **kwargs)
            self.failure_count = 0
            return result
        except Exception as e:
            if self.classifier.should_trip_circuit(e):
                self.failure_count += 1
                if self.failure_count >= self.threshold:
                    self.open_circuit()
            raise

With Fallback

async def call_with_fallback(primary_fn, fallback_fn, *args):
    try:
        return await primary_fn(*args)
    except Exception as e:
        categories = classifier.classify(e)

        if ErrorCategory.DEGRADABLE in categories:
            logger.info(f"Degrading to fallback: {e}")
            return await fallback_fn(*args)

        raise

Error Context Enrichment

class EnrichedError(Exception):
    """Exception with classification and context."""

    def __init__(
        self,
        original: Exception,
        classifier: ErrorClassifier,
        context: dict = None,
    ):
        self.original = original
        self.categories = classifier.classify(original)
        self.context = context or {}
        self.timestamp = datetime.now(UTC)
        self.trace_id = get_current_trace_id()

        super().__init__(str(original))

    @property
    def is_retryable(self) -> bool:
        return ErrorCategory.RETRYABLE in self.categories

    @property
    def should_alert(self) -> bool:
        return ErrorCategory.ALERTABLE in self.categories

    def to_dict(self) -> dict:
        return {
            "error": str(self.original),
            "type": type(self.original).__name__,
            "categories": [c.value for c in self.categories],
            "context": self.context,
            "timestamp": self.timestamp.isoformat(),
            "trace_id": self.trace_id,
        }

Best Practices

1. Default to non-retryable: Unknown errors should fail fast 2. Log all classifications: Helps tune classification rules 3. Include context: Error classification without context is useless 4. Review regularly: New error types emerge, update rules 5. Test classification: Unit test your classification logic

LLM-Specific Resilience Patterns

Overview

LLM APIs have unique failure modes that require specialized resilience patterns. This guide covers fallback chains, token budget management, rate limiting, and cost optimization through resilience.

Unique LLM Failure Modes

┌────────────────────────────────────────────────────────────┐
│                    LLM Failure Taxonomy                     │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   TRANSIENT (Retry)          PERMANENT (Fail Fast)         │
│   ─────────────────          ──────────────────────         │
│   • rate_limit_exceeded      • invalid_api_key             │
│   • model_overloaded         • content_policy_violation    │
│   • server_error             • invalid_request_error       │
│   • timeout                  • insufficient_quota          │
│   • context_length_exceeded* • model_not_found             │
│                                                             │
│   * Can retry with truncation                              │
│                                                             │
│   DEGRADABLE (Fallback)      COSTLY (Budget Control)       │
│   ─────────────────────      ────────────────────────       │
│   • Primary model down       • Large context = high cost   │
│   • Quality below threshold  • Streaming = token overhead  │
│   • Latency too high         • Retries multiply cost       │
│                                                             │
└────────────────────────────────────────────────────────────┘

Pattern 1: Fallback Chain

┌────────────────────────────────────────────────────────────┐
│                    LLM Fallback Chain                       │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   Request ─▶ [Primary Model] ──success──▶ Response         │
│                    │                                        │
│                  fail (timeout, rate limit, error)          │
│                    ▼                                        │
│              [Fallback Model] ──success──▶ Response        │
│                    │                                        │
│                  fail                                       │
│                    ▼                                        │
│              [Semantic Cache] ──hit──▶ Response            │
│                    │                                        │
│                  miss                                       │
│                    ▼                                        │
│              [Default Response] ──▶ Graceful Degradation   │
│                                                             │
└────────────────────────────────────────────────────────────┘

Implementation

from dataclasses import dataclass
from typing import Optional, List, Callable, Awaitable

@dataclass
class LLMConfig:
    name: str
    model: str
    api_key: str
    timeout: float = 30.0
    max_tokens: int = 4096
    temperature: float = 0.7

class FallbackChain:
    def __init__(
        self,
        primary: LLMConfig,
        fallbacks: List[LLMConfig],
        cache: Optional[SemanticCache] = None,
        default_response: Optional[Callable[[str], str]] = None,
    ):
        self.primary = primary
        self.fallbacks = fallbacks
        self.cache = cache
        self.default_response = default_response

    async def complete(self, prompt: str, **kwargs) -> LLMResponse:
        # Try primary
        try:
            return await self._call_model(self.primary, prompt, **kwargs)
        except RetryableError as e:
            logger.warning(f"Primary model failed: {e}")

        # Try fallbacks
        for fallback in self.fallbacks:
            try:
                response = await self._call_model(fallback, prompt, **kwargs)
                response.is_fallback = True
                return response
            except RetryableError as e:
                logger.warning(f"Fallback {fallback.name} failed: {e}")

        # Try cache
        if self.cache:
            cached = await self.cache.get_similar(prompt, threshold=0.85)
            if cached:
                logger.info("Returning cached response")
                return LLMResponse(
                    content=cached.content,
                    is_cached=True,
                    cache_similarity=cached.similarity,
                )

        # Default response
        if self.default_response:
            return LLMResponse(
                content=self.default_response(prompt),
                is_degraded=True,
            )

        raise AllModelsFailedError("All LLM options exhausted")

Recommended Fallback Configurations

Pattern 2: Token Budget Management

┌────────────────────────────────────────────────────────────┐
│                     Token Budget Guard                      │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   Context Window: 128K tokens                              │
│   ┌────────────────────────────────────────────────────┐   │
│   │████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░│   │
│   └────────────────────────────────────────────────────┘   │
│   Used: 32K (25%)         Available: 96K                   │
│                                                             │
│   Budget Allocation:                                        │
│   ├── System prompt:     2K  (fixed)                       │
│   ├── Conversation:     20K  (sliding window)              │
│   ├── Retrieved docs:    8K  (chunked)                     │
│   ├── Output reserve:    2K  (for response)                │
│   └── Safety margin:     5K  (overflow buffer)             │
│                                                             │
│   When approaching limit:                                   │
│   1. Summarize conversation history (4:1 compression)      │
│   2. Reduce retrieved chunks                               │
│   3. Truncate oldest messages                              │
│   4. Fail with "context too large" error                   │
│                                                             │
└────────────────────────────────────────────────────────────┘

Implementation

import tiktoken
from dataclasses import dataclass
from typing import List

@dataclass
class BudgetAllocation:
    system_prompt: int = 2000
    conversation: int = 20000
    retrieved_docs: int = 8000
    output_reserve: int = 2000
    safety_margin: int = 5000

    @property
    def total_budget(self) -> int:
        return (
            self.system_prompt +
            self.conversation +
            self.retrieved_docs +
            self.output_reserve +
            self.safety_margin
        )

class TokenBudgetGuard:
    def __init__(
        self,
        model: str,
        context_limit: int,
        allocation: BudgetAllocation = None,
    ):
        self.encoding = tiktoken.encoding_for_model(model)
        self.context_limit = context_limit
        self.allocation = allocation or BudgetAllocation()

    def count_tokens(self, text: str) -> int:
        return len(self.encoding.encode(text))

    def fit_to_budget(
        self,
        system_prompt: str,
        messages: List[dict],
        retrieved_docs: List[str],
    ) -> tuple[str, List[dict], List[str]]:
        """Fit content to token budget, compressing as needed."""

        # Count fixed costs
        system_tokens = self.count_tokens(system_prompt)
        if system_tokens > self.allocation.system_prompt:
            raise TokenBudgetError("System prompt exceeds budget")

        # Fit messages with sliding window
        fitted_messages = self._fit_messages(
            messages,
            self.allocation.conversation
        )

        # Fit retrieved docs
        fitted_docs = self._fit_docs(
            retrieved_docs,
            self.allocation.retrieved_docs
        )

        return system_prompt, fitted_messages, fitted_docs

    def _fit_messages(self, messages: List[dict], budget: int) -> List[dict]:
        """Keep most recent messages that fit in budget."""
        fitted = []
        used = 0

        # Always keep system message if present
        for msg in reversed(messages):
            tokens = self.count_tokens(msg["content"])
            if used + tokens <= budget:
                fitted.insert(0, msg)
                used += tokens
            elif msg["role"] == "system":
                # Summarize old messages
                summary = self._summarize_old_messages(messages[:-len(fitted)])
                fitted.insert(0, {"role": "system", "content": summary})
                break

        return fitted

    def _fit_docs(self, docs: List[str], budget: int) -> List[str]:
        """Keep highest-scoring docs that fit in budget."""
        fitted = []
        used = 0

        for doc in docs:  # Assume already sorted by relevance
            tokens = self.count_tokens(doc)
            if used + tokens <= budget:
                fitted.append(doc)
                used += tokens
            else:
                break

        return fitted

Pattern 3: Rate Limit Management

from asyncio import Semaphore, sleep
from collections import deque
from time import time

class RateLimiter:
    """Token bucket rate limiter for LLM APIs."""

    def __init__(
        self,
        requests_per_minute: int = 60,
        tokens_per_minute: int = 100000,
    ):
        self.rpm_limit = requests_per_minute
        self.tpm_limit = tokens_per_minute
        self.request_times = deque(maxlen=rpm_limit)
        self.token_counts = deque(maxlen=1000)
        self.semaphore = Semaphore(rpm_limit)

    async def acquire(self, estimated_tokens: int):
        """Wait until rate limit allows the request."""
        async with self.semaphore:
            now = time()

            # Check RPM
            while len(self.request_times) >= self.rpm_limit:
                oldest = self.request_times[0]
                wait_time = 60 - (now - oldest)
                if wait_time > 0:
                    await sleep(wait_time)
                    now = time()
                self.request_times.popleft()

            # Check TPM
            recent_tokens = sum(
                t for t, ts in self.token_counts
                if now - ts < 60
            )
            if recent_tokens + estimated_tokens > self.tpm_limit:
                wait_time = 60 - (now - self.token_counts[0][1])
                await sleep(wait_time)

            # Record this request
            self.request_times.append(now)

    def record_usage(self, actual_tokens: int):
        """Record actual token usage after request completes."""
        self.token_counts.append((actual_tokens, time()))

Pattern 4: Cost Control Circuit Breaker

class CostCircuitBreaker:
    """Opens when LLM costs exceed budget."""

    def __init__(
        self,
        hourly_budget: float = 10.0,  # $10/hour
        daily_budget: float = 100.0,   # $100/day
        alert_threshold: float = 0.8,  # Alert at 80%
    ):
        self.hourly_budget = hourly_budget
        self.daily_budget = daily_budget
        self.alert_threshold = alert_threshold
        self.hourly_spend = 0.0
        self.daily_spend = 0.0
        self.last_hour_reset = time()
        self.last_day_reset = time()

    def record_cost(self, input_tokens: int, output_tokens: int, model: str):
        """Record cost and check budget."""
        self._reset_if_needed()

        cost = self._calculate_cost(input_tokens, output_tokens, model)
        self.hourly_spend += cost
        self.daily_spend += cost

        # Alert at threshold
        if self.hourly_spend > self.hourly_budget * self.alert_threshold:
            logger.warning(
                f"Hourly LLM budget at {self.hourly_spend/self.hourly_budget:.0%}"
            )

        # Trip circuit at limit
        if self.hourly_spend >= self.hourly_budget:
            raise CostBudgetExceeded("Hourly LLM budget exceeded")

        if self.daily_spend >= self.daily_budget:
            raise CostBudgetExceeded("Daily LLM budget exceeded")

    def _calculate_cost(self, input_tokens: int, output_tokens: int, model: str) -> float:
        """Calculate cost based on model pricing (Dec 2025)."""
        PRICING = {
            "claude-sonnet-4-5-20251101": {"input": 3.0, "output": 15.0},
            "gpt-5.2": {"input": 2.5, "output": 10.0},
            "gpt-5.2-mini": {"input": 0.15, "output": 0.60},
            "claude-haiku-4-5-20251101": {"input": 0.80, "output": 4.0},
        }

        prices = PRICING.get(model, {"input": 1.0, "output": 3.0})
        return (
            (input_tokens / 1_000_000) * prices["input"] +
            (output_tokens / 1_000_000) * prices["output"]
        )

Pattern 5: Quality-Aware Fallback

class QualityAwareFallback:
    """Falls back when response quality is below threshold."""

    def __init__(
        self,
        primary_chain: FallbackChain,
        quality_evaluator: Callable[[str, str], float],
        quality_threshold: float = 0.7,
        max_retries: int = 2,
    ):
        self.chain = primary_chain
        self.evaluate = quality_evaluator
        self.threshold = quality_threshold
        self.max_retries = max_retries

    async def complete(self, prompt: str, **kwargs) -> LLMResponse:
        for attempt in range(self.max_retries + 1):
            response = await self.chain.complete(prompt, **kwargs)

            # Evaluate quality
            quality_score = await self.evaluate(prompt, response.content)

            if quality_score >= self.threshold:
                response.quality_score = quality_score
                return response

            logger.warning(
                f"Response quality {quality_score:.2f} below threshold",
                attempt=attempt + 1,
            )

            # Try with different parameters on retry
            if attempt < self.max_retries:
                kwargs["temperature"] = max(0.3, kwargs.get("temperature", 0.7) - 0.2)

        # Return best effort with warning
        response.quality_warning = f"Below threshold: {quality_score:.2f}"
        return response

Best Practices (2026)

1. Always have a fallback: Even a cached or default response is better than an error 2. Monitor costs per-request: Track token usage in traces (Langfuse) 3. Use streaming for long responses: Better UX and partial results on failure 4. Cache aggressively: Semantic cache with 0.85+ similarity saves 60-80% costs 5. Set appropriate timeouts: 30s for completion, 5s for embeddings 6. Log all fallback events: Critical for understanding system behavior

OrchestKit Integration

# Example integration for OrchestKit analysis pipeline

llm_chain = FallbackChain(
    primary=LLMConfig(
        name="primary",
        model="claude-sonnet-4-5-20251101",
        timeout=30.0,
    ),
    fallbacks=[
        LLMConfig(
            name="fallback",
            model="gpt-5.2-mini",
            timeout=20.0,
        ),
    ],
    cache=semantic_cache,  # Redis-backed
    default_response=lambda p: "Analysis temporarily unavailable",
)

budget_guard = TokenBudgetGuard(
    model="claude-sonnet-4-5-20251101",
    context_limit=200000,
    allocation=BudgetAllocation(
        system_prompt=3000,
        conversation=10000,
        retrieved_docs=15000,  # RAG context
        output_reserve=4000,
        safety_margin=5000,
    ),
)

rate_limiter = RateLimiter(
    requests_per_minute=50,  # Leave headroom
    tokens_per_minute=80000,
)

Retry Strategies

Overview

Retry strategies handle transient failures by automatically re-attempting operations. The key is knowing when to retry, how long to wait, and when to give up.

Core Concepts

Exponential Backoff with Jitter

┌────────────────────────────────────────────────────────────┐
│              Exponential Backoff + Full Jitter             │
├────────────────────────────────────────────────────────────┤
│                                                             │
│   Attempt  Base Delay   With Jitter (random 0-base)        │
│   ───────  ──────────   ─────────────────────────          │
│      1        1s              0.0s - 1.0s                  │
│      2        2s              0.0s - 2.0s                  │
│      3        4s              0.0s - 4.0s                  │
│      4        8s              0.0s - 8.0s                  │
│      5       16s              0.0s - 16.0s                 │
│                                                             │
│   Formula: sleep = random(0, min(cap, base * 2^attempt))   │
│                                                             │
│   Full jitter prevents thundering herd when many clients   │
│   retry simultaneously after an outage.                    │
│                                                             │
└────────────────────────────────────────────────────────────┘

Jitter Strategies

Error Classification

Retryable Errors

RETRYABLE_ERRORS = {
    # HTTP Status Codes
    408: "Request Timeout",
    429: "Too Many Requests",
    500: "Internal Server Error",
    502: "Bad Gateway",
    503: "Service Unavailable",
    504: "Gateway Timeout",

    # Python Exceptions
    ConnectionError,
    TimeoutError,
    ConnectionResetError,
    BrokenPipeError,

    # LLM API Errors
    "rate_limit_exceeded",
    "model_overloaded",
    "server_error",
    "timeout",
    "context_length_exceeded",  # Retry with truncation
}

def is_retryable(error: Exception) -> bool:
    # HTTP errors
    if hasattr(error, "status_code"):
        return error.status_code in RETRYABLE_ERRORS

    # Exception types
    if type(error) in RETRYABLE_ERRORS:
        return True

    # LLM API error codes
    if hasattr(error, "code"):
        return error.code in RETRYABLE_ERRORS

    return False

Non-Retryable Errors

NON_RETRYABLE_ERRORS = {
    # HTTP Status Codes
    400: "Bad Request",
    401: "Unauthorized",
    403: "Forbidden",
    404: "Not Found",
    405: "Method Not Allowed",
    422: "Unprocessable Entity",

    # LLM API Errors
    "invalid_api_key",
    "invalid_request_error",
    "content_policy_violation",
    "model_not_found",
    "insufficient_quota",
}

Implementation Patterns

Basic Retry Decorator

import asyncio
import random
from functools import wraps
from typing import TypeVar, Callable, Awaitable, Set, Type

T = TypeVar("T")

def retry(
    max_attempts: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    exponential_base: float = 2.0,
    jitter: bool = True,
    retryable_exceptions: Set[Type[Exception]] = None,
):
    """Async retry decorator with exponential backoff."""

    retryable = retryable_exceptions or {Exception}

    def decorator(fn: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]:
        @wraps(fn)
        async def wrapper(*args, **kwargs) -> T:
            last_exception = None

            for attempt in range(1, max_attempts + 1):
                try:
                    return await fn(*args, **kwargs)
                except tuple(retryable) as e:
                    last_exception = e

                    if attempt == max_attempts:
                        raise

                    # Calculate delay
                    delay = min(base_delay * (exponential_base ** (attempt - 1)), max_delay)

                    # Apply jitter
                    if jitter:
                        delay = random.uniform(0, delay)

                    logger.warning(
                        f"Retry {attempt}/{max_attempts} after {delay:.2f}s",
                        error=str(e),
                        function=fn.__name__,
                    )

                    await asyncio.sleep(delay)

            raise last_exception

        return wrapper
    return decorator

Retry with Modification

async def retry_with_truncation(
    fn: Callable[[str], Awaitable[T]],
    content: str,
    max_attempts: int = 3,
) -> T:
    """Retry LLM call, truncating content on context_length_exceeded."""

    for attempt in range(1, max_attempts + 1):
        try:
            return await fn(content)
        except ContextLengthExceededError:
            if attempt == max_attempts:
                raise

            # Truncate content by 25% each retry
            truncate_to = int(len(content) * 0.75)
            content = content[:truncate_to]

            logger.warning(
                f"Truncating content to {truncate_to} chars",
                attempt=attempt,
            )

Retry Budget

class RetryBudget:
    """Limits total retries across all calls to prevent retry storms."""

    def __init__(
        self,
        budget_per_second: float = 10.0,
        min_retries_per_second: float = 1.0,
    ):
        self.budget = budget_per_second
        self.min_budget = min_retries_per_second
        self.last_update = time.time()

    def can_retry(self) -> bool:
        self._replenish()
        return self.budget >= 1.0

    def use_retry(self):
        if self.budget >= 1.0:
            self.budget -= 1.0

    def _replenish(self):
        now = time.time()
        elapsed = now - self.last_update
        self.budget = min(
            self.budget + elapsed * self.min_budget,
            10.0  # Max budget
        )
        self.last_update = now

Best Practices (2026)

1. Set Appropriate Limits

# BAD: Too many retries
@retry(max_attempts=10, base_delay=0.1)  # 10 retries in ~3s = hammering

# GOOD: Reasonable limits
@retry(max_attempts=3, base_delay=1.0, max_delay=30.0)

2. Always Use Jitter

# BAD: No jitter (thundering herd)
delay = base * 2 ** attempt  # All clients retry at same time

# GOOD: Full jitter
delay = random.uniform(0, base * 2 ** attempt)  # Spread retries

3. Different Strategies per Operation

# Fast-fail for user-facing
@retry(max_attempts=2, base_delay=0.5, max_delay=2.0)
async def get_user_data():
    ...

# More patient for background jobs
@retry(max_attempts=5, base_delay=2.0, max_delay=60.0)
async def sync_data():
    ...

4. Log All Retries

async def retry_with_logging(fn, *args, **kwargs):
    for attempt in range(1, max_attempts + 1):
        try:
            return await fn(*args, **kwargs)
        except RetryableError as e:
            logger.warning(
                "Retry attempt",
                attempt=attempt,
                max_attempts=max_attempts,
                error_type=type(e).__name__,
                error_message=str(e),
                function=fn.__name__,
                # Include trace ID for correlation
                trace_id=get_current_trace_id(),
            )
            await asyncio.sleep(calculate_delay(attempt))

5. Combine with Circuit Breaker

# Retry INSIDE circuit breaker
# Circuit only counts final failures after retries exhausted

@circuit_breaker(failure_threshold=5)
@retry(max_attempts=3)
async def call_external_api():
    ...

# NOT the other way around:
# @retry  # Would retry when circuit is open!
# @circuit_breaker

Anti-Patterns

1. Retrying Non-Retryable Errors

# BAD: Retry everything
@retry(max_attempts=5, retryable_exceptions={Exception})
async def call_api():
    ...  # Will retry 401 Unauthorized 5 times!

# GOOD: Specific exceptions
@retry(
    max_attempts=3,
    retryable_exceptions={
        ConnectionError,
        TimeoutError,
        RateLimitError,
    }
)

2. No Backoff

# BAD: Fixed delay
for attempt in range(5):
    try:
        return await call()
    except Exception:
        await asyncio.sleep(1)  # Same delay every time

# GOOD: Exponential backoff
for attempt in range(5):
    try:
        return await call()
    except Exception:
        await asyncio.sleep(2 ** attempt)  # 1, 2, 4, 8, 16

3. Infinite Retries

# BAD: Never gives up
while True:
    try:
        return await call()
    except Exception:
        await asyncio.sleep(1)

# GOOD: Bounded retries
for attempt in range(max_attempts):
    ...
raise MaxRetriesExceeded()

LLM-Specific Retry Strategies

Rate Limit Handling

async def call_llm_with_rate_limit_handling(prompt: str) -> str:
    for attempt in range(3):
        try:
            return await llm.complete(prompt)
        except RateLimitError as e:
            # Use retry-after header if provided
            retry_after = e.headers.get("retry-after", 60)
            logger.warning(f"Rate limited, waiting {retry_after}s")
            await asyncio.sleep(int(retry_after))

    raise MaxRetriesExceeded("Rate limit persists after retries")

Context Length Handling

async def call_with_context_management(prompt: str, max_tokens: int = 4096) -> str:
    for attempt in range(3):
        try:
            return await llm.complete(prompt, max_tokens=max_tokens)
        except ContextLengthExceededError:
            # Reduce by 25% each attempt
            prompt = truncate_prompt(prompt, ratio=0.75 ** attempt)
            logger.warning(f"Truncated prompt to {len(prompt)} chars")

    raise ContextLengthExceededError("Cannot fit in context after truncation")

Monitoring

# Retry rate
rate(retries_total[5m])

# Retry success rate (retries that eventually succeed)
sum(rate(retry_success_total[5m])) / sum(rate(retries_total[5m]))

# Average attempts before success
histogram_quantile(0.95, retry_attempts_bucket)

# Retry budget utilization
retry_budget_used / retry_budget_total