Qa Resilience

QA Resilience (Jan 2026) - Failure Mode Testing & Production Hardening

This skill provides execution-ready patterns for building resilient, fault-tolerant systems that handle failures gracefully, and for validating those behaviors with tests.

Core sources are curated in data/sources.json.

Common Requests

Use this skill when a user requests:

• Circuit breaker implementation
• Retry strategies and exponential backoff
• Bulkhead pattern for resource isolation
• Backpressure, load shedding, and overload protection
• Timeout policies for external dependencies
• Graceful degradation and fallback mechanisms
• Health check design (liveness vs readiness)
• Error handling best practices
• Chaos engineering setup
• Game days / DR / failover testing (with guardrails)
• Production hardening strategies
• Fault injection testing

When NOT to use this skill:

• Simple CRUD apps with no external dependencies — use basic error handling
• Single database, no network calls — standard connection pooling sufficient
• Pure batch jobs with manual retry — scheduled job frameworks handle this
• Frontend-only validation — see software-frontend instead

Quick Start (Default Workflow)

If key context is missing, ask for: critical user journeys, dependency inventory (including third parties), SLO/SLI targets, current timeout/retry/circuit-breaker settings, idempotency/dedup strategy, and where fault injection is allowed (local/staging/prod).

1. Define scope: critical user journeys, top N dependencies, and SLOs/SLIs (latency, errors, saturation). 2. Build a dependency contract per dependency: timeout budget, retry policy (bounded + jitter), idempotency/dedup expectations, circuit breaker thresholds, and fallback/degraded behavior. 3. Choose a test harness: deterministic fault injection first (mocks/fakes, fault proxy, service mesh faults), then staged chaos experiments, then game day/DR drills if applicable. 4. Define pass/fail signals: error budget burn, p95/p99 budgets, fallback rates, queue backlog, circuit breaker state changes, and recovery time. 5. Produce artifacts (use templates): Resilience Test Plan Template, Fault Injection Playbook, Resilience Runbook Template.

Core QA (Default)

Failure Mode Testing (What to Validate)

• Timeouts: every network call and DB query has a bounded timeout; validate timeout budgets across chained calls and deadline/cancellation propagation.
• Retries: bounded retries with backoff + jitter; validate idempotency/dedup and retry storm safeguards (caps, budgets, and per-try timeouts).
• Dependency failure: partial outage, slow downstream, rate limiting, DNS failures, auth failures, and corrupted/invalid responses.
• Overload/saturation: connection pool exhaustion, queue backlog, thread pool starvation, and rate limiting; validate backpressure and load shedding.
• Degraded-mode UX: what the user sees/gets when dependencies fail (cached/stale/partial responses) and what consistency guarantees apply.
• Health checks: validate liveness/readiness/startup probe behavior (Kubernetes probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/).

Right-Sized Chaos Engineering (Safe by Construction)

• Define steady state and hypothesis (Principles of Chaos Engineering: https://principlesofchaos.org/).
• Start in non-prod; in prod, use minimal blast radius, timeboxed runs, and explicit abort criteria.
• REQUIRED: rollback plan, owners, and observability signals before running experiments.
• REQUIRED (prod): change window + on-call aware, error budget healthy, and an explicit stop condition based on customer impact signals.

Load/Perf + Production Guardrails

• Load tests validate capacity and tail latency; resilience tests validate behavior under failure.
• Guardrails:
• Run heavy resilience/perf suites on schedule (nightly) and on canary deploys, not on every PR.
• Gate releases on regression budgets (p99 latency, error rate, saturation) rather than on raw CPU/memory.

Flake Control for Resilience Tests

• Chaos/fault injection can look "flaky" if the experiment is not deterministic.
• Stabilize the experiment first: fixed blast radius, controlled fault parameters, deterministic duration, strong observability.

Debugging Ergonomics

• Every resilience test run should capture: experiment parameters, target scope, timestamps, and trace/log links for failures.
• Prefer tracing/metrics to confirm the failure is the expected one (not collateral damage).

Do / Avoid

Do:

• Test degraded mode explicitly; document expected UX and API responses.
• Validate retries/timeouts in integration tests with fault injection.

Avoid:

• Unbounded retries and missing timeouts (amplifies incidents).
• "Happy-path only" testing that ignores downstream failure classes.

Quick Reference

Decision Tree: Resilience Pattern Selection

Failure scenario: [System Dependency Type]
    ├─ External API/Service?
    │   ├─ Transient errors? → Retry with exponential backoff + jitter
    │   ├─ Cascading failures? → Circuit breaker + fallback
    │   ├─ Rate limiting? → Retry with Retry-After header respect
    │   └─ Slow response? → Timeout + circuit breaker
    │
    ├─ Database Dependency?
    │   ├─ Connection pool exhaustion? → Bulkhead isolation + timeout
    │   ├─ Query timeout? → Statement timeout (5-10s)
    │   ├─ Replica lag? → Read from primary fallback
    │   └─ Connection failures? → Retry + circuit breaker
    │
    ├─ Overload/Saturation?
    │   ├─ Queue/pool growing? → Backpressure + bound queues + admission control
    │   ├─ Thundering herd? → Jitter + request coalescing + caching
    │   └─ Expensive paths? → Load shedding + feature flag degradation
    │
    ├─ Non-Critical Feature?
    │   ├─ ML recommendations? → Feature flag + default values fallback
    │   ├─ Search service? → Cached results or basic SQL fallback
    │   ├─ Email/notifications? → Log error, don't block main flow
    │   └─ Analytics? → Fire-and-forget, circuit breaker for protection
    │
    ├─ Kubernetes/Orchestration?
    │   ├─ Service discovery? → Liveness + readiness + startup probes
    │   ├─ Slow startup? → Startup probe (failureThreshold: 30)
    │   ├─ Load balancing? → Readiness probe (check dependencies)
    │   └─ Auto-restart? → Liveness probe (simple check)
    │
    └─ Testing Resilience?
        ├─ Pre-production? → Chaos Toolkit experiments
        ├─ Production (low risk)? → Feature flags + canary deployments
        ├─ Scheduled testing? → Game days (quarterly)
        └─ Continuous chaos? → Low-blast-radius fault injection with strong guardrails

Navigation: Core Resilience Patterns

• [Circuit Breaker Patterns](references/circuit-breaker-patterns.md) - Prevent cascading failures
• Classic circuit breaker implementation (Node.js, Python)
• Tuning, alerting, and fallback strategies

• [Retry Patterns](references/retry-patterns.md) - Handle transient failures
• Exponential backoff with jitter
• Retry decision table (which errors to retry)
• Idempotency patterns and Retry-After headers

• [Bulkhead Isolation](references/bulkhead-isolation.md) - Resource compartmentalization
• Semaphore pattern for thread/connection pools
• Database connection pooling strategies
• Queue-based bulkheads with load shedding

• [Timeout Policies](references/timeout-policies.md) - Prevent resource exhaustion
• Connection, request, and idle timeouts
• Database query timeouts (PostgreSQL, MySQL)
• Nested timeout budgets for chained operations

• [Graceful Degradation](references/graceful-degradation.md) - Maintain partial functionality
• Cached fallback strategies
• Default values and feature toggles
• Partial responses with Promise.allSettled

• [Health Check Patterns](references/health-check-patterns.md) - Service availability monitoring
• Liveness, readiness, and startup probes
• Kubernetes probe configuration
• Shallow vs deep health checks

• [Load Shedding & Backpressure](references/load-shedding-backpressure.md) - Overload protection patterns
• Admission control and queue-based shedding
• Backpressure propagation across services
• Priority-based request handling

• [Cascading Failure Prevention](references/cascading-failure-prevention.md) - Multi-layer containment
• Failure propagation analysis
• Dependency isolation strategies
• Blast radius limitation techniques

• [Disaster Recovery Testing](references/disaster-recovery-testing.md) - DR drill execution
• RTO/RPO verification
• Failover and failback procedures
• Game day planning and execution

Navigation: Operational Resources

• [Resilience Checklists](references/resilience-checklists.md) - Production hardening checklists
• Dependency resilience
• Health and readiness probes
• Observability for resilience
• Failure testing

• [Chaos Engineering Guide](references/chaos-engineering-guide.md) - Safe reliability experiments
• Planning chaos experiments
• Common failure injection scenarios
• Execution steps and debrief checklist

Navigation: Templates

• [Resilience Runbook Template](assets/runbooks/resilience-runbook-template.md) - Service hardening profile
• Dependencies and SLOs
• Fallback strategies
• Rollback procedures

• [Fault Injection Playbook](assets/testing/fault-injection-playbook.md) - Chaos testing script
• Success signals
• Rollback criteria
• Post-experiment debrief

• [Resilience Test Plan Template](assets/testing/template-resilience-test-plan.md) - Failure mode test plan (timeouts/retries/degraded mode)
• Scope and dependencies
• Fault matrix and expected behavior
• Observability signals and pass/fail criteria

Quick Decision Matrix

Anti-Patterns to Avoid

• No timeouts - Infinite waits exhaust resources
• Infinite retries - Amplifies problems (thundering herd)
• Retries without idempotency - Duplicate side effects and data corruption
• No circuit breakers - Cascading failures
• Tight coupling - One failure breaks everything
• Silent failures - No observability into degraded state
• No bulkheads - Shared thread pools exhaust all resources
• Failover never tested - DR plan fails during a real incident
• Testing only happy path - Production reveals failures

Optional: AI / Automation

Do:

• Use AI to propose failure-mode scenarios from an explicit risk register; keep only scenarios that map to known dependencies and business journeys.
• Use AI to summarize experiment results (metrics deltas, error clusters) and draft postmortem timelines; verify with telemetry.

Avoid:

• "Scenario generation" without a risk map (creates noise and wasted load).
• Letting AI relax timeouts/retries or remove guardrails.

Related Skills

• ../ops-devops-platform/SKILL.md — Incident response, SLOs, and platform runbooks
• ../software-backend/SKILL.md — API error handling, retries, and database reliability patterns
• ../software-architecture-design/SKILL.md — System decomposition and dependency design for reliability
• ../qa-testing-strategy/SKILL.md — Regression, load, and fault-injection testing strategies
• ../software-security-appsec/SKILL.md — Security failure modes and guardrails
• ../qa-observability/SKILL.md — Metrics, tracing, logging, and performance monitoring
• ../qa-debugging/SKILL.md — Production debugging and incident investigation
• ../data-sql-optimization/SKILL.md — Database resilience, connection pooling, and query timeouts
• ../dev-api-design/SKILL.md — API design patterns including error handling and retry semantics

Usage Notes

Pattern Selection:

• Start with circuit breakers for external dependencies
• Add retries for transient failures (network, rate limits)
• Use bulkheads to prevent resource exhaustion
• Combine patterns for defense-in-depth

Observability:

• Track circuit breaker state changes
• Monitor retry attempts and success rates
• Alert on degraded mode duration
• Measure recovery time after failures

Testing:

• Start chaos experiments in non-production
• Define hypothesis before failure injection
• Set blast radius limits and auto-revert
• Document learnings and action items

Success criteria: systems gracefully handle failures, recover automatically, maintain partial functionality during outages, and fail fast to prevent cascading failures. Resilience is tested proactively through fault injection and game days (with guardrails).

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Resilience Runbook Template

Fill this template when documenting a service's resilience strategy.

• Service:
• Owner / Pager:
• SLOs: Availability %, latency targets, error budget burn alerts
• Dependencies (ranked): Service → dependency, timeout, retry policy, circuit breaker config, fallback
• Failure Modes: What can go wrong? (timeout, error spike, thundering herd, saturation)
• Protection: Bulkheads, backpressure, rate limits, load shedding rules
• Observability: Key dashboards, alerts, traces, logging fields (correlation IDs)
• Graceful Degradation: What to show users when degraded; data freshness guarantees
• Failover/DR: Region/AZ strategy, replication lag budget, RPO/RTO, failover test cadence
• Run Steps: Verification checklist before/after change; rollback triggers and commands
• Validation: Chaos experiment IDs, last test date, next scheduled game day

Fault Injection Playbook

Use this to plan and execute a single fault injection test.

• Scenario: e.g., dependency latency +2s, 500 errors, network drop, AZ outage
• Hypothesis: What should happen? (timeouts respected, circuit opens, fallback serves stale data)
• Blast Radius: Scope (service, namespace, AZ), guardrails, auto-revert time
• Pre-Checks: SLO burn rate, alerts healthy, dashboards bookmarked, rollback commands verified
• Execution Steps: Ordered commands/scripts with expected signals per step
• Success Signals: Error budget burn stays <X, latency <Y, no 5xx beyond Z%, no queue overload
• Rollback Criteria: Thresholds that trigger immediate stop; rollback steps and owner
• Post-Test Actions: Findings, gaps, new alerts, runbook updates, follow-up owners + due dates

Resilience Test Plan Template (Timeouts, Retries, Degraded Mode)

Use this plan to validate how the system behaves when dependencies fail or degrade.

Core

Scope

• Service/system under test: _________________________________
• Environments: staging / pre-prod / prod (if approved)
• Owners: engineering / QA / SRE: ____________________________

Dependencies (Inventory)

List critical dependencies and their failure modes.

Steady State (What “Healthy” Means)

• SLIs and targets (SLOs): availability, error rate, p95/p99 latency
• Baseline metrics window: last ____ days

Fault Matrix (Test Cases)

Execution Plan (Right-Sized Chaos)

• Hypothesis and steady state documented (Principles of Chaos Engineering: https://principlesofchaos.org/)
• Blast radius controls:
• Target scope (service/region/tenant): ______________________
• Timebox: _________________________________________________
• Abort criteria: ___________________________________________
• Rollback plan: _____________________________________________

Observability Requirements

• Correlation IDs captured on failure (request ID / trace ID)
• Dashboards and alerts ready (SLO burn, error rate, tail latency)
• Runbook link: ______________________________________________

CI Economics and Scheduling

• PR gate: smoke resilience checks only (mocked fault injection)
• Nightly/release: full fault matrix and load/stress scenarios

Flake Control

• Deterministic experiment parameters (fixed duration, fixed blast radius)
• Clear “expected failure” vs “unexpected collateral damage” signals

Optional: AI / Automation

Do:

• Use AI to propose scenario candidates from the dependency inventory; keep only scenarios mapped to explicit risks.
• Use AI to summarize experiment results and produce a draft postmortem timeline; verify with telemetry.

Avoid:

• Generating scenarios without a risk map or without observability signals.

Bulkhead Isolation Pattern

Preventing resource exhaustion from cascading failures using compartmentalization.

---

Pattern: Bulkhead Isolation

Use when: Preventing resource exhaustion from cascading failures.

Bulkhead Pattern:

Thread Pool A (Payment API) - 10 threads
Thread Pool B (Email API) - 5 threads
Thread Pool C (Analytics API) - 3 threads

If Analytics API hangs, only 3 threads blocked.
Payment and Email APIs remain unaffected.

Node.js Implementation (Semaphore Pattern):

class Semaphore {
  constructor(maxConcurrency) {
    this.maxConcurrency = maxConcurrency;
    this.currentConcurrency = 0;
    this.queue = [];
  }

  async acquire() {
    if (this.currentConcurrency < this.maxConcurrency) {
      this.currentConcurrency++;
      return;
    }

    // Wait in queue
    return new Promise((resolve) => {
      this.queue.push(resolve);
    });
  }

  release() {
    this.currentConcurrency--;
    if (this.queue.length > 0) {
      const resolve = this.queue.shift();
      this.currentConcurrency++;
      resolve();
    }
  }

  async runExclusive(fn) {
    await this.acquire();
    try {
      return await fn();
    } finally {
      this.release();
    }
  }
}

// Create bulkheads for different services
const paymentSemaphore = new Semaphore(10);
const emailSemaphore = new Semaphore(5);
const analyticsSemaphore = new Semaphore(3);

// Usage
async function processPayment(order) {
  return paymentSemaphore.runExclusive(async () => {
    return await paymentAPI.charge(order);
  });
}

async function sendEmail(recipient, message) {
  return emailSemaphore.runExclusive(async () => {
    return await emailAPI.send(recipient, message);
  });
}

---

Thread Pool Sizing

Formulas:

CPU-bound tasks: threads = CPU cores
I/O-bound tasks: threads = 2 * CPU cores (or higher)
Mixed workload: threads = CPU cores + (wait time / service time)

Example:
Service time: 10ms
Wait time: 90ms (network I/O)
CPU cores: 4

Optimal threads = 4 + (90 / 10) = 4 + 9 = 13 threads

Sizing by Workload Type:

---

Database Connection Pooling

PostgreSQL with node-postgres:

const { Pool } = require('pg');

// Separate pools for different workloads
const readPool = new Pool({
  host: 'replica.db.example.com',
  max: 20,           // Max connections
  min: 2,            // Min idle connections
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

const writePool = new Pool({
  host: 'primary.db.example.com',
  max: 10,           // Smaller pool for writes
  min: 2,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

// Usage
async function getUser(userId) {
  const client = await readPool.connect();
  try {
    const result = await client.query('SELECT * FROM users WHERE id = $1', [userId]);
    return result.rows[0];
  } finally {
    client.release();
  }
}

async function createUser(userData) {
  const client = await writePool.connect();
  try {
    await client.query('BEGIN');
    const result = await client.query(
      'INSERT INTO users (name, email) VALUES ($1, $2) RETURNING *',
      [userData.name, userData.email]
    );
    await client.query('COMMIT');
    return result.rows[0];
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

Python with SQLAlchemy:

from sqlalchemy import create_engine
from sqlalchemy.pool import QueuePool

# Separate engines for read and write
read_engine = create_engine(
    'postgresql://user:pass@replica.db.example.com/dbname',
    poolclass=QueuePool,
    pool_size=20,           # Max connections
    max_overflow=10,        # Additional connections during spikes
    pool_timeout=30,        # Wait timeout
    pool_pre_ping=True,     # Verify connections before use
)

write_engine = create_engine(
    'postgresql://user:pass@primary.db.example.com/dbname',
    poolclass=QueuePool,
    pool_size=10,
    max_overflow=5,
    pool_timeout=30,
    pool_pre_ping=True,
)

---

Queue-Based Bulkheads

Use when: Need to throttle work and prevent overload.

class WorkQueue {
  constructor(concurrency, queueSize) {
    this.concurrency = concurrency;
    this.queueSize = queueSize;
    this.running = 0;
    this.queue = [];
  }

  async enqueue(fn) {
    // Reject if queue full
    if (this.queue.length >= this.queueSize) {
      throw new Error('Queue full - load shedding');
    }

    // Queue the work
    return new Promise((resolve, reject) => {
      this.queue.push({ fn, resolve, reject });
      this.processQueue();
    });
  }

  async processQueue() {
    if (this.running >= this.concurrency || this.queue.length === 0) {
      return;
    }

    this.running++;
    const { fn, resolve, reject } = this.queue.shift();

    try {
      const result = await fn();
      resolve(result);
    } catch (error) {
      reject(error);
    } finally {
      this.running--;
      this.processQueue();
    }
  }
}

// Create separate queues for different services
const paymentQueue = new WorkQueue(10, 100);  // 10 concurrent, 100 queue size
const emailQueue = new WorkQueue(5, 500);     // 5 concurrent, 500 queue size

// Usage
async function processPayment(order) {
  try {
    return await paymentQueue.enqueue(async () => {
      return await paymentAPI.charge(order);
    });
  } catch (error) {
    if (error.message === 'Queue full - load shedding') {
      // Return 503 Service Unavailable
      throw new Error('Payment system overloaded, try again later');
    }
    throw error;
  }
}

---

Monitoring Bulkheads

Metrics to Track:

// Track queue depth and active workers
setInterval(() => {
  metrics.gauge('bulkhead.payment.active', paymentSemaphore.currentConcurrency);
  metrics.gauge('bulkhead.payment.queued', paymentSemaphore.queue.length);
  metrics.gauge('bulkhead.email.active', emailSemaphore.currentConcurrency);
  metrics.gauge('bulkhead.email.queued', emailSemaphore.queue.length);
}, 5000);

// Alert when pools saturate
if (paymentSemaphore.currentConcurrency >= paymentSemaphore.maxConcurrency * 0.8) {
  alerts.warn('Payment pool at 80% capacity');
}

if (emailSemaphore.queue.length > 100) {
  alerts.warn('Email queue backing up');
}

Dashboard Queries (Prometheus):

# Pool utilization
bulkhead_active_workers / bulkhead_max_workers

# Queue depth
bulkhead_queue_size

# Wait time in queue
rate(bulkhead_queue_wait_seconds_sum[5m]) / rate(bulkhead_queue_wait_seconds_count[5m])

---

Partition-Scoped Failure Isolation

Use when: A single consumer or worker handles multiple logical partitions (e.g., Kafka partitions, sharded queues) and a failure in one partition must not affect others.

Partition-scoped isolation is narrower than service-level bulkheads — it applies within a single consumer handling multiple partitions:

• When a retry/DLQ publish fails for one partition, pause only that partition rather than sleeping the entire consumer loop or killing the consumer task.
• Failure-routing failures should produce visible host-level faulting (health check degradation, metrics), not leave a dead background task behind.
• A silent failure in one partition's retry path should not prevent other partitions from making progress.
• This pattern complements service-level bulkheads: bulkheads isolate between services; partition-scoped isolation operates within a single service's internal processing.

---

Checklist

• [ ] Separate thread pools/queues for different services
• [ ] Pool sizes based on workload characteristics
• [ ] Timeouts prevent thread starvation
• [ ] Queue depth limits prevent memory exhaustion
• [ ] Bulkhead metrics monitored (active threads, queue depth)
• [ ] Alerts when pools saturate
• [ ] Load shedding when queue fills
• [ ] Database connection pools separated (read/write)
• [ ] Partition-scoped failures isolated (no cross-partition impact within a consumer)

---

Related Resources

• timeout-policies.md - Prevent thread starvation with timeouts
• circuit-breaker-patterns.md - Protect bulkheads with circuit breakers
• resilience-checklists.md - Comprehensive bulkhead hardening

Cascading Failure Prevention

Techniques for containing and preventing cascading failures in distributed systems.

Contents

• Cascade Anatomy
• Blast Radius Containment
• Dependency Isolation Patterns
• Circuit Breaker and Bulkhead Combination
• Retry Storm Prevention
• Connection and Thread Pool Exhaustion
• Architectural Isolation
• Testing for Cascading Failures
• Real-World Case Studies
• Prevention Checklist
• Related Resources

---

Cascade Anatomy

A cascading failure follows three phases: trigger, propagation, and amplification.

TRIGGER                    PROPAGATION                   AMPLIFICATION
─────────                  ───────────                   ─────────────
Single component fails     Dependent services slow       Retry storms multiply
  ↓                          ↓                             ↓
Database goes read-only    API gateway queues fill       Clients retry 3x each
  ↓                          ↓                             ↓
Timeouts start             Thread pools exhaust          Load triples
  ↓                          ↓                             ↓
Error rate spikes          Memory pressure rises         Health checks fail
                             ↓                             ↓
                           Cascading service failures    Full system outage

Common Triggers

---

Blast Radius Containment

Limit how far a failure can spread by isolating failure domains.

Failure Domain Hierarchy

Level 1: Process   →  Single container/pod
Level 2: Service   →  All replicas of one service
Level 3: Cell      →  Group of services sharing infrastructure
Level 4: Region    →  Entire cloud region
Level 5: Global    →  Control plane, DNS, CDN

Containment Strategies by Level

# Example: Feature flag kill switch for blast radius control
class FeatureFlags:
    """Disable features under cascading failure conditions."""

    def __init__(self, flag_service):
        self.flags = flag_service

    def should_call_recommendation_engine(self, user_id: str) -> bool:
        # Kill switch: disable non-critical dependency entirely
        if not self.flags.is_enabled("recommendations.enabled"):
            return False

        # Percentage rollout: limit blast radius during incidents
        if self.flags.get_percentage("recommendations.traffic") < 100:
            return hash(user_id) % 100 < self.flags.get_percentage(
                "recommendations.traffic"
            )

        return True

    def get_recommendations(self, user_id: str) -> list:
        if not self.should_call_recommendation_engine(user_id):
            return self.get_cached_recommendations(user_id)

        try:
            return self.recommendation_client.fetch(user_id)
        except Exception:
            return self.get_cached_recommendations(user_id)

---

Dependency Isolation Patterns

Critical vs Non-Critical Dependencies

Classify every external dependency and apply different failure policies.

// Node.js: Dependency classification with failure policies
class DependencyManager {
  constructor() {
    this.dependencies = new Map();
  }

  register(name, { type, client, fallback, circuitBreaker }) {
    this.dependencies.set(name, { type, client, fallback, circuitBreaker });
  }

  async call(name, method, ...args) {
    const dep = this.dependencies.get(name);
    if (!dep) throw new Error(`Unknown dependency: ${name}`);

    try {
      if (dep.circuitBreaker) {
        return await dep.circuitBreaker.fire(() => dep.client[method](...args));
      }
      return await dep.client[method](...args);
    } catch (error) {
      switch (dep.type) {
        case 'critical':
          throw error; // propagate -- caller must handle
        case 'degradable':
          console.warn(`${name} degraded: ${error.message}`);
          return dep.fallback ? dep.fallback(...args) : null;
        case 'optional':
          console.warn(`${name} unavailable: ${error.message}`);
          return null;
        case 'async':
          await this.enqueueForRetry(name, method, args);
          return { queued: true };
        default:
          throw error;
      }
    }
  }
}

---

Circuit Breaker and Bulkhead Combination

Use circuit breakers to detect failure and bulkheads to contain resource consumption. Together they prevent both failure propagation and resource exhaustion.

import asyncio
from dataclasses import dataclass, field
from enum import Enum
import time

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class ProtectedDependency:
    """Combines circuit breaker + bulkhead for a single dependency."""
    name: str
    max_concurrent: int = 10
    failure_threshold: int = 5
    reset_timeout: float = 30.0

    # Bulkhead state
    semaphore: asyncio.Semaphore = field(init=False)
    # Circuit breaker state
    state: CircuitState = CircuitState.CLOSED
    failure_count: int = 0
    last_failure_time: float = 0.0

    def __post_init__(self):
        self.semaphore = asyncio.Semaphore(self.max_concurrent)

    async def call(self, func, *args, **kwargs):
        # Circuit breaker check
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.reset_timeout:
                self.state = CircuitState.HALF_OPEN
            else:
                raise CircuitOpenError(f"{self.name} circuit is open")

        # Bulkhead: non-blocking acquire
        if self.semaphore._value == 0:
            raise BulkheadFullError(
                f"{self.name} bulkhead full ({self.max_concurrent} slots)"
            )

        async with self.semaphore:
            try:
                result = await func(*args, **kwargs)
                self._on_success()
                return result
            except Exception as e:
                self._on_failure()
                raise

    def _on_success(self):
        self.failure_count = 0
        self.state = CircuitState.CLOSED

    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN

Configuration Matrix:

---

Retry Storm Prevention

Retries amplify failures. Uncoordinated retries across multiple layers can multiply traffic 3x-27x.

The Multiplication Problem

Client retries 3x → Gateway retries 3x → Service retries 3x
Total attempts per original request: 3 × 3 × 3 = 27

If 1000 requests/sec normally:
During outage with naive retries: up to 27,000 requests/sec

Mitigation Strategies

// 1. Retry budget: cap total retries at a percentage of successful requests
class RetryBudget {
  constructor({ ratio = 0.1, minRetries = 10, windowMs = 10_000 }) {
    this.ratio = ratio;
    this.minRetries = minRetries;
    this.windowMs = windowMs;
    this.requests = [];
    this.retries = [];
  }

  canRetry() {
    const now = Date.now();
    this.requests = this.requests.filter(t => now - t < this.windowMs);
    this.retries = this.retries.filter(t => now - t < this.windowMs);

    const budget = Math.max(
      this.minRetries,
      Math.floor(this.requests.length * this.ratio)
    );

    return this.retries.length < budget;
  }

  recordRequest() { this.requests.push(Date.now()); }
  recordRetry()   { this.retries.push(Date.now()); }
}

// 2. Exponential backoff with full jitter
function backoffWithJitter(attempt, baseMs = 100, maxMs = 30_000) {
  const exponential = Math.min(maxMs, baseMs * Math.pow(2, attempt));
  return Math.random() * exponential; // full jitter
}

Retry Rules of Thumb:

• [ ] Retry only at one layer (closest to the caller, not at every hop)
• [ ] Use exponential backoff with full jitter (not fixed intervals)
• [ ] Implement retry budgets (max 10% of successful request volume)
• [ ] Never retry non-idempotent operations without explicit safeguards
• [ ] Add circuit breakers to stop retries when a dependency is down

---

Connection and Thread Pool Exhaustion

Pool exhaustion is the most common cascade propagation mechanism.

Connection Pool Sizing

# SQLAlchemy: properly sized connection pool
from sqlalchemy import create_engine

engine = create_engine(
    "postgresql://user:pass@db:5432/app",
    pool_size=20,          # steady-state connections
    max_overflow=10,       # burst capacity
    pool_timeout=5,        # wait at most 5s for a connection
    pool_recycle=1800,     # recycle connections every 30 min
    pool_pre_ping=True,    # verify connection before use
)

Thread Pool Isolation (Java/Kotlin)

import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit

// Separate thread pools per dependency
val paymentExecutor = Executors.newFixedThreadPool(20).also {
    it as java.util.concurrent.ThreadPoolExecutor
    it.setRejectedExecutionHandler { _, _ ->
        throw ServiceUnavailableException("Payment pool exhausted")
    }
}

val analyticsExecutor = Executors.newFixedThreadPool(5)

// If analytics pool exhausts, payments are unaffected
fun processPayment(order: Order) = paymentExecutor.submit {
    paymentGateway.charge(order)
}

fun trackEvent(event: Event) = analyticsExecutor.submit {
    analyticsService.track(event)
}

---

Architectural Isolation

Shared-Nothing Architecture

Each service owns its data store and has no shared state with other services.

SHARED (risky)                    SHARED-NOTHING (resilient)
──────────────                    ─────────────────────────
Service A ──┐                     Service A → DB-A
             ├── Shared DB        Service B → DB-B
Service B ──┘                     Service C → DB-C
             │
Service C ──┘                     Communication via async events only

Cell-Based Architecture

Partition users into isolated cells. A failure in cell 1 cannot affect cell 2.

Cell 1 (users A-M)          Cell 2 (users N-Z)
┌────────────────┐           ┌────────────────┐
│ API Gateway    │           │ API Gateway    │
│ App Servers    │           │ App Servers    │
│ Database       │           │ Database       │
│ Cache          │           │ Cache          │
└────────────────┘           └────────────────┘

Router (stateless) directs user to correct cell

Cell sizing guidance:

---

Testing for Cascading Failures

Chaos Engineering Scenarios

Design experiments that specifically test cascade propagation.

# Litmus chaos experiment: kill a critical dependency
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: cascade-test
spec:
  appinfo:
    appns: production
    applabel: app=payment-service
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: CHAOS_INTERVAL
              value: "10"
            - name: FORCE
              value: "true"

Cascade Test Matrix

Gameday Runbook Template

GAMEDAY: Cascading Failure Test
Date: ___________
Participants: ___________

PRE-GAME:
1. [ ] Notify on-call and stakeholders
2. [ ] Verify rollback procedures
3. [ ] Confirm monitoring dashboards are live
4. [ ] Set blast radius limits (which cells/regions)

EXPERIMENT:
1. [ ] Inject failure: ___________
2. [ ] Observe: error rates, latency, dependent services
3. [ ] Record time-to-detection: ___ seconds
4. [ ] Record time-to-mitigation: ___ seconds
5. [ ] Verify blast radius containment

POST-GAME:
1. [ ] Remove fault injection
2. [ ] Verify full recovery
3. [ ] Document findings
4. [ ] Create action items for gaps

---

Real-World Case Studies

AWS us-east-1 Kinesis Outage (2020)

• Trigger: A small capacity addition to Kinesis front-end fleet
• Propagation: New servers triggered a burst of thread creation in a shared component; threads exhausted OS limits
• Amplification: Cascaded to Cognito, CloudWatch, Lambda, and dozens of services dependent on Kinesis
• Duration: ~10 hours
• Lesson: Shared dependencies (Kinesis) become single points of failure. Services must handle Kinesis unavailability gracefully.

Netflix Cascading Failure Prevention

Netflix uses a layered defense:

1. Hystrix (now Resilience4j): Circuit breakers per dependency 2. Zuul: Adaptive load shedding at the edge 3. Cell architecture: Regional isolation with failover 4. Chaos Monkey / Chaos Kong: Regular failure injection including full region evacuation

Key Netflix principle: "Design for the failure case first. The happy path is the exception in distributed systems."

---

Prevention Checklist

• [ ] Every dependency classified as critical, degradable, optional, or async
• [ ] Circuit breakers on all synchronous external calls
• [ ] Bulkheads isolate resource pools per dependency
• [ ] Retry logic exists at only one layer with backoff and jitter
• [ ] Retry budgets cap retry volume at 10% of successful traffic
• [ ] Connection pools have bounded size and acquisition timeouts
• [ ] Thread pools are isolated per dependency (no shared pools)
• [ ] Timeouts are set on every network call (connect + read + total)
• [ ] Health checks distinguish between liveness and readiness
• [ ] Feature flags can disable non-critical dependencies instantly
• [ ] Chaos experiments test cascade scenarios quarterly
• [ ] Runbooks document cascade containment procedures
• [ ] Monitoring alerts on early cascade signals (pool utilization, error rate spikes)

---

Related Resources

• circuit-breaker-patterns.md -- Circuit breaker implementations
• bulkhead-isolation.md -- Bulkhead resource isolation
• load-shedding-backpressure.md -- Overload protection
• retry-patterns.md -- Retry strategies with backoff
• timeout-policies.md -- Timeout configuration
• chaos-engineering-guide.md -- Chaos experiment design
• health-check-patterns.md -- Liveness and readiness checks

Chaos Engineering Quick Guide

Use this guide to design safe, high-signal reliability experiments.

Planning

• Define objective and success criteria (SLO impact, user impact)
• Pick hypothesis tied to a specific failure mode (e.g., dependency timeout)
• Limit blast radius (namespace, AZ, service subset) and set auto-revert
• Notify stakeholders and set a clear stop condition

Common Experiments

• Kill or drain a pod/instance; verify rescheduling and traffic rebalancing
• Increase latency or error rate for a dependency; verify timeouts and fallbacks
• Drop network packets or DNS for a dependency; verify circuit breakers open
• Exhaust a resource (CPU, memory, file descriptors); verify autoscaling or load shedding
• Zonal outage simulation; verify multi-AZ failover and data replication

Execution Steps

1. Baseline metrics and SLO burn rate 2. Run the experiment with live monitoring 3. Observe user impact (error budgets, latency, conversion) 4. Roll back if thresholds hit; otherwise finish after the planned window 5. Record findings, gaps, and actions

Debrief Checklist

• [ ] New failure modes discovered and documented
• [ ] SLO/SLA coverage validated or updated
• [ ] Runbooks updated with verified steps
• [ ] Automation opportunities captured (alerts, auto-remediation)
• [ ] Follow-up owners and due dates assigned

Circuit Breaker Patterns

Production-ready circuit breaker implementations for preventing cascading failures.

---

Pattern: Circuit Breaker (Classic)

Use when: Preventing cascading failures from external dependencies.

Circuit Breaker States:

CLOSED → requests flow normally
   ↓ (failure threshold reached)
OPEN → requests fail immediately, no calls to dependency
   ↓ (timeout period expires)
HALF-OPEN → test request allowed
   ↓ (success) → CLOSED | (failure) → OPEN

Node.js Implementation (opossum library):

const CircuitBreaker = require('opossum');

// Wrap external service call
async function callExternalAPI(data) {
  const response = await fetch('https://api.example.com/data', {
    method: 'POST',
    body: JSON.stringify(data),
  });

  if (!response.ok) {
    throw new Error(`API error: ${response.status}`);
  }

  return response.json();
}

// Circuit breaker configuration
const options = {
  timeout: 3000,              // Timeout after 3s
  errorThresholdPercentage: 50, // Open after 50% errors
  resetTimeout: 30000,         // Try again after 30s
  volumeThreshold: 10,         // Min 10 requests before opening
};

const breaker = new CircuitBreaker(callExternalAPI, options);

// Fallback when circuit is open
breaker.fallback(() => {
  return { status: 'degraded', data: getCachedData() };
});

// Event listeners
breaker.on('open', () => console.log('Circuit opened'));
breaker.on('halfOpen', () => console.log('Circuit half-open'));
breaker.on('close', () => console.log('Circuit closed'));

// Usage
try {
  const result = await breaker.fire(requestData);
  console.log(result);
} catch (error) {
  console.error('Request failed:', error);
}

Python Implementation (pybreaker-style):

import pybreaker
import requests

# Configure circuit breaker
breaker = pybreaker.CircuitBreaker(
    fail_max=5,        # Open after 5 failures
    reset_timeout=30,  # Try again after 30s
)

@breaker
def call_external_api(data):
    response = requests.post(
        'https://api.example.com/data',
        json=data,
        timeout=3,
    )
    response.raise_for_status()
    return response.json()

# Fallback function
def fallback_handler():
    return {'status': 'degraded', 'data': get_cached_data()}

# Usage with fallback
try:
    result = call_external_api(request_data)
except Exception as e:
    if breaker.current_state == 'open':
        result = fallback_handler()
    else:
        raise

Checklist:

• Circuit breaker wraps external dependencies (not business logic).
• Failure thresholds and windows are tuned to traffic volume + error budget.
• Reset timeout is set (30-60s typical) and half-open probes are bounded.
• Fallback behavior is explicit (cache, stale reads, partial response, or hard fail).
• Circuit state changes are emitted as metrics/logs and alertable.

---

Pattern: Adaptive Circuit Breaker (2024-2025)

Use when: Static thresholds cause false positives during traffic spikes or miss real issues during low traffic.

Evolution from Static to Adaptive:

Traditional circuit breakers use fixed thresholds (e.g., "open after 50% errors"). Adaptive circuit breakers use real-time data to adjust behavior based on context:

• Traffic-aware thresholds: Different thresholds for peak hours vs low traffic
• Baseline learning: ML models learn normal failure rates over time
• Anomaly detection: Detect unusual patterns (sudden spike vs gradual degradation)
• Dynamic timeouts: Adjust based on observed latency percentiles

Conceptual Implementation (Node.js with adaptive logic):

class AdaptiveCircuitBreaker {
  constructor(service) {
    this.service = service;
    this.window = []; // Rolling window of results
    this.baselineFailureRate = 0.05; // Learned over time
    this.state = 'CLOSED';
    this.consecutiveSuccesses = 0;
  }

  async call(fn, context) {
    if (this.state === 'OPEN') {
      if (this.shouldAttemptReset()) {
        this.state = 'HALF-OPEN';
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await fn();
      this.recordSuccess();
      return result;
    } catch (error) {
      this.recordFailure();

      // Adaptive threshold based on current traffic volume
      const currentVolume = this.getRequestVolume();
      const adaptiveThreshold = this.calculateAdaptiveThreshold(currentVolume);

      if (this.currentFailureRate() > adaptiveThreshold) {
        this.state = 'OPEN';
        this.openedAt = Date.now();
      }

      throw error;
    }
  }

  calculateAdaptiveThreshold(volume) {
    // Higher threshold during low traffic (more tolerance)
    // Lower threshold during high traffic (strict)
    const baseThreshold = 0.5;

    if (volume < 10) {
      return 0.7; // 70% for low volume (avoid false positives)
    } else if (volume < 100) {
      return baseThreshold; // 50% for medium volume
    } else {
      // High volume: use anomaly detection
      const currentRate = this.currentFailureRate();
      const isAnomaly = currentRate > (this.baselineFailureRate * 3);
      return isAnomaly ? 0.3 : baseThreshold; // Stricter when anomaly detected
    }
  }

  updateBaseline() {
    // Update baseline from historical data (run periodically)
    const recentWindow = this.window.slice(-1000);
    const failures = recentWindow.filter(r => !r.success).length;
    this.baselineFailureRate = failures / recentWindow.length;
  }

  currentFailureRate() {
    const recent = this.window.slice(-100);
    if (recent.length === 0) return 0;
    const failures = recent.filter(r => !r.success).length;
    return failures / recent.length;
  }

  getRequestVolume() {
    const oneMinuteAgo = Date.now() - 60000;
    return this.window.filter(r => r.timestamp > oneMinuteAgo).length;
  }

  // ... other methods
}

Key Adaptive Strategies:

1. Volume-Based Thresholds:

• Low traffic (<10 req/min): Higher tolerance (70% threshold) to avoid false positives
• High traffic (>100 req/min): Anomaly detection vs learned baseline

2. Baseline Learning:

• Track historical failure rates (e.g., last 1000 requests)
• Update baseline during stable periods
• Compare current rate to 3x baseline for anomaly detection

3. Context-Aware Decisions:

• Time-of-day patterns (weekday vs weekend)
• Seasonal traffic variations
• Deployment events (expect higher errors post-deploy)

4. Dynamic Timeout Adjustment:

   function calculateAdaptiveTimeout(service) {
     const p95Latency = getP95Latency(service); // From metrics
     const baseTimeout = p95Latency * 2; // 2x P95 as baseline

     // Add buffer during traffic spikes
     const currentLoad = getCurrentLoad();
     const loadMultiplier = currentLoad > 0.8 ? 1.5 : 1.0;

     return baseTimeout * loadMultiplier;
   }

Checklist:

• [ ] Collect baseline metrics before enabling adaptive logic
• [ ] Track P95/P99 latency for dynamic timeouts
• [ ] Implement traffic volume detection (requests per minute)
• [ ] Define anomaly threshold (e.g., 3x baseline)
• [ ] Add observability for threshold changes
• [ ] Test with production-like traffic patterns
• [ ] Monitor false positive rate (circuits opened unnecessarily)
• [ ] Review and retrain baseline monthly

When NOT to Use Adaptive Patterns:

• Low-traffic services: Static thresholds simpler and sufficient
• Predictable failure modes: If failures are binary (works/doesn't work), no need for ML
• Early-stage systems: Need stable baseline data first (6-12 months)
• Regulatory constraints: Some industries require fixed, auditable thresholds

Production Example (Conceptual):

// Combine with observability
const adaptiveBreaker = new AdaptiveCircuitBreaker('payment-api');

// Update baseline nightly
cron.schedule('0 3 * * *', () => {
  adaptiveBreaker.updateBaseline();
  metrics.gauge('circuit_breaker.baseline_failure_rate',
    adaptiveBreaker.baselineFailureRate);
});

// Use in application
app.post('/checkout', async (req, res) => {
  try {
    const result = await adaptiveBreaker.call(async () => {
      return await paymentAPI.charge(req.body);
    });
    res.json(result);
  } catch (error) {
    // Circuit open or payment failed
    res.status(503).json({ error: 'Service temporarily unavailable' });
  }
});

Further Reading:

• https://martinfowler.com/bliki/CircuitBreaker.html
• https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker
• https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html

---

Related Resources

• retry-patterns.md - Combine circuit breakers with retry logic
• timeout-policies.md - Configure timeouts for circuit breakers
• resilience-checklists.md - Comprehensive dependency hardening

Disaster Recovery Testing

DR drill execution, RTO/RPO verification, and failover validation for production systems.

Contents

• DR Plan Components
• Types of DR Tests
• Database Failover Testing
• Backup Verification
• Multi-Region Failover
• DNS Failover
• Stateful Service Recovery
• DR Test Scheduling
• Runbook Validation
• Compliance Requirements
• Post-Drill Report Template
• Related Resources

---

DR Plan Components

RTO and RPO Definitions

Tiered Recovery Objectives

DR Plan Document Structure

1. SCOPE AND OBJECTIVES
   - Systems covered
   - RTO/RPO targets per tier
   - Responsible teams

2. CONTACT INFORMATION
   - Escalation chain
   - Vendor support contacts
   - Communication channels

3. RECOVERY PROCEDURES
   - Step-by-step per system tier
   - Decision trees for failure scenarios
   - Rollback procedures

4. DEPENDENCIES
   - Inter-service dependencies
   - Infrastructure requirements
   - Third-party service dependencies

5. TESTING SCHEDULE
   - Test frequency per tier
   - Success criteria
   - Reporting requirements

---

Types of DR Tests

Test Types Comparison

Tabletop Exercise

No actual systems are touched. Teams walk through scenarios verbally.

TABLETOP SCENARIO: Primary database corruption

Facilitator reads:
"At 2:00 AM, the on-call engineer receives alerts that the primary
PostgreSQL database is returning corruption errors. Write queries are
failing. Read replicas are 30 seconds behind and may have replicated
corrupted data."

Discussion questions:
1. What is your first action?
2. How do you determine if replicas are safe?
3. What is the failover procedure?
4. How do you validate data integrity after failover?
5. What is the communication plan for affected users?

Simulation Test

Trigger partial failover in a controlled environment.

#!/bin/bash
# DR simulation: database primary failover

set -euo pipefail

DR_LOG="/var/log/dr-drill-$(date +%Y%m%d).log"
START_TIME=$(date +%s)

log() { echo "[$(date -u +%H:%M:%S)] $1" | tee -a "$DR_LOG"; }

log "=== DR DRILL START: Database Failover Simulation ==="

# Step 1: Verify pre-conditions
log "Checking replication lag..."
LAG=$(psql -h replica.internal -c \
  "SELECT EXTRACT(EPOCH FROM replay_lag) FROM pg_stat_replication;" -t)
if (( $(echo "$LAG > 5" | bc -l) )); then
  log "ABORT: Replication lag ${LAG}s exceeds threshold"
  exit 1
fi

# Step 2: Record baseline metrics
log "Recording baseline..."
BASELINE_QPS=$(curl -s http://metrics.internal/api/v1/query?query=rate\(http_requests_total[1m]\) \
  | jq '.data.result[0].value[1]' -r)
log "Baseline QPS: $BASELINE_QPS"

# Step 3: Promote replica
log "Promoting replica to primary..."
pg_ctl promote -D /var/lib/postgresql/data

# Step 4: Update DNS / connection string
log "Updating service discovery..."
consul kv put db/primary/host replica.internal

# Step 5: Validate
log "Validating write capability on new primary..."
psql -h replica.internal -c \
  "INSERT INTO dr_validation (tested_at) VALUES (NOW());"

# Step 6: Measure recovery time
END_TIME=$(date +%s)
RECOVERY_SECONDS=$((END_TIME - START_TIME))
log "Recovery time: ${RECOVERY_SECONDS}s"

# Step 7: Verify RTO
RTO_TARGET=900  # 15 minutes
if [ "$RECOVERY_SECONDS" -le "$RTO_TARGET" ]; then
  log "PASS: RTO ${RECOVERY_SECONDS}s <= target ${RTO_TARGET}s"
else
  log "FAIL: RTO ${RECOVERY_SECONDS}s > target ${RTO_TARGET}s"
fi

log "=== DR DRILL COMPLETE ==="

---

Database Failover Testing

Primary-Replica Promotion

# PostgreSQL: verify replication status before failover
psql -h primary.internal -c "
  SELECT
    client_addr,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn,
    replay_lag
  FROM pg_stat_replication;
"

# Promote replica
pg_ctl promote -D /var/lib/postgresql/data

# Verify new primary accepts writes
psql -h new-primary.internal -c "
  CREATE TABLE IF NOT EXISTS dr_test (id SERIAL, ts TIMESTAMPTZ DEFAULT NOW());
  INSERT INTO dr_test DEFAULT VALUES;
  SELECT * FROM dr_test ORDER BY ts DESC LIMIT 1;
"

Cross-Region Database Failover

---

Backup Verification

Restore Testing

Backups that are never tested are not backups. Run restore tests on every backup type.

#!/bin/bash
# Automated backup restore verification

BACKUP_FILE="s3://backups/db/daily/2025-01-15.sql.gz"
TEST_DB="dr_restore_test_$(date +%Y%m%d)"

log "Downloading backup..."
aws s3 cp "$BACKUP_FILE" /tmp/restore.sql.gz

log "Restoring to test database..."
createdb "$TEST_DB"
gunzip -c /tmp/restore.sql.gz | psql "$TEST_DB"

log "Running integrity checks..."

# Row count comparison
EXPECTED_USERS=50000
ACTUAL_USERS=$(psql "$TEST_DB" -t -c "SELECT COUNT(*) FROM users;")
if [ "$ACTUAL_USERS" -lt "$EXPECTED_USERS" ]; then
  log "FAIL: Expected >= $EXPECTED_USERS users, got $ACTUAL_USERS"
fi

# Schema validation
EXPECTED_TABLES=42
ACTUAL_TABLES=$(psql "$TEST_DB" -t -c \
  "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='public';")
if [ "$ACTUAL_TABLES" -ne "$EXPECTED_TABLES" ]; then
  log "FAIL: Expected $EXPECTED_TABLES tables, got $ACTUAL_TABLES"
fi

# Foreign key integrity
FK_VIOLATIONS=$(psql "$TEST_DB" -t -c "
  SELECT COUNT(*) FROM orders o
  LEFT JOIN users u ON o.user_id = u.id
  WHERE u.id IS NULL;
")
if [ "$FK_VIOLATIONS" -gt 0 ]; then
  log "FAIL: $FK_VIOLATIONS orphaned order records"
fi

# Cleanup
dropdb "$TEST_DB"
rm /tmp/restore.sql.gz

log "Restore verification complete."

Data Integrity Checklist

• [ ] Backup file can be downloaded from storage
• [ ] Backup file is not corrupted (checksum verification)
• [ ] Restore completes without errors
• [ ] Row counts match expected values (within RPO window)
• [ ] Schema matches production (tables, indexes, constraints)
• [ ] Foreign key relationships are intact
• [ ] Application can connect and perform basic operations
• [ ] Point-in-time recovery (PITR) works to a specific timestamp

---

Multi-Region Failover

Active-Passive Failover Pattern

Normal Operation:
  Users → DNS (us-east-1) → Primary Region
                             └── DB Primary
                             └── App Servers (active)

  Standby: us-west-2 (warm, receiving replication)

Failover:
  Users → DNS (us-west-2) → DR Region
                             └── DB Replica (promoted)
                             └── App Servers (activated)

Failover Orchestration Script

import boto3
import time

class RegionFailover:
    """Orchestrate multi-region failover."""

    def __init__(self, primary_region: str, dr_region: str):
        self.primary = primary_region
        self.dr = dr_region
        self.route53 = boto3.client("route53")
        self.rds = boto3.client("rds", region_name=dr_region)

    def execute_failover(self, hosted_zone_id: str, record_name: str):
        steps = [
            ("Verify DR health", self.verify_dr_health),
            ("Promote DR database", self.promote_dr_database),
            ("Scale up DR compute", self.scale_dr_compute),
            ("Update DNS", lambda: self.update_dns(hosted_zone_id, record_name)),
            ("Validate end-to-end", self.validate_e2e),
        ]

        for step_name, step_fn in steps:
            print(f"[{time.strftime('%H:%M:%S')}] {step_name}...")
            try:
                step_fn()
                print(f"  -> OK")
            except Exception as e:
                print(f"  -> FAILED: {e}")
                raise FailoverAborted(f"Failed at: {step_name}")

    def promote_dr_database(self):
        self.rds.promote_read_replica_db_cluster(
            DBClusterIdentifier="dr-cluster"
        )
        waiter = self.rds.get_waiter("db_cluster_available")
        waiter.wait(
            DBClusterIdentifier="dr-cluster",
            WaiterConfig={"Delay": 10, "MaxAttempts": 60},
        )

    def update_dns(self, hosted_zone_id: str, record_name: str):
        self.route53.change_resource_record_sets(
            HostedZoneId=hosted_zone_id,
            ChangeBatch={
                "Changes": [{
                    "Action": "UPSERT",
                    "ResourceRecordSet": {
                        "Name": record_name,
                        "Type": "CNAME",
                        "TTL": 60,
                        "ResourceRecords": [
                            {"Value": f"app.{self.dr}.internal"},
                        ],
                    },
                }],
            },
        )

---

DNS Failover

TTL Considerations

Rule: DR-critical records should have TTL <= 60 seconds.

Health Check Configuration

# AWS Route 53 health check (CLI)
aws route53 create-health-check --caller-reference "dr-$(date +%s)" \
  --health-check-config '{
    "IPAddress": "203.0.113.10",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health",
    "RequestInterval": 10,
    "FailureThreshold": 3,
    "EnableSNI": true
  }'

---

Stateful Service Recovery

Recovery Priority Order

1. DNS / Load Balancers (stateless, fast)
2. Databases (stateful, promote replicas)
3. Message Queues (drain or replay)
4. Cache Layer (cold start acceptable)
5. Application Servers (stateless, scale up)
6. Background Workers (restart, reprocess)

Message Queue Recovery

# Kafka: verify consumer group lag after failover
from confluent_kafka.admin import AdminClient

admin = AdminClient({"bootstrap.servers": "dr-kafka:9092"})

def check_consumer_lag(group_id: str, max_lag: int = 10000):
    """Verify consumer group is catching up after DR failover."""
    groups = admin.list_consumer_group_offsets([group_id])
    total_lag = 0

    for topic_partition, offset_and_metadata in groups[group_id].result().items():
        watermarks = consumer.get_watermark_offsets(topic_partition)
        high_watermark = watermarks[1]
        committed = offset_and_metadata.offset
        lag = high_watermark - committed
        total_lag += lag

    if total_lag > max_lag:
        print(f"WARNING: Consumer lag {total_lag} exceeds threshold {max_lag}")
    return total_lag

---

DR Test Scheduling

---

Runbook Validation

Every DR drill should validate the runbook itself, not just the systems.

Runbook Validation Checklist

• [ ] Runbook is accessible when primary systems are down
• [ ] Contact information is current
• [ ] Commands execute successfully (copy-paste test)
• [ ] Credentials and access are pre-provisioned
• [ ] Screenshots and diagrams match current architecture
• [ ] Estimated times match actual drill times
• [ ] Escalation paths work (test page the on-call)
• [ ] External vendor contacts are reachable
• [ ] Runbook covers rollback/failback procedures

---

Compliance Requirements

SOC 2

• DR plan must be documented and reviewed annually
• DR tests must be executed at least annually
• Test results must be documented with evidence
• Gaps must have remediation plans with deadlines

ISO 27001

• Business continuity plan required (Annex A.17)
• DR capabilities must be tested at regular intervals
• Results must be reported to management
• Plans must be updated after organizational changes

Audit Evidence Template

DR TEST EVIDENCE
Date: ___________
Test type: [Tabletop | Walkthrough | Simulation | Full Failover]
Systems tested: ___________
Participants: ___________

Results:
- RTO achieved: ___ minutes (target: ___ minutes) [PASS/FAIL]
- RPO achieved: ___ minutes (target: ___ minutes) [PASS/FAIL]
- Data integrity verified: [YES/NO]
- All runbook steps valid: [YES/NO]

Findings:
1. ___________
2. ___________

Remediation items:
1. ___________ (owner: ___, due: ___)
2. ___________ (owner: ___, due: ___)

Approved by: ___________

---

Post-Drill Report Template

# DR Drill Report

## Summary
- Date: YYYY-MM-DD
- Duration: X hours
- Type: [Tabletop | Simulation | Full Failover]
- Scope: [Systems and regions tested]
- Overall result: [PASS | PARTIAL PASS | FAIL]

## Objectives
1. Validate RTO of X minutes for Tier Y systems
2. Verify RPO of X minutes for database Z
3. Test runbook accuracy for scenario W

## Results

| Objective                  | Target    | Actual    | Result |
|----------------------------|-----------|-----------|--------|
| RTO - Payment Service      | 15 min    | 12 min    | PASS   |
| RPO - User Database        | 1 min     | 45 sec    | PASS   |
| Runbook accuracy           | 100%      | 85%       | FAIL   |

## Findings
1. [Finding]: [Description and impact]
2. [Finding]: [Description and impact]

## Action Items
| Item                       | Owner     | Priority  | Due Date   |
|----------------------------|-----------|-----------|------------|
| Update DNS TTL             | SRE Team  | High      | YYYY-MM-DD |
| Fix runbook step 7         | On-call   | Medium    | YYYY-MM-DD |

## Lessons Learned
1. ___________
2. ___________

## Next Drill
- Scheduled: YYYY-MM-DD
- Focus areas: ___________

---

Related Resources

• chaos-engineering-guide.md -- Chaos experiment design for failure injection
• health-check-patterns.md -- Liveness and readiness probes
• graceful-degradation.md -- Fallback strategies during partial failure
• resilience-checklists.md -- Comprehensive resilience verification
• circuit-breaker-patterns.md -- Preventing cascade during failover

Graceful Degradation Patterns

Maintaining partial functionality during failures with strategic fallback mechanisms.

---

Pattern: Graceful Degradation

Use when: Maintaining partial functionality during failures.

Core Principle: Critical path succeeds even when non-critical dependencies fail.

---

Degradation Strategy 1: Cached Fallback

Use when: Stale data is better than no data.

async function getUserProfile(userId) {
  try {
    // Try primary source
    const user = await api.getUser(userId);

    // Update cache on success
    await cache.set(`user:${userId}`, user, 3600); // 1 hour TTL

    return user;
  } catch (error) {
    // Fall back to cache
    const cached = await cache.get(`user:${userId}`);
    if (cached) {
      logger.warn('Using cached user profile', { userId, error });
      return { ...cached, _degraded: true, _cached: true };
    }

    throw error;
  }
}

Cache-Aside Pattern with Graceful Degradation:

async function getProductDetails(productId) {
  // 1. Try cache first
  const cached = await cache.get(`product:${productId}`);
  if (cached && !cached._expired) {
    return cached;
  }

  // 2. Try database
  try {
    const product = await db.query('SELECT * FROM products WHERE id = $1', [productId]);

    if (!product) {
      throw new Error('Product not found');
    }

    // Update cache
    await cache.set(`product:${productId}`, product, 3600);

    return product;
  } catch (error) {
    // 3. Fall back to stale cache if database fails
    if (cached) {
      logger.warn('Database failed, using stale cache', { productId, error });
      return { ...cached, _degraded: true, _stale: true };
    }

    throw error;
  }
}

---

Degradation Strategy 2: Default Values

Use when: Sensible defaults maintain functionality.

async function getRecommendations(userId) {
  try {
    return await mlService.getRecommendations(userId);
  } catch (error) {
    logger.error('ML service failed, falling back to popular items', { error });

    // Fall back to popular items
    return {
      items: await getPopularItems(),
      _degraded: true,
      _reason: 'ML service unavailable',
      _fallback: 'popular_items',
    };
  }
}

async function getPopularItems() {
  // Return cached popular items (updated daily)
  return await cache.get('popular_items') || DEFAULT_POPULAR_ITEMS;
}

const DEFAULT_POPULAR_ITEMS = [
  { id: 1, name: 'Product A', score: 0.95 },
  { id: 2, name: 'Product B', score: 0.90 },
  { id: 3, name: 'Product C', score: 0.85 },
];

---

Degradation Strategy 3: Feature Toggles

Use when: Non-critical features can be disabled during failures.

async function processOrder(order) {
  // Critical: save order
  const result = await saveOrder(order);

  // Non-critical: send confirmation email
  if (featureFlags.isEnabled('email-notifications')) {
    try {
      await sendConfirmationEmail(order);
    } catch (error) {
      // Log but don't fail order
      logger.error('Email failed', { error, orderId: order.id });
      metrics.increment('email.send.failed');
    }
  }

  // Non-critical: update analytics
  if (featureFlags.isEnabled('analytics')) {
    try {
      await analytics.trackPurchase(order);
    } catch (error) {
      // Log but don't fail order
      logger.error('Analytics failed', { error, orderId: order.id });
    }
  }

  return result;
}

Dynamic Feature Flags with Circuit Breaker:

class FeatureToggle {
  constructor() {
    this.features = new Map();
    this.circuitBreakers = new Map();
  }

  isEnabled(featureName) {
    // Check if feature manually disabled
    if (!this.features.get(featureName)?.enabled) {
      return false;
    }

    // Check if circuit breaker is open
    const breaker = this.circuitBreakers.get(featureName);
    return !breaker || breaker.state !== 'OPEN';
  }

  async executeIfEnabled(featureName, fn, fallback = null) {
    if (!this.isEnabled(featureName)) {
      return fallback;
    }

    try {
      return await fn();
    } catch (error) {
      logger.error(`Feature ${featureName} failed`, { error });

      // Open circuit if too many failures
      const breaker = this.circuitBreakers.get(featureName);
      if (breaker) {
        breaker.recordFailure();
      }

      return fallback;
    }
  }
}

// Usage
const featureFlags = new FeatureToggle();

async function processOrder(order) {
  const result = await saveOrder(order);

  // Execute non-critical features with fallback
  await featureFlags.executeIfEnabled(
    'email-notifications',
    () => sendConfirmationEmail(order),
    null
  );

  return result;
}

---

Degradation Strategy 4: Reduced Functionality

Use when: Simpler fallback provides core value.

async function search(query) {
  try {
    // Try full-text search with advanced features
    return await elasticsearchService.search(query, {
      fuzzyMatch: true,
      synonyms: true,
      facets: true,
      personalization: true,
    });
  } catch (error) {
    logger.warn('Elasticsearch down, using SQL fallback', { error });

    // Fall back to basic SQL LIKE search
    const results = await db.query(
      'SELECT * FROM products WHERE name ILIKE $1 LIMIT 20',
      [`%${query}%`]
    );

    return {
      results,
      _degraded: true,
      _fallback: 'sql_search',
      _features_disabled: ['fuzzy_match', 'synonyms', 'facets', 'personalization'],
    };
  }
}

---

Degradation Strategy 5: Partial Responses

Use when: Some data is better than complete failure.

async function getUserDashboard(userId) {
  const results = await Promise.allSettled([
    getProfile(userId),
    getRecentOrders(userId),
    getRecommendations(userId),
    getNotifications(userId),
  ]);

  const [profile, orders, recommendations, notifications] = results;

  return {
    profile: profile.status === 'fulfilled' ? profile.value : null,
    orders: orders.status === 'fulfilled' ? orders.value : [],
    recommendations: recommendations.status === 'fulfilled' ? recommendations.value : [],
    notifications: notifications.status === 'fulfilled' ? notifications.value : [],
    _partial: results.some(r => r.status === 'rejected'),
    _errors: results
      .filter(r => r.status === 'rejected')
      .map((r, i) => ({ section: ['profile', 'orders', 'recommendations', 'notifications'][i], error: r.reason })),
  };
}

---

Degradation Strategy 6: Queue-Based Async Processing

Use when: Operation can be deferred.

async function createOrder(orderData) {
  // Critical: save order to database
  const order = await db.insert('orders', orderData);

  // Non-critical: enqueue async operations
  try {
    await queue.enqueue('send-confirmation-email', { orderId: order.id });
    await queue.enqueue('update-inventory', { orderId: order.id });
    await queue.enqueue('notify-analytics', { orderId: order.id });
  } catch (error) {
    // If queue fails, log but don't block order creation
    logger.error('Failed to enqueue async tasks', { error, orderId: order.id });
  }

  return order;
}

// Background worker processes queue
async function processQueue() {
  const job = await queue.dequeue();

  try {
    await processJob(job);
  } catch (error) {
    // Retry with exponential backoff
    await queue.enqueue(job.type, job.data, {
      delay: calculateBackoff(job.attempts),
      maxAttempts: 5,
    });
  }
}

---

User Experience for Degraded Mode

Indicate Degraded State:

// API response
{
  "data": { /* partial data */ },
  "status": {
    "degraded": true,
    "message": "Some features temporarily unavailable",
    "missing": ["recommendations", "personalization"],
    "eta": "2025-11-22T15:30:00Z"
  }
}

Frontend Handling:

function DashboardComponent({ userId }) {
  const { data, status } = useDashboard(userId);

  return (
    <div>
      {status.degraded && (
        <Alert severity="warning">
          Some features are temporarily unavailable. We're working on it!
        </Alert>
      )}

      {data.profile ? (
        <UserProfile profile={data.profile} />
      ) : (
        <Skeleton variant="rectangular" />
      )}

      {data.recommendations.length > 0 ? (
        <Recommendations items={data.recommendations} />
      ) : (
        <div>Recommendations currently unavailable</div>
      )}
    </div>
  );
}

---

Monitoring Degraded State

Metrics to Track:

// Track degradation rate
metrics.gauge('service.degraded', isDegraded ? 1 : 0);
metrics.increment('service.degradation.event', { reason: 'elasticsearch_down' });

// Track which features are degraded
metrics.gauge('feature.email.degraded', emailServiceDown ? 1 : 0);
metrics.gauge('feature.search.degraded', searchServiceDown ? 1 : 0);

// Track fallback usage
metrics.increment('fallback.cache.used', { service: 'user-profile' });
metrics.increment('fallback.default.used', { service: 'recommendations' });

Alerts:

# Alert when running in degraded mode for > 10 min
- alert: ServiceDegraded
  expr: service_degraded == 1
  for: 10m
  annotations:
    summary: "Service running in degraded mode"
    description: "{{ $labels.service }} has been degraded for > 10 min"

---

Checklist

• [ ] Critical path vs non-critical operations identified
• [ ] Fallback behavior defined for each dependency
• [ ] Degraded mode clearly indicated to users
• [ ] Cache strategies in place for common data
• [ ] Feature flags control non-essential features
• [ ] Degradation metrics tracked
• [ ] Alerts when running in degraded mode
• [ ] Default values configured for ML/recommendations
• [ ] Partial responses acceptable (Promise.allSettled)
• [ ] Queue-based processing for non-critical operations

---

Related Resources

• circuit-breaker-patterns.md - Trigger degradation when circuit opens
• retry-patterns.md - Retry before degrading
• timeout-policies.md - Timeout triggers degradation
• resilience-checklists.md - Comprehensive fallback strategies

Health Check Patterns

Monitoring service availability for orchestration systems with liveness, readiness, and startup probes.

---

Pattern: Health Checks

Use when: Monitoring service availability for orchestration systems.

Health Check Types:

1. Liveness Probe - Is the app alive? 2. Readiness Probe - Is the app ready to serve traffic? 3. Startup Probe - Has the app finished starting?

---

Liveness Probe

Purpose: Determine if the app is alive (restart if not).

Implementation:

app.get('/health/liveness', (req, res) => {
  // Simple: just respond (app is running)
  res.status(200).json({ status: 'alive' });
});

Best Practices:

• Keep it simple (no dependency checks)
• Fast response (<100ms)
• Don't include database or external service checks
• Use it to detect deadlocks, infinite loops, or process crashes

---

Readiness Probe

Purpose: Determine if the app is ready to serve traffic (remove from load balancer if not).

Implementation:

app.get('/health/readiness', async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    cache: await checkRedis(),
    externalAPI: await checkExternalAPI(),
  };

  const allHealthy = Object.values(checks).every((check) => check.healthy);

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'ready' : 'not_ready',
    checks,
  });
});

async function checkDatabase() {
  try {
    await db.raw('SELECT 1');
    return { healthy: true };
  } catch (error) {
    return { healthy: false, error: error.message };
  }
}

async function checkRedis() {
  try {
    await redis.ping();
    return { healthy: true };
  } catch (error) {
    return { healthy: false, error: error.message };
  }
}

async function checkExternalAPI() {
  try {
    const response = await fetch('https://api.example.com/health', {
      signal: AbortSignal.timeout(1000), // 1s timeout
    });
    return { healthy: response.ok };
  } catch (error) {
    return { healthy: false, error: error.message };
  }
}

Best Practices:

• Check all critical dependencies
• Fast checks (<2s total)
• Return 503 when not ready
• Include details for debugging
• Use timeouts for all checks

---

Startup Probe

Purpose: Determine if the app has finished starting (slow-starting apps).

Implementation:

let isReady = false;

// During startup
async function initialize() {
  await connectToDatabase();
  await warmupCache();
  await loadConfiguration();
  await preloadModels(); // ML models, etc.
  isReady = true;
}

app.get('/health/startup', (req, res) => {
  if (isReady) {
    res.status(200).json({ status: 'started' });
  } else {
    res.status(503).json({ status: 'starting' });
  }
});

// Start initialization on app launch
initialize().catch((error) => {
  console.error('Startup failed:', error);
  process.exit(1);
});

Best Practices:

• Use for slow-starting apps (>30s)
• Prevent premature restarts during initialization
• Higher failure threshold than liveness
• Disable liveness probe until startup succeeds

---

Kubernetes Configuration

Complete Probe Setup:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      image: myapp:latest

      # Liveness: Restart if app is dead
      livenessProbe:
        httpGet:
          path: /health/liveness
          port: 3000
        initialDelaySeconds: 30   # Wait 30s before first check
        periodSeconds: 10          # Check every 10s
        timeoutSeconds: 1          # 1s timeout
        failureThreshold: 3        # Restart after 3 failures

      # Readiness: Remove from LB if not ready
      readinessProbe:
        httpGet:
          path: /health/readiness
          port: 3000
        initialDelaySeconds: 5     # Start checking after 5s
        periodSeconds: 5           # Check every 5s
        timeoutSeconds: 1          # 1s timeout
        failureThreshold: 3        # Mark not ready after 3 failures
        successThreshold: 1        # Mark ready after 1 success

      # Startup: Allow slow initialization
      startupProbe:
        httpGet:
          path: /health/startup
          port: 3000
        initialDelaySeconds: 0     # Start checking immediately
        periodSeconds: 5           # Check every 5s
        timeoutSeconds: 1          # 1s timeout
        failureThreshold: 30       # Allow 150s startup (30 * 5s)

---

Advanced Health Checks

Shallow vs Deep Checks:

// Shallow: fast, minimal dependencies
app.get('/health', async (req, res) => {
  const isHealthy = await quickHealthCheck();
  res.status(isHealthy ? 200 : 503).json({ status: isHealthy ? 'healthy' : 'unhealthy' });
});

// Deep: comprehensive, slower
app.get('/health/deep', async (req, res) => {
  const checks = {
    database: await checkDatabaseConnection(),
    databasePerformance: await checkDatabaseQueryTime(),
    cache: await checkRedis(),
    disk: await checkDiskSpace(),
    memory: await checkMemoryUsage(),
    dependencies: await checkAllDependencies(),
  };

  const allHealthy = Object.values(checks).every((c) => c.healthy);

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'healthy' : 'unhealthy',
    checks,
    timestamp: new Date().toISOString(),
  });
});

async function checkDatabaseQueryTime() {
  const start = Date.now();
  try {
    await db.raw('SELECT 1');
    const duration = Date.now() - start;
    return {
      healthy: duration < 100, // <100ms is healthy
      duration,
    };
  } catch (error) {
    return { healthy: false, error: error.message };
  }
}

async function checkDiskSpace() {
  const disk = await getDiskUsage();
  return {
    healthy: disk.percentUsed < 90,
    percentUsed: disk.percentUsed,
    available: disk.available,
  };
}

async function checkMemoryUsage() {
  const memUsage = process.memoryUsage();
  const percentUsed = (memUsage.heapUsed / memUsage.heapTotal) * 100;
  return {
    healthy: percentUsed < 90,
    percentUsed,
    heapUsed: memUsage.heapUsed,
    heapTotal: memUsage.heapTotal,
  };
}

---

Dependency Health Checks

Aggregate Dependencies:

const dependencies = [
  { name: 'database', check: checkDatabase, critical: true },
  { name: 'cache', check: checkRedis, critical: true },
  { name: 'email', check: checkEmailService, critical: false },
  { name: 'analytics', check: checkAnalytics, critical: false },
];

app.get('/health/readiness', async (req, res) => {
  const results = await Promise.allSettled(
    dependencies.map(async (dep) => ({
      name: dep.name,
      critical: dep.critical,
      result: await dep.check(),
    }))
  );

  const checks = {};
  let criticalFailure = false;

  results.forEach((result, i) => {
    const dep = dependencies[i];
    if (result.status === 'fulfilled') {
      checks[dep.name] = result.value.result;
      if (dep.critical && !result.value.result.healthy) {
        criticalFailure = true;
      }
    } else {
      checks[dep.name] = { healthy: false, error: result.reason.message };
      if (dep.critical) {
        criticalFailure = true;
      }
    }
  });

  res.status(criticalFailure ? 503 : 200).json({
    status: criticalFailure ? 'not_ready' : 'ready',
    checks,
  });
});

---

Health Check Security

Prevent Abuse:

const rateLimit = require('express-rate-limit');

const healthCheckLimiter = rateLimit({
  windowMs: 1 * 60 * 1000, // 1 minute
  max: 60, // 60 requests per minute
  message: 'Too many health check requests',
});

app.get('/health', healthCheckLimiter, async (req, res) => {
  // Health check logic
});

// Don't require auth for health checks (load balancers need access)
// But log suspicious patterns
app.use('/health', (req, res, next) => {
  const userAgent = req.headers['user-agent'];
  if (!userAgent || !userAgent.includes('kube-probe')) {
    logger.warn('Health check from non-k8s source', {
      ip: req.ip,
      userAgent,
    });
  }
  next();
});

---

Monitoring Health Checks

Metrics:

app.get('/health/readiness', async (req, res) => {
  const startTime = Date.now();
  const checks = await performHealthChecks();
  const duration = Date.now() - startTime;

  const allHealthy = Object.values(checks).every((c) => c.healthy);

  // Record metrics
  metrics.histogram('health_check.duration', duration);
  metrics.gauge('health_check.status', allHealthy ? 1 : 0);

  Object.entries(checks).forEach(([name, check]) => {
    metrics.gauge(`health_check.dependency.${name}`, check.healthy ? 1 : 0);
  });

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'ready' : 'not_ready',
    checks,
    duration,
  });
});

---

Checklist

• [ ] Liveness probe checks app is alive (simple check)
• [ ] Readiness probe checks dependencies (database, cache, APIs)
• [ ] Startup probe for slow-starting apps
• [ ] Health checks timeout quickly (1-2s)
• [ ] Failed health checks logged
• [ ] Health check endpoints don't require auth
• [ ] Health checks don't overload dependencies
• [ ] Critical vs non-critical dependencies differentiated
• [ ] Health check metrics tracked
• [ ] Rate limiting prevents abuse

---

Related Resources

• graceful-degradation.md - Handle failed health checks gracefully
• timeout-policies.md - Configure health check timeouts
• resilience-checklists.md - Health and readiness checklist

Load Shedding and Backpressure

Overload protection patterns that keep systems stable under extreme traffic.

Contents

• Load Shedding Strategies
• Backpressure Propagation
• Queue-Based Buffering
• Rate Limiting vs Load Shedding
• Server-Side Admission Control
• Client-Side Throttling
• Implementation Patterns
• Monitoring Overload Signals
• Kubernetes Resource Limits and HPA
• Related Resources

---

Load Shedding Strategies

Load shedding rejects excess work early to protect the system from collapse. Unlike rate limiting (which caps throughput), load shedding adapts dynamically to current capacity.

Priority-Based Shedding

Assign priority tiers to requests and shed lowest-priority work first.

from enum import IntEnum
from fastapi import FastAPI, Request, HTTPException
import psutil

class Priority(IntEnum):
    CRITICAL = 1   # payments, auth
    HIGH = 2       # user-facing reads
    NORMAL = 3     # background syncs
    LOW = 4        # analytics, telemetry

app = FastAPI()

def get_request_priority(request: Request) -> Priority:
    path = request.url.path
    if path.startswith("/payments") or path.startswith("/auth"):
        return Priority.CRITICAL
    if request.method == "GET" and path.startswith("/api"):
        return Priority.HIGH
    if path.startswith("/analytics"):
        return Priority.LOW
    return Priority.NORMAL

def current_load() -> float:
    """Return CPU utilization as a fraction 0.0-1.0."""
    return psutil.cpu_percent(interval=0.1) / 100.0

SHED_THRESHOLDS = {
    Priority.LOW: 0.70,
    Priority.NORMAL: 0.80,
    Priority.HIGH: 0.90,
    Priority.CRITICAL: 0.98,
}

@app.middleware("http")
async def load_shedding_middleware(request: Request, call_next):
    load = current_load()
    priority = get_request_priority(request)
    threshold = SHED_THRESHOLDS[priority]

    if load > threshold:
        raise HTTPException(
            status_code=503,
            detail="Service overloaded",
            headers={"Retry-After": "5"},
        )

    return await call_next(request)

Random Early Detection (RED)

Probabilistic shedding that increases drop probability as load rises, inspired by network congestion control.

import random

def should_shed(current_load: float, min_thresh: float = 0.6,
                max_thresh: float = 0.9) -> bool:
    """Probabilistic shedding using RED algorithm."""
    if current_load < min_thresh:
        return False
    if current_load > max_thresh:
        return True

    # Linear probability between thresholds
    drop_prob = (current_load - min_thresh) / (max_thresh - min_thresh)
    return random.random() < drop_prob

Adaptive Shedding (CoDel-inspired)

Track request latency and shed when queue delay exceeds a target.

---

Backpressure Propagation

Backpressure slows producers when consumers cannot keep up, preventing unbounded resource growth.

Reactive Streams (Project Reactor / RxJava)

// Reactor: bounded request with backpressure
Flux.range(1, 1_000_000)
    .onBackpressureBuffer(256, dropped -> log.warn("Dropped: {}", dropped))
    .publishOn(Schedulers.boundedElastic())
    .flatMap(this::processItem, /* concurrency */ 16)
    .subscribe();

// Backpressure strategies
Flux.create(sink -> producer.onData(sink::next))
    .onBackpressureDrop(item -> metrics.increment("dropped"))    // drop newest
    .onBackpressureLatest()                                       // keep only latest
    .onBackpressureBuffer(1024, BufferOverflowStrategy.DROP_OLDEST)
    .subscribe();

gRPC Flow Control

gRPC uses HTTP/2 flow control windows. Servers can signal backpressure by pausing reads.

// Go gRPC server-side flow control
func (s *server) StreamData(req *pb.Request,
    stream pb.Service_StreamDataServer) error {

    for item := range dataChannel {
        // stream.Send blocks when the client's receive window is full,
        // naturally applying backpressure to the producer
        if err := stream.Send(&pb.DataItem{Value: item}); err != nil {
            return err
        }
    }
    return nil
}

TCP Backpressure

When the application stops reading from a socket, the TCP receive buffer fills, the window shrinks to zero, and the sender pauses. This is implicit backpressure -- no application code required, but it can cause head-of-line blocking across multiplexed connections.

Checklist -- Backpressure:

• [ ] Every producer/consumer pair has an explicit backpressure strategy
• [ ] Bounded buffers are used between pipeline stages
• [ ] Drop/overflow policy is documented and monitored
• [ ] gRPC streaming services rely on HTTP/2 flow control (do not disable)
• [ ] TCP receive buffer sizes are tuned for expected throughput

---

Queue-Based Buffering

Queues absorb short bursts but must be bounded to prevent memory exhaustion.

Bounded Queue Patterns

import asyncio

# Bounded asyncio queue with timeout
queue: asyncio.Queue[dict] = asyncio.Queue(maxsize=500)

async def enqueue(item: dict, timeout: float = 1.0):
    try:
        await asyncio.wait_for(queue.put(item), timeout=timeout)
    except asyncio.TimeoutError:
        # Queue full -- shed this request
        metrics.increment("queue.rejected")
        raise ServiceUnavailable("Queue full")

Overflow Policies

// Node.js bounded queue with reject policy
class BoundedQueue {
  constructor(maxSize) {
    this.maxSize = maxSize;
    this.items = [];
  }

  enqueue(item) {
    if (this.items.length >= this.maxSize) {
      throw new Error('Queue full');
    }
    this.items.push(item);
  }

  dequeue() {
    return this.items.shift();
  }

  get depth() {
    return this.items.length;
  }

  get utilization() {
    return this.items.length / this.maxSize;
  }
}

---

Rate Limiting vs Load Shedding

Use both together: Rate limiting caps individual clients. Load shedding protects the system when aggregate load from all clients exceeds capacity.

---

Server-Side Admission Control

Admission control gates requests before they consume significant resources.

package main

import (
    "net/http"
    "runtime"
    "sync/atomic"
)

type AdmissionController struct {
    inFlight     int64
    maxInFlight  int64
    cpuThreshold float64
}

func NewAdmissionController(maxInFlight int64) *AdmissionController {
    return &AdmissionController{
        maxInFlight:  maxInFlight,
        cpuThreshold: 0.85,
    }
}

func (ac *AdmissionController) Allow() bool {
    current := atomic.LoadInt64(&ac.inFlight)
    if current >= ac.maxInFlight {
        return false
    }

    // Check goroutine count as a proxy for load
    if runtime.NumGoroutine() > 10000 {
        return false
    }

    atomic.AddInt64(&ac.inFlight, 1)
    return true
}

func (ac *AdmissionController) Release() {
    atomic.AddInt64(&ac.inFlight, -1)
}

func (ac *AdmissionController) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !ac.Allow() {
            w.Header().Set("Retry-After", "5")
            http.Error(w, "Service overloaded", http.StatusServiceUnavailable)
            return
        }
        defer ac.Release()
        next.ServeHTTP(w, r)
    })
}

---

Client-Side Throttling

Clients should self-throttle when receiving overload signals instead of retrying aggressively.

class AdaptiveThrottle {
  constructor() {
    this.requests = 0;
    this.accepts = 0;
    this.windowMs = 60_000;
    this.history = [];
  }

  // Google SRE client-side throttling formula
  rejectionProbability() {
    const now = Date.now();
    this.history = this.history.filter(h => now - h.time < this.windowMs);
    const requests = this.history.length;
    const accepts = this.history.filter(h => h.accepted).length;

    // P(reject) = max(0, (requests - K * accepts) / (requests + 1))
    const K = 2.0; // multiplier (higher = more lenient)
    return Math.max(0, (requests - K * accepts) / (requests + 1));
  }

  async send(requestFn) {
    const rejectProb = this.rejectionProbability();
    if (Math.random() < rejectProb) {
      throw new Error('Client-side throttled');
    }

    this.history.push({ time: Date.now(), accepted: false });
    const entry = this.history[this.history.length - 1];

    try {
      const result = await requestFn();
      entry.accepted = true;
      return result;
    } catch (err) {
      if (err.status === 503 || err.status === 429) {
        // Server rejected -- entry stays accepted=false
      } else {
        entry.accepted = true; // non-overload errors count as accepts
      }
      throw err;
    }
  }
}

---

Implementation Patterns

Node.js -- Express Overload Protection

const toobusy = require('toobusy-js');

// Tune lag threshold (ms of event loop delay)
toobusy.maxLag(70);
toobusy.interval(500);

app.use((req, res, next) => {
  if (toobusy()) {
    res.status(503).set('Retry-After', '5').json({
      error: 'Server too busy',
    });
    return;
  }
  next();
});

process.on('SIGINT', () => {
  toobusy.shutdown();
  process.exit();
});

Python -- Uvicorn with Concurrency Limit

from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
import asyncio

class ConcurrencyLimitMiddleware(BaseHTTPMiddleware):
    def __init__(self, app, max_concurrent: int = 100):
        super().__init__(app)
        self.semaphore = asyncio.Semaphore(max_concurrent)

    async def dispatch(self, request, call_next):
        if self.semaphore._value == 0:
            return JSONResponse(
                {"error": "Too many concurrent requests"},
                status_code=503,
                headers={"Retry-After": "2"},
            )
        async with self.semaphore:
            return await call_next(request)

---

Monitoring Overload Signals

Track these metrics to detect overload before it causes failures.

Checklist -- Monitoring:

• [ ] Dashboard shows in-flight requests, queue depth, and shed rate
• [ ] Alerts fire when shedding begins (indicates capacity issue)
• [ ] Shed reason is logged (CPU, queue, concurrency, memory)
• [ ] Client-facing 503 responses include Retry-After header
• [ ] Load tests validate shedding activates at expected thresholds

---

Kubernetes Resource Limits and HPA

Resource Limits

# deployment.yaml
resources:
  requests:
    cpu: "500m"
    memory: "256Mi"
  limits:
    cpu: "1000m"     # hard ceiling prevents noisy neighbors
    memory: "512Mi"  # OOMKill if exceeded

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: http_requests_in_flight
        target:
          type: AverageValue
          averageValue: "50"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

Key principle: HPA handles gradual load growth. Load shedding handles sudden spikes that outpace autoscaling (scale-up takes 30-120s; overload happens in seconds).

---

Related Resources

• circuit-breaker-patterns.md -- Circuit breakers for dependency failure isolation
• bulkhead-isolation.md -- Resource compartmentalization patterns
• timeout-policies.md -- Timeout configuration for bounded waits
• retry-patterns.md -- Retry strategies that avoid retry storms
• graceful-degradation.md -- Fallback behavior during overload
• resilience-checklists.md -- Comprehensive resilience verification

Resilience and Reliability Checklists (2026 Best Practices)

Use these checklists when hardening distributed systems against failures.

Dependency Resilience

• Circuit breakers on all external dependencies with alerting on open/half-open states
• Timeouts per dependency (short, explicit, never infinite)
• Retries with exponential backoff + jitter; cap attempts to avoid storms; use per-try timeouts and a total deadline budget
• Bulkhead isolation for noisy neighbors (connection pools, worker pools, queuing)
• Idempotent handlers for retried operations; use request IDs for dedupe
• Fallbacks: cached data, stale reads, default responses, partial renders
• Backpressure and load shedding when latency or queue depth breach SLOs
• DR readiness: failover drills, backup/restore tests, and clear RPO/RTO targets (when applicable)

Health and Readiness

• Liveness probes for process health; readiness probes for dependency health
• Startup probes for slow boot sequences
• Dependency contracts documented (SLO, timeout budget, error budget owner)
• Synthetic checks for critical user journeys, not only host-level checks
• Graceful shutdown hooks with in-flight request drains

Observability for Resilience

• Golden signals tracked (latency, error rate, traffic, saturation)
• High-cardinality logs avoided; structured logging with correlation IDs
• Distributed tracing across service boundaries with clear service names
• Error budget burn alerts (multi-window, multi-burn-rate)
• Automated rollback triggers tied to SLO breaches

Failure Testing

• Chaos experiments planned with blast radius limits and auto-revert
• Load tests cover peak + failover scenarios (zonal outage, dependency slowness)
• Game days scheduled with documented scenarios and owners
• Post-incident reviews with action items tracked to completion

Retry Patterns (Backoff, Jitter, Retry Budgets)

Production-ready retry guidance for transient failures in distributed systems.

---

Core Rules

• Bound retries by an overall deadline (timeout budget) and a retry budget.
• Use exponential backoff with jitter.
• Retry only idempotent operations (or require idempotency keys / dedupe).
• Respect server guidance (for example Retry-After) for 429 / 503.
• Prevent retry storms: cap attempts, cap max delay, and add client-side rate limiting.

---

Retry Decision Table (Starting Point)

---

Reference Implementation (Node.js fetch)

This is intentionally library-agnostic so it works even when retry libraries cannot honor Retry-After precisely.

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function parseRetryAfterMs(retryAfter) {
  if (!retryAfter) return null;
  const seconds = Number(retryAfter);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const dateMs = Date.parse(retryAfter);
  if (Number.isFinite(dateMs)) return Math.max(0, dateMs - Date.now());
  return null;
}

function computeBackoffMs(attempt, baseMs, maxMs) {
  const exp = Math.min(maxMs, baseMs * 2 ** (attempt - 1));
  const jitter = exp * (0.5 + Math.random()); // 0.5x..1.5x
  return Math.min(maxMs, Math.floor(jitter));
}

async function fetchWithRetry(
  url,
  init = {},
  {
    attempts = 3,
    perTryTimeoutMs = 3000,
    baseBackoffMs = 200,
    maxBackoffMs = 5000,
    overallDeadlineMs = 10000,
  } = {}
) {
  const deadlineAt = Date.now() + overallDeadlineMs;

  for (let attempt = 1; attempt <= attempts; attempt++) {
    const remainingMs = deadlineAt - Date.now();
    if (remainingMs <= 0) throw new Error('Retry deadline exhausted');

    const controller = new AbortController();
    const timeoutId = setTimeout(
      () => controller.abort(),
      Math.min(perTryTimeoutMs, remainingMs)
    );

    try {
      const res = await fetch(url, { ...init, signal: controller.signal });

      if (res.status === 429 || res.status === 503) {
        const retryAfterMs = parseRetryAfterMs(res.headers.get('Retry-After'));
        const err = new Error(`HTTP ${res.status}`);
        err.retryAfterMs = retryAfterMs;
        throw err;
      }

      if (res.status >= 500 && res.status < 600) {
        throw new Error(`HTTP ${res.status}`);
      }

      if (!res.ok) {
        const err = new Error(`HTTP ${res.status}`);
        err.retryable = false;
        throw err;
      }

      return res;
    } catch (err) {
      const retryable =
        err?.retryable !== false &&
        (err?.name === 'AbortError' || err?.retryAfterMs != null || err instanceof TypeError);

      if (!retryable || attempt === attempts) throw err;

      const serverDelayMs = err?.retryAfterMs ?? 0;
      const backoffMs = computeBackoffMs(attempt, baseBackoffMs, maxBackoffMs);
      const delayMs = Math.min(
        Math.max(serverDelayMs, backoffMs),
        Math.max(0, deadlineAt - Date.now())
      );

      await sleep(delayMs);
    } finally {
      clearTimeout(timeoutId);
    }
  }

  throw new Error('Unreachable');
}

---

Idempotency Notes

• Safe to retry: GET, PUT (same payload), DELETE, and POST with an idempotency key + server-side dedupe.
• Avoid retrying: non-idempotent writes without a dedupe strategy (creates duplicate side effects).

---

Checklist

• Every retry loop has an overall deadline and a max-attempt cap.
• Backoff uses jitter and caps maximum delay.
• Retries are safe (idempotent) or protected by idempotency keys/dedup.
• 429/503 honor Retry-After when provided.
• Retries are paired with timeouts, bulkheads, and circuit breakers to avoid cascading failures.

---

Related Resources

• timeout-policies.md - Per-try + overall deadline budgets
• circuit-breaker-patterns.md - Avoid retrying into a broken dependency
• resilience-checklists.md - Release and production hardening checks

Timeout Policies

Preventing resource exhaustion from slow dependencies with comprehensive timeout strategies.

---

Pattern: Timeout Policies

Use when: Preventing resource exhaustion from slow dependencies.

Timeout Types:

1. Overall Request Deadline (Typical):

// Overall request deadline (fetch does not expose a portable "connect timeout")
const response = await fetch('https://api.example.com/data', {
  signal: AbortSignal.timeout(5000), // 5s total deadline
});

2. Request Deadline (AbortController):

// Total time for request + response
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 10000);

try {
  const response = await fetch('https://api.example.com/data', {
    signal: controller.signal,
  });
  const data = await response.json();
  return data;
} finally {
  clearTimeout(timeoutId);
}

3. Idle / Read Timeout (Streaming):

Idle/read timeouts are client-specific. For streaming responses, enforce a per-chunk deadline in your HTTP client (or implement an application-level watchdog) rather than relying on a single global timer.

---

Database Query Timeouts

PostgreSQL with Prisma:

// Postgres statement_timeout (one option): set it per-transaction.
// Note: adjust the model/query to match your schema.
const result = await prisma.$transaction(async (tx) => {
  await tx.$executeRawUnsafe('SET LOCAL statement_timeout = 5000'); // 5s
  return tx.order.findMany({ where: { userId } });
});

MySQL:

await connection.query({
  sql: 'SELECT * FROM orders WHERE user_id = ?',
  timeout: 5000,
  values: [userId],
});

PostgreSQL with node-postgres:

const client = await pool.connect();
try {
  // Set statement timeout for this connection
  await client.query('SET statement_timeout = 5000'); // 5s
  const result = await client.query('SELECT * FROM large_table WHERE condition = $1', [value]);
  return result.rows;
} finally {
  client.release();
}

Python SQLAlchemy:

from sqlalchemy import text
from sqlalchemy.exc import DBAPIError
import asyncio

async def query_with_timeout(session, query, timeout=5):
    try:
        # Set statement timeout
        await session.execute(text(f"SET statement_timeout = '{timeout * 1000}'"))
        result = await session.execute(query)
        return result.fetchall()
    except DBAPIError as e:
        if 'statement timeout' in str(e):
            raise TimeoutError('Query exceeded timeout')
        raise

---

HTTP Client Timeouts

Node.js fetch with AbortController:

async function fetchWithTimeout(url, options = {}, timeout = 10000) {
  const controller = new AbortController();
  const id = setTimeout(() => controller.abort(), timeout);

  try {
    const response = await fetch(url, {
      ...options,
      signal: controller.signal,
    });

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    return await response.json();
  } catch (error) {
    if (error.name === 'AbortError') {
      throw new Error(`Request timeout after ${timeout}ms`);
    }
    throw error;
  } finally {
    clearTimeout(id);
  }
}

// Usage
try {
  const data = await fetchWithTimeout('https://api.example.com/data', {}, 5000);
} catch (error) {
  console.error('Request failed:', error);
}

Python requests:

import requests

try:
    response = requests.get(
        'https://api.example.com/data',
        timeout=(5, 10)  # (connect timeout, read timeout)
    )
    response.raise_for_status()
    return response.json()
except requests.Timeout:
    print('Request timeout')
except requests.RequestException as e:
    print(f'Request failed: {e}')

---

Timeout Recommendations

---

Nested Timeout Budgets

Use when: Multiple dependent operations with overall deadline.

class TimeoutBudget {
  constructor(totalTimeout) {
    this.deadline = Date.now() + totalTimeout;
  }

  remaining() {
    const remaining = this.deadline - Date.now();
    if (remaining <= 0) {
      throw new Error('Budget exhausted');
    }
    return remaining;
  }

  async execute(fn, label) {
    const timeout = this.remaining();
    const controller = new AbortController();
    const id = setTimeout(() => controller.abort(), timeout);

    try {
      return await fn({ signal: controller.signal, timeout });
    } catch (error) {
      if (error.name === 'AbortError') {
        throw new Error(`${label} timeout (${timeout}ms remaining)`);
      }
      throw error;
    } finally {
      clearTimeout(id);
    }
  }
}

// Usage: Chain operations with shared budget
async function processOrder(orderId) {
  const budget = new TimeoutBudget(30000); // 30s total

  // Step 1: Fetch order (max 10s)
  const order = await budget.execute(
    async ({ signal, timeout }) => {
      return await fetch(`/api/orders/${orderId}`, { signal });
    },
    'Fetch order'
  );

  // Step 2: Process payment (max remaining time)
  const payment = await budget.execute(
    async ({ signal, timeout }) => {
      return await paymentAPI.charge(order, { signal });
    },
    'Process payment'
  );

  // Step 3: Send confirmation (max remaining time)
  await budget.execute(
    async ({ signal, timeout }) => {
      return await emailAPI.send(order.email, { signal });
    },
    'Send confirmation'
  );

  return { order, payment };
}

---

Timeout Error Handling

Graceful Degradation:

async function getUserProfile(userId) {
  try {
    return await fetchWithTimeout(`/api/users/${userId}`, {}, 5000);
  } catch (error) {
    if (error.message.includes('timeout')) {
      // Return cached data on timeout
      const cached = await cache.get(`user:${userId}`);
      if (cached) {
        return { ...cached, _degraded: true };
      }
    }
    throw error;
  }
}

Logging and Metrics:

async function fetchWithTimeoutAndMetrics(url, options = {}, timeout = 10000) {
  const startTime = Date.now();

  try {
    const result = await fetchWithTimeout(url, options, timeout);
    metrics.histogram('http.request.duration', Date.now() - startTime);
    return result;
  } catch (error) {
    metrics.histogram('http.request.duration', Date.now() - startTime);

    if (error.message.includes('timeout')) {
      metrics.increment('http.request.timeout');
      logger.warn('Request timeout', { url, timeout });
    }

    throw error;
  }
}

---

Checklist

• [ ] All external calls have timeouts
• [ ] Database queries have statement timeouts
• [ ] HTTP client timeouts configured
• [ ] Timeouts logged when exceeded
• [ ] Graceful degradation on timeout
• [ ] Timeout metrics tracked (P50, P99)
• [ ] Connection timeout < request timeout
• [ ] Timeout budgets for nested operations
• [ ] Alerts when timeout rates spike

---

Related Resources

• circuit-breaker-patterns.md - Combine timeouts with circuit breakers
• retry-patterns.md - Retry timed-out operations
• graceful-degradation.md - Handle timeout failures gracefully
• resilience-checklists.md - Comprehensive timeout hardening