Error Handling Patterns

Error Handling Patterns

Design error handling strategies that make failures explicit, recoverable, and debuggable. The central skill is matching error handling style to error semantics: not all errors are equal, and treating them equally produces systems that are equally bad at handling all of them.

When to Use

✅ Use for:

• Choosing between exceptions, Result types, or error codes for a domain
• Designing typed error hierarchies in TypeScript or Python
• Implementing retry logic with backoff, jitter, and circuit breaking
• Building React error boundaries and graceful degradation
• Structuring error information for both users and developers
• Python exception chaining and __cause__ / __context__ semantics

❌ NOT for:

• Debugging a specific runtime error (use debugger or domain skill)
• Logging pipeline infrastructure (use observability skill)
• APM/monitoring configuration (use site-reliability-engineer skill)
• Writing tests for error paths (use vitest-testing-patterns skill)

---

Core Decision: Exception vs Result Type vs Error Code

flowchart TD
    Q1{Is this a programming error\nor contract violation?} -->|Yes| EX[Throw exception\nlet it crash]
    Q1 -->|No| Q2{Is the error part of\nnormal control flow?}
    Q2 -->|Yes| Q3{What is the call site context?}
    Q2 -->|No| Q4{Do callers need to\ndistinguish error types?}
    Q3 -->|Functional / monad-friendly| RT[Result or Either type]
    Q3 -->|Simple script or CLI| EC[Error code + message]
    Q4 -->|Yes| EH[Typed exception hierarchy]
    Q4 -->|No| GE[Generic exception\nwith structured message]
    EX --> NOTE1[Never catch at boundary —\nlet process restart]
    RT --> NOTE2[Compose with map/flatMap;\ncheck references/error-hierarchy-examples.md]
    EH --> NOTE3[See hierarchy design rules below]

Rules of thumb:

• Library code: prefer Result types — never force callers to handle your exceptions
• Application code: typed exception hierarchies work well; errors are exceptional
• CLI / scripts: error codes are fine; the user is the error boundary
• Async workers: Result types or structured error objects with retry metadata

---

Error Classification

Classify every error along two axes before deciding how to handle it:

This classification determines:

• Whether to retry (transient only)
• What to show the user (user-actionable → message; system → generic error + tracking ID)
• Whether to alert on-call (system permanent → page; transient spikes → alert)

---

Should This Error Be Retried?

flowchart TD
    E[Error occurs] --> C1{Is error transient?\nTimeout, 429, 503, connection reset}
    C1 -->|No| FAIL[Fail immediately\nReturn error to caller]
    C1 -->|Yes| C2{Have we exceeded\nmax retry attempts?}
    C2 -->|Yes| DLQ[Send to dead letter queue\nor return final failure]
    C2 -->|No| C3{Is circuit breaker OPEN?}
    C3 -->|Yes| CB[Return circuit-open error\nDo not attempt request]
    C3 -->|No| WAIT[Wait: exponential backoff\n+ full jitter]
    WAIT --> RETRY[Retry request]
    RETRY --> C1
    CB --> PROBE{After timeout:\nsend probe request}
    PROBE -->|Success| CLOSE[Close circuit\nResume normal traffic]
    PROBE -->|Fail| CB

Consult references/retry-patterns.md for backoff formulas, jitter strategies, and circuit breaker implementation.

---

TypeScript: Error Hierarchy Design

// Base application error — all domain errors extend this
class AppError extends Error {
  readonly code: string;
  readonly statusCode: number;
  readonly isOperational: boolean; // false = programmer error, crash process

  constructor(message: string, code: string, statusCode: number, isOperational = true) {
    super(message);
    this.name = this.constructor.name;
    this.code = code;
    this.statusCode = statusCode;
    this.isOperational = isOperational;
    Error.captureStackTrace(this, this.constructor);
  }
}

// Domain-specific errors
class ValidationError extends AppError {
  readonly fields: Record<string, string[]>;
  constructor(fields: Record<string, string[]>) {
    super('Validation failed', 'VALIDATION_ERROR', 422);
    this.fields = fields;
  }
}

class NotFoundError extends AppError {
  constructor(resource: string, id: string) {
    super(`${resource} ${id} not found`, 'NOT_FOUND', 404);
  }
}

class RateLimitError extends AppError {
  readonly retryAfterMs: number;
  constructor(retryAfterMs: number) {
    super('Rate limit exceeded', 'RATE_LIMIT', 429);
    this.retryAfterMs = retryAfterMs;
  }
}

Consult references/error-hierarchy-examples.md for Python equivalents, Result type implementations, and full hierarchy patterns.

---

Result Type Pattern (TypeScript)

When errors are expected outcomes of operations (parsing, API calls, DB queries), use Result instead of throw:

type Result<T, E = AppError> =
  | { ok: true; value: T }
  | { ok: false; error: E };

// Helpers
const ok = <T>(value: T): Result<T, never> => ({ ok: true, value });
const err = <E>(error: E): Result<never, E> => ({ ok: false, error });

// Usage — caller is forced to handle both cases
async function fetchUser(id: string): Promise<Result<User, NotFoundError | NetworkError>> {
  try {
    const user = await db.users.findById(id);
    if (!user) return err(new NotFoundError('User', id));
    return ok(user);
  } catch (e) {
    return err(new NetworkError('DB unavailable', { cause: e }));
  }
}

// At call site — no silent failures
const result = await fetchUser(userId);
if (!result.ok) {
  if (result.error instanceof NotFoundError) return res.status(404).json(...);
  return res.status(500).json(...);
}
const user = result.value; // typed, safe

---

React Error Boundaries

Error boundaries catch render-time exceptions. They do NOT catch async errors (fetch failures, setTimeout, event handlers).

class RouteErrorBoundary extends React.Component<Props, State> {
  static getDerivedStateFromError(error: Error): State {
    return { hasError: true, error };
  }

  componentDidCatch(error: Error, info: React.ErrorInfo) {
    // Log to error tracking, not console.error in production
    logger.error('Render error', { error, componentStack: info.componentStack });
  }

  render() {
    if (this.state.hasError) {
      return <ErrorFallback error={this.state.error} onRetry={this.reset} />;
    }
    return this.props.children;
  }
}

Place boundaries at route level (one per page) and around isolated expensive subtrees (charts, rich editors). Do not wrap every component — too granular breaks the benefit.

---

Python: Exception Chaining

Python's raise X from Y syntax preserves causal chains — use it always when re-raising:

class AppError(Exception):
    """Base error. All domain errors subclass this."""
    def __init__(self, message: str, code: str, status: int = 500):
        super().__init__(message)
        self.code = code
        self.status = status

class DatabaseError(AppError):
    def __init__(self, operation: str, cause: Exception):
        super().__init__(f"DB error during {operation}", "DB_ERROR", 503)
        self.__cause__ = cause  # explicit chain

# In application code
try:
    result = db.execute(query)
except psycopg2.OperationalError as e:
    raise DatabaseError("user_fetch", e) from e  # preserves full traceback

---

Structured Error Logging

Log errors with enough context to diagnose without reading code:

// Good: structured, queryable, developer-oriented
logger.error('Payment processing failed', {
  error: {
    code: error.code,
    message: error.message,
    stack: error.stack,
  },
  context: {
    userId,
    orderId,
    amount,
    paymentProvider,
    attempt: retryCount,
  },
  correlation: { requestId, traceId },
});

// Then surface a sanitized message to the user
// NEVER leak error.message to users — it may contain internals
return res.status(500).json({
  error: 'Payment could not be processed. Please try again.',
  errorId: requestId, // so support can look it up
});

---

Anti-Patterns

Anti-Pattern: Pokemon Exception Handling

Novice: "Wrap everything in try/catch and log the error. At least it won't crash."

Expert: Catching all exceptions unconditionally ("gotta catch 'em all") hides programmer errors, masks resource leaks, and converts loud failures into silent corruption. The system appears healthy while data is being silently dropped.

// Wrong — swallows everything including programming errors
try {
  await processOrder(order);
} catch (e) {
  console.error('something went wrong', e); // lost forever
}

// Right — catch only what you can handle, let the rest propagate
try {
  await processOrder(order);
} catch (e) {
  if (e instanceof RateLimitError) {
    await queue.requeue(order, { delay: e.retryAfterMs });
    return;
  }
  // programming errors, unexpected DB errors — let them crash
  throw e;
}

Detection: catch (e) { }, catch (e) { log(e) } with no rethrow, except Exception as e: pass in Python. Any catch block with no condition and no rethrow.

Timeline: This has always been wrong. Renewed urgency in async/await era (2017+) because swallowed promise rejections are even harder to detect than swallowed sync exceptions.

---

Anti-Pattern: Stringly-Typed Errors

Novice: "I'll put the error type in the message string: throw new Error('NOT_FOUND: User 123')"

Expert: String-based error types force callers to parse strings, break under refactoring, provide no IDE support, and make exhaustive matching impossible. Callers pattern-match on strings that drift as the codebase evolves.

// Wrong — caller must parse strings, breaks silently on rename
throw new Error(`RATE_LIMIT: retry after ${ms}ms`);
// Caller: if (error.message.startsWith('RATE_LIMIT')) { ... }

// Right — typed, refactor-safe, IDE-navigable
throw new RateLimitError(ms);
// Caller: if (error instanceof RateLimitError) { ... error.retryAfterMs ... }

Python equivalent:

# Wrong
raise Exception(f"rate_limit:{retry_after}")

# Right
raise RateLimitError(retry_after_ms=retry_after)

LLM mistake: LLMs trained on StackOverflow examples frequently generate stringly-typed errors because SO answers prioritize brevity over correctness. Error codes as strings look concise in tutorials.

Detection: instanceof Error checks everywhere, string .startsWith() or .includes() in catch blocks, error codes stored in message field rather than a dedicated property.

---

References

• references/retry-patterns.md — Consult when implementing retry logic: exponential backoff formulas, full vs equal jitter, circuit breaker state machine, dead letter queues
• references/error-hierarchy-examples.md — Consult for complete TypeScript and Python typed error class examples, Result monad implementations, and error boundary patterns

Error Hierarchy Examples

Complete, production-ready error class hierarchies for TypeScript and Python. These are reference implementations, not templates — copy and adapt them.

---

TypeScript: Full Application Error Hierarchy

// ============================================================================
// Base Error
// ============================================================================

/**
 * All application errors extend AppError.
 * isOperational: true  = expected failure, log and handle
 * isOperational: false = programming error, crash the process
 */
export class AppError extends Error {
  readonly code: string;
  readonly statusCode: number;
  readonly isOperational: boolean;
  readonly context?: Record<string, unknown>;

  constructor(
    message: string,
    code: string,
    statusCode: number,
    isOperational = true,
    context?: Record<string, unknown>
  ) {
    super(message);
    this.name = this.constructor.name;
    this.code = code;
    this.statusCode = statusCode;
    this.isOperational = isOperational;
    this.context = context;
    Error.captureStackTrace(this, this.constructor);
  }

  toJSON() {
    return {
      name: this.name,
      code: this.code,
      message: this.message,
      statusCode: this.statusCode,
      context: this.context,
    };
  }
}

// ============================================================================
// HTTP / API Errors
// ============================================================================

export class BadRequestError extends AppError {
  constructor(message: string, context?: Record<string, unknown>) {
    super(message, 'BAD_REQUEST', 400, true, context);
  }
}

export class ValidationError extends AppError {
  readonly fields: Record<string, string[]>;

  constructor(fields: Record<string, string[]>, message = 'Validation failed') {
    super(message, 'VALIDATION_ERROR', 422, true);
    this.fields = fields;
  }

  toJSON() {
    return { ...super.toJSON(), fields: this.fields };
  }
}

export class UnauthorizedError extends AppError {
  constructor(message = 'Authentication required') {
    super(message, 'UNAUTHORIZED', 401);
  }
}

export class ForbiddenError extends AppError {
  constructor(message = 'Insufficient permissions') {
    super(message, 'FORBIDDEN', 403);
  }
}

export class NotFoundError extends AppError {
  constructor(resource: string, identifier?: string | number) {
    const message = identifier
      ? `${resource} '${identifier}' not found`
      : `${resource} not found`;
    super(message, 'NOT_FOUND', 404, true, { resource, identifier });
  }
}

export class ConflictError extends AppError {
  constructor(message: string, context?: Record<string, unknown>) {
    super(message, 'CONFLICT', 409, true, context);
  }
}

export class RateLimitError extends AppError {
  readonly retryAfterMs: number;

  constructor(retryAfterMs: number, message = 'Rate limit exceeded') {
    super(message, 'RATE_LIMIT', 429, true, { retryAfterMs });
    this.retryAfterMs = retryAfterMs;
  }
}

// ============================================================================
// Infrastructure Errors
// ============================================================================

export class DatabaseError extends AppError {
  readonly operation: string;

  constructor(operation: string, cause?: Error) {
    super(`Database error during ${operation}`, 'DATABASE_ERROR', 503, true, { operation });
    this.operation = operation;
    if (cause) this.cause = cause;
  }
}

export class NetworkError extends AppError {
  readonly endpoint: string;

  constructor(endpoint: string, cause?: Error) {
    super(`Network error reaching ${endpoint}`, 'NETWORK_ERROR', 503, true, { endpoint });
    this.endpoint = endpoint;
    if (cause) this.cause = cause;
  }
}

export class TimeoutError extends AppError {
  readonly timeoutMs: number;

  constructor(operation: string, timeoutMs: number) {
    super(`${operation} timed out after ${timeoutMs}ms`, 'TIMEOUT', 504, true, { operation, timeoutMs });
    this.timeoutMs = timeoutMs;
  }
}

export class CircuitOpenError extends AppError {
  constructor(service: string) {
    super(`Circuit breaker OPEN for ${service}`, 'CIRCUIT_OPEN', 503, true, { service });
  }
}

// ============================================================================
// Business Logic Errors
// ============================================================================

export class InsufficientFundsError extends AppError {
  readonly required: number;
  readonly available: number;

  constructor(required: number, available: number, currency = 'USD') {
    super(
      `Insufficient funds: required ${required} ${currency}, available ${available} ${currency}`,
      'INSUFFICIENT_FUNDS',
      402,
      true,
      { required, available, currency }
    );
    this.required = required;
    this.available = available;
  }
}

export class FeatureNotAvailableError extends AppError {
  constructor(feature: string, requiredPlan: string) {
    super(
      `${feature} requires ${requiredPlan} plan`,
      'FEATURE_NOT_AVAILABLE',
      403,
      true,
      { feature, requiredPlan }
    );
  }
}

// ============================================================================
// Type guards
// ============================================================================

export function isAppError(e: unknown): e is AppError {
  return e instanceof AppError;
}

export function isOperationalError(e: unknown): boolean {
  if (isAppError(e)) return e.isOperational;
  return false; // unknown errors are non-operational by default
}

---

TypeScript: Express Error Handler

import { Request, Response, NextFunction } from 'express';

export function errorHandler(
  err: unknown,
  req: Request,
  res: Response,
  next: NextFunction
) {
  const requestId = req.headers['x-request-id'] as string;

  if (err instanceof AppError) {
    // Expected operational error — log at warn, return structured response
    logger.warn('Operational error', {
      code: err.code,
      message: err.message,
      statusCode: err.statusCode,
      context: err.context,
      requestId,
      path: req.path,
      method: req.method,
    });

    return res.status(err.statusCode).json({
      error: {
        code: err.code,
        message: err.message,
        ...(err instanceof ValidationError && { fields: err.fields }),
        requestId,
      },
    });
  }

  // Unexpected error — log full stack, do not leak internals
  logger.error('Unexpected error', {
    error: err instanceof Error ? {
      message: err.message,
      stack: err.stack,
      name: err.name,
    } : String(err),
    requestId,
    path: req.path,
    method: req.method,
  });

  return res.status(500).json({
    error: {
      code: 'INTERNAL_ERROR',
      message: 'An unexpected error occurred. Please try again.',
      requestId,
    },
  });
}

---

TypeScript: Result Type with Utilities

// ============================================================================
// Result type — for operations where failure is an expected outcome
// ============================================================================

export type Result<T, E extends Error = AppError> =
  | { ok: true; value: T }
  | { ok: false; error: E };

export const ok = <T>(value: T): Result<T, never> => ({ ok: true, value });
export const err = <E extends Error>(error: E): Result<never, E> => ({ ok: false, error });

// Async variant
export type AsyncResult<T, E extends Error = AppError> = Promise<Result<T, E>>;

// Map over success value (like Array.map but for Result)
export function mapResult<T, U, E extends Error>(
  result: Result<T, E>,
  fn: (value: T) => U
): Result<U, E> {
  if (result.ok) return ok(fn(result.value));
  return result;
}

// Flat map for chaining Result-returning operations
export async function flatMapResult<T, U, E extends Error>(
  result: Result<T, E>,
  fn: (value: T) => Promise<Result<U, E>>
): Promise<Result<U, E>> {
  if (!result.ok) return result;
  return fn(result.value);
}

// Unwrap or throw (use at boundaries where you've already handled errors)
export function unwrap<T, E extends Error>(result: Result<T, E>): T {
  if (result.ok) return result.value;
  throw result.error;
}

// Convert exception-throwing function to Result-returning function
export async function tryCatch<T>(
  fn: () => Promise<T>,
  mapError?: (e: unknown) => AppError
): AsyncResult<T> {
  try {
    return ok(await fn());
  } catch (e) {
    const mapped = mapError ? mapError(e) : toAppError(e);
    return err(mapped);
  }
}

function toAppError(e: unknown): AppError {
  if (e instanceof AppError) return e;
  if (e instanceof Error) return new AppError(e.message, 'UNKNOWN', 500, false);
  return new AppError(String(e), 'UNKNOWN', 500, false);
}

---

Python: Full Application Error Hierarchy

from __future__ import annotations
from typing import Any, Optional
import logging

logger = logging.getLogger(__name__)


class AppError(Exception):
    """
    Base error for all application errors.

    is_operational=True  → expected failure; log and handle gracefully
    is_operational=False → programming error; crash and alert
    """

    def __init__(
        self,
        message: str,
        code: str,
        status: int = 500,
        is_operational: bool = True,
        context: Optional[dict[str, Any]] = None,
    ) -> None:
        super().__init__(message)
        self.code = code
        self.status = status
        self.is_operational = is_operational
        self.context = context or {}

    def to_dict(self) -> dict[str, Any]:
        return {
            "type": self.__class__.__name__,
            "code": self.code,
            "message": str(self),
            "status": self.status,
            "context": self.context,
        }

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}(code={self.code!r}, message={str(self)!r})"


# ============================================================================
# HTTP / API Errors
# ============================================================================

class BadRequestError(AppError):
    def __init__(self, message: str, context: Optional[dict] = None) -> None:
        super().__init__(message, "BAD_REQUEST", 400, context=context)


class ValidationError(AppError):
    def __init__(self, fields: dict[str, list[str]], message: str = "Validation failed") -> None:
        super().__init__(message, "VALIDATION_ERROR", 422)
        self.fields = fields

    def to_dict(self) -> dict[str, Any]:
        return {**super().to_dict(), "fields": self.fields}


class UnauthorizedError(AppError):
    def __init__(self, message: str = "Authentication required") -> None:
        super().__init__(message, "UNAUTHORIZED", 401)


class ForbiddenError(AppError):
    def __init__(self, message: str = "Insufficient permissions") -> None:
        super().__init__(message, "FORBIDDEN", 403)


class NotFoundError(AppError):
    def __init__(self, resource: str, identifier: Any = None) -> None:
        message = f"{resource} '{identifier}' not found" if identifier else f"{resource} not found"
        super().__init__(message, "NOT_FOUND", 404, context={"resource": resource, "identifier": identifier})


class RateLimitError(AppError):
    def __init__(self, retry_after_ms: int, message: str = "Rate limit exceeded") -> None:
        super().__init__(message, "RATE_LIMIT", 429, context={"retry_after_ms": retry_after_ms})
        self.retry_after_ms = retry_after_ms


# ============================================================================
# Infrastructure Errors
# ============================================================================

class DatabaseError(AppError):
    def __init__(self, operation: str, cause: Optional[Exception] = None) -> None:
        super().__init__(
            f"Database error during {operation}",
            "DATABASE_ERROR",
            503,
            context={"operation": operation},
        )
        if cause:
            self.__cause__ = cause  # preserves traceback chain


class NetworkError(AppError):
    def __init__(self, endpoint: str, cause: Optional[Exception] = None) -> None:
        super().__init__(
            f"Network error reaching {endpoint}",
            "NETWORK_ERROR",
            503,
            context={"endpoint": endpoint},
        )
        if cause:
            self.__cause__ = cause


class TimeoutError(AppError):
    def __init__(self, operation: str, timeout_ms: int) -> None:
        super().__init__(
            f"{operation} timed out after {timeout_ms}ms",
            "TIMEOUT",
            504,
            context={"operation": operation, "timeout_ms": timeout_ms},
        )
        self.timeout_ms = timeout_ms


# ============================================================================
# Python Exception Chaining Best Practices
# ============================================================================

# CORRECT: use 'from' to preserve the causal chain
def fetch_user(user_id: str) -> dict:
    try:
        return db.execute("SELECT * FROM users WHERE id = %s", [user_id]).fetchone()
    except psycopg2.OperationalError as e:
        raise DatabaseError("user_fetch", e) from e  # ← explicit chain

# WRONG: this hides the original exception context
def fetch_user_bad(user_id: str) -> dict:
    try:
        return db.execute("SELECT * FROM users WHERE id = %s", [user_id]).fetchone()
    except psycopg2.OperationalError:
        raise DatabaseError("user_fetch")  # ← original traceback lost

# When re-raising to suppress context (intentional suppression):
def parse_config(raw: str) -> dict:
    try:
        return json.loads(raw)
    except json.JSONDecodeError as e:
        raise ValidationError({"config": ["Invalid JSON format"]}) from None  # ← suppress internal

---

Python: FastAPI Error Handler

from fastapi import Request
from fastapi.responses import JSONResponse
import uuid


async def app_error_handler(request: Request, exc: AppError) -> JSONResponse:
    request_id = request.headers.get("x-request-id", str(uuid.uuid4()))

    log_data = {
        "code": exc.code,
        "message": str(exc),
        "status": exc.status,
        "context": exc.context,
        "request_id": request_id,
        "path": str(request.url.path),
        "method": request.method,
    }

    if exc.is_operational:
        logger.warning("Operational error", extra=log_data)
    else:
        logger.error("Non-operational error", extra={**log_data, "stack": True})

    body = {
        "error": {
            "code": exc.code,
            "message": str(exc),
            "request_id": request_id,
        }
    }

    if isinstance(exc, ValidationError):
        body["error"]["fields"] = exc.fields

    return JSONResponse(status_code=exc.status, content=body)


async def unhandled_error_handler(request: Request, exc: Exception) -> JSONResponse:
    request_id = request.headers.get("x-request-id", str(uuid.uuid4()))

    logger.exception(
        "Unhandled exception",
        extra={"request_id": request_id, "path": str(request.url.path)},
    )

    return JSONResponse(
        status_code=500,
        content={
            "error": {
                "code": "INTERNAL_ERROR",
                "message": "An unexpected error occurred. Please try again.",
                "request_id": request_id,
            }
        },
    )


# Register handlers
app.add_exception_handler(AppError, app_error_handler)
app.add_exception_handler(Exception, unhandled_error_handler)

---

Error Classification Helper

Use this at process exit to decide whether to crash:

// In the global uncaught exception handler
process.on('uncaughtException', (error: Error) => {
  logger.fatal('Uncaught exception', { error });

  if (!isOperationalError(error)) {
    // Non-operational: programmer error — crash so the process manager restarts cleanly
    process.exit(1);
  }
  // Operational: the error was logged, service can continue
  // (rare: usually operational errors are caught at the API boundary)
});

# Python equivalent in main entrypoint
import sys
import logging

def handle_exception(exc_type, exc_value, exc_traceback):
    if isinstance(exc_value, AppError) and exc_value.is_operational:
        logging.error("Unhandled operational error", exc_info=(exc_type, exc_value, exc_traceback))
        # Allow process to continue if possible
    else:
        logging.critical("Non-operational error — crashing", exc_info=(exc_type, exc_value, exc_traceback))
        sys.exit(1)

sys.excepthook = handle_exception

Retry Patterns Reference

Consult this file when implementing retry logic, backoff strategies, circuit breakers, or dead letter queue handling.

---

Exponential Backoff Formulas

Basic Exponential Backoff

delay = base * (multiplier ^ attempt)

• base: Starting delay (e.g., 100ms)
• multiplier: Growth factor (typically 2)
• attempt: Zero-indexed attempt number

Example: 100ms, 200ms, 400ms, 800ms, 1600ms...

Problem: All retrying clients synchronize on the same delay, causing thundering herd when a service recovers.

Full Jitter (Recommended for Most Cases)

delay = random(0, min(cap, base * (2 ^ attempt)))

Adds full randomization across the entire range. Produces the best throughput distribution when many clients retry simultaneously.

function retryDelay(attempt: number, base = 100, cap = 30_000): number {
  const exponential = Math.min(cap, base * Math.pow(2, attempt));
  return Math.random() * exponential; // full jitter
}

Equal Jitter

temp = min(cap, base * (2 ^ attempt))
delay = temp/2 + random(0, temp/2)

Guarantees a minimum wait (temp/2) while adding jitter. Useful when you want some delay guaranteed but still want distribution.

Decorrelated Jitter (AWS Recommendation)

delay = min(cap, random(base, prev_delay * 3))

Each retry is uncorrelated from the previous. Best when clients have different base delays or retry independently.

function* decorrelatedBackoff(base = 100, cap = 30_000) {
  let prev = base;
  while (true) {
    const next = Math.min(cap, base + Math.random() * (prev * 3 - base));
    prev = next;
    yield next;
  }
}

---

Retry Decision Matrix

---

Circuit Breaker Pattern

The circuit breaker prevents cascading failures by stopping requests to a failing dependency before it overwhelms or queues endlessly.

States

CLOSED ──(failure threshold exceeded)──► OPEN
  ▲                                        │
  │                                        │ (timeout elapsed)
  └──(probe succeeds)──── HALF-OPEN ◄──────┘
                               │
                      (probe fails)
                               │
                               ▼
                             OPEN (reset timeout)

Implementation (TypeScript)

type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

interface CircuitBreakerConfig {
  failureThreshold: number;     // failures before opening
  successThreshold: number;     // successes in HALF_OPEN before closing
  openTimeoutMs: number;        // how long to stay OPEN before probing
  requestTimeoutMs: number;     // individual request timeout
}

class CircuitBreaker<T> {
  private state: CircuitState = 'CLOSED';
  private failures = 0;
  private successes = 0;
  private openedAt: number | null = null;

  constructor(
    private fn: () => Promise<T>,
    private config: CircuitBreakerConfig
  ) {}

  async call(): Promise<T> {
    if (this.state === 'OPEN') {
      if (Date.now() - this.openedAt! < this.config.openTimeoutMs) {
        throw new CircuitOpenError('Circuit breaker is OPEN');
      }
      this.state = 'HALF_OPEN';
      this.successes = 0;
    }

    try {
      const result = await Promise.race([
        this.fn(),
        this.timeout(),
      ]);
      this.onSuccess();
      return result;
    } catch (e) {
      this.onFailure();
      throw e;
    }
  }

  private onSuccess() {
    this.failures = 0;
    if (this.state === 'HALF_OPEN') {
      this.successes++;
      if (this.successes >= this.config.successThreshold) {
        this.state = 'CLOSED';
      }
    }
  }

  private onFailure() {
    this.failures++;
    if (this.failures >= this.config.failureThreshold) {
      this.state = 'OPEN';
      this.openedAt = Date.now();
    }
    if (this.state === 'HALF_OPEN') {
      this.state = 'OPEN';
      this.openedAt = Date.now();
    }
  }

  private timeout(): Promise<never> {
    return new Promise((_, reject) =>
      setTimeout(() => reject(new TimeoutError('Request timed out')), this.config.requestTimeoutMs)
    );
  }
}

Configuration Guidelines

---

Dead Letter Queue (DLQ) Patterns

Messages that have exhausted retries go to a DLQ rather than being discarded.

When to Use DLQ

• Background job failed all retries (permanent or unknown failure)
• Message processing is non-idempotent and failed partway through
• You need auditability of all failures
• Human review or manual replay may be needed later

DLQ Message Schema

interface DLQMessage<T> {
  // Original message
  originalMessage: T;
  originalQueue: string;

  // Failure metadata
  failureReason: string;
  failureCode: string;
  lastAttemptAt: string;        // ISO 8601
  totalAttempts: number;
  errorStack?: string;          // sanitized — no secrets

  // Routing metadata
  messageId: string;
  correlationId: string;
  enqueuedAt: string;
  dlqEnqueuedAt: string;

  // Replay support
  replayable: boolean;          // false if side effects partially applied
  replayInstructions?: string;  // human-readable notes for operators
}

DLQ Strategies

Alarm on DLQ depth: Alert when DLQ grows beyond expected volume. Silence on DLQ growth = hidden failures accumulating.

Replay pipeline: Build a separate process to inspect DLQ messages and replay them to the original queue after root cause is fixed. Never replay blindly — check idempotency first.

DLQ TTL: Set expiration on DLQ messages (7-30 days). After expiry, log a final failure metric and discard. Indefinite DLQ retention causes storage bloat and operational debt.

Separate DLQs per severity: High-value failures (payment processing) → monitored DLQ with pager. Low-value (analytics events) → silent DLQ with daily review.

---

Python Retry Implementation

import asyncio
import random
import logging
from functools import wraps
from typing import TypeVar, Callable, Awaitable

T = TypeVar('T')
logger = logging.getLogger(__name__)


def with_retry(
    max_attempts: int = 3,
    base_delay: float = 0.1,       # seconds
    max_delay: float = 30.0,
    retryable_exceptions: tuple = (Exception,),
    jitter: bool = True,
):
    """Decorator for async functions with exponential backoff + full jitter."""
    def decorator(fn: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]:
        @wraps(fn)
        async def wrapper(*args, **kwargs) -> T:
            for attempt in range(max_attempts):
                try:
                    return await fn(*args, **kwargs)
                except retryable_exceptions as e:
                    if attempt == max_attempts - 1:
                        logger.error(
                            "Max retries exceeded",
                            extra={"function": fn.__name__, "attempts": attempt + 1, "error": str(e)}
                        )
                        raise

                    delay = min(max_delay, base_delay * (2 ** attempt))
                    if jitter:
                        delay = random.uniform(0, delay)

                    logger.warning(
                        "Retrying after error",
                        extra={"function": fn.__name__, "attempt": attempt + 1, "delay": delay, "error": str(e)}
                    )
                    await asyncio.sleep(delay)
            raise RuntimeError("Unreachable")  # type checker satisfaction
        return wrapper
    return decorator


# Usage
@with_retry(max_attempts=3, retryable_exceptions=(aiohttp.ClientError, asyncio.TimeoutError))
async def fetch_user(user_id: str) -> dict:
    async with aiohttp.ClientSession() as session:
        async with session.get(f"/api/users/{user_id}", timeout=aiohttp.ClientTimeout(total=5)) as resp:
            resp.raise_for_status()
            return await resp.json()

---

Idempotency and Retry Safety

Before retrying any operation, confirm it is idempotent or make it idempotent:

Naturally idempotent: GET, PUT (full replacement), DELETE (on already-deleted resource) Not idempotent by default: POST (creates new record), PATCH (incremental update), financial debits

Making POST idempotent: Idempotency keys. Send a client-generated UUID with every request. Server stores the key and returns the same response for duplicate requests within a TTL window.

// Client sends idempotency key
await fetch('/api/payments', {
  method: 'POST',
  headers: {
    'Idempotency-Key': crypto.randomUUID(),
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(paymentData),
});

// Server checks key in Redis before processing
const existing = await redis.get(`idempotency:${key}`);
if (existing) return JSON.parse(existing); // replay cached response

Store idempotency results for 24-48 hours. Use a key namespace that includes the operation type to prevent cross-operation collisions.