Api Error Handling

API Error Handling

Table of Contents

• Overview
• When to Use
• Quick Start
• Reference Guides
• Best Practices

Overview

Build robust error handling systems with standardized error responses, detailed logging, error categorization, and user-friendly error messages. This skill covers the full lifecycle from throwing typed errors through logging, monitoring, and client-facing response formatting.

When to Use

• Handling API errors consistently across endpoints
• Debugging production issues with request tracing
• Implementing error recovery strategies (retry, circuit breaker)
• Monitoring and alerting on error rates
• Providing meaningful, actionable error messages to clients
• Validating request inputs before processing
• Tracking error patterns over time

Quick Start

Minimal standardized error response format:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Input validation failed",
    "statusCode": 422,
    "requestId": "req_abc123xyz789",
    "timestamp": "2025-01-15T10:30:00Z",
    "details": [
      { "field": "email", "message": "Invalid email format", "code": "INVALID_EMAIL" }
    ]
  }
}

Custom error class (Node.js):

class ApiError extends Error {
  constructor(code, message, statusCode = null, details = null) {
    super(message);
    this.code = code;
    this.statusCode = statusCode || ERROR_CODES[code]?.status || 500;
    this.details = details;
    this.timestamp = new Date().toISOString();
  }
}

// Usage
throw new ApiError("NOT_FOUND", "User not found", 404);
throw new ApiError("VALIDATION_ERROR", "Missing fields", 422, fieldErrors);

Reference Guides

Detailed implementations in the references/ directory:

Best Practices

✅ DO

• Use a consistent error response format across all endpoints
• Include requestId and traceId in every error for observability
• Log 5xx errors at ERROR level; log 4xx at WARN level
• Provide actionable error messages — tell the client what to fix
• Use standard HTTP status codes (4xx client errors, 5xx server errors)
• Implement retry with exponential backoff for transient failures
• Use circuit breakers to prevent cascade failures
• Validate inputs early and return all field errors at once
• Monitor error rates and alert on anomalous spikes

❌ DON'T

• Expose stack traces or internal implementation details to clients
• Return HTTP 200 for error responses
• Silently swallow errors
• Log sensitive data (passwords, tokens, PII)
• Use vague messages like "Something went wrong"
• Mix error handling logic with business logic
• Retry non-idempotent operations or client errors (4xx)
• Return different error shapes from different endpoints

Error Codes & Response Format

ERROR_CODES Map

const ERROR_CODES = {
  VALIDATION_ERROR:    { status: 422, message: "Validation failed" },
  NOT_FOUND:           { status: 404, message: "Resource not found" },
  UNAUTHORIZED:        { status: 401, message: "Authentication required" },
  FORBIDDEN:           { status: 403, message: "Access denied" },
  CONFLICT:            { status: 409, message: "Resource conflict" },
  RATE_LIMITED:        { status: 429, message: "Too many requests" },
  INTERNAL_ERROR:      { status: 500, message: "Internal server error" },
  SERVICE_UNAVAILABLE: { status: 503, message: "Service unavailable" },
};

Global Error Middleware — Node.js (Express)

// Error response formatter
function formatErrorResponse(error, requestId, traceId) {
  return {
    error: {
      code: error.code,
      message: error.message,
      statusCode: error.statusCode,
      requestId,
      timestamp: error.timestamp,
      ...(error.details && { details: error.details }),
      traceId,
    },
  };
}

// Error logger (severity by status code)
function logError(error, context) {
  const logData = {
    timestamp: new Date().toISOString(),
    errorCode: error.code,
    errorMessage: error.message,
    statusCode: error.statusCode,
    stack: error.stack,
    context,
  };

  if (error.statusCode >= 500) {
    console.error("[ERROR]", JSON.stringify(logData));
    trackError(logData); // send to Sentry, etc.
  } else if (error.statusCode >= 400) {
    console.warn("[WARN]", JSON.stringify(logData));
  }
}

// Global error handler middleware
app.use((err, req, res, next) => {
  const requestId = req.id || `req_${Date.now()}`;
  const traceId = req.traceId;

  logError(err, {
    requestId, traceId,
    method: req.method, path: req.path,
    query: req.query, userId: req.user?.id,
  });

  if (err instanceof ApiError) {
    return res.status(err.statusCode).json(formatErrorResponse(err, requestId, traceId));
  }

  if (err instanceof SyntaxError && "body" in err) {
    const e = new ApiError("VALIDATION_ERROR", "Invalid JSON", 400);
    return res.status(400).json(formatErrorResponse(e, requestId, traceId));
  }

  if (err.name === "ValidationError") {
    const details = Object.keys(err.errors).map((field) => ({
      field, message: err.errors[field].message, code: "VALIDATION_FAILED",
    }));
    const e = new ApiError("VALIDATION_ERROR", "Validation failed", 422, details);
    return res.status(422).json(formatErrorResponse(e, requestId, traceId));
  }

  if (err.name === "CastError") {
    const e = new ApiError("NOT_FOUND", "Invalid resource ID", 404);
    return res.status(404).json(formatErrorResponse(e, requestId, traceId));
  }

  const e = new ApiError("INTERNAL_ERROR", "An unexpected error occurred", 500);
  res.status(500).json(formatErrorResponse(e, requestId, traceId));
});

// Async route wrapper — eliminates try/catch boilerplate
const asyncHandler = (fn) => (req, res, next) =>
  Promise.resolve(fn(req, res, next)).catch(next);

// Catch unhandled promise rejections
process.on("unhandledRejection", (reason) => {
  console.error("Unhandled Rejection:", reason);
  trackError({ type: "unhandledRejection", reason });
});

Global Error Handlers — Python (Flask)

from flask import Flask, jsonify, request
from datetime import datetime
import logging

app = Flask(__name__)
logger = logging.getLogger(__name__)

class APIError(Exception):
    def __init__(self, code, message, status_code=500, details=None):
        super().__init__()
        self.code = code
        self.message = message
        self.status_code = status_code
        self.details = details or []
        self.timestamp = datetime.utcnow().isoformat()

ERROR_CODES = {
    'VALIDATION_ERROR': 422,
    'NOT_FOUND': 404,
    'UNAUTHORIZED': 401,
    'FORBIDDEN': 403,
    'CONFLICT': 409,
    'INTERNAL_ERROR': 500,
}

def format_error(error, request_id, trace_id):
    return {
        'error': {
            'code': error.code,
            'message': error.message,
            'statusCode': error.status_code,
            'requestId': request_id,
            'timestamp': error.timestamp,
            'traceId': trace_id,
            'details': error.details or None,
        }
    }

def log_error(error, context):
    entry = {
        'timestamp': datetime.utcnow().isoformat(),
        'code': error.code,
        'message': error.message,
        'status': error.status_code,
        'context': context,
    }
    if error.status_code >= 500:
        logger.error(entry)
    elif error.status_code >= 400:
        logger.warning(entry)

@app.errorhandler(APIError)
def handle_api_error(error):
    request_id = request.headers.get('X-Request-ID', f'req_{int(datetime.utcnow().timestamp())}')
    trace_id = request.headers.get('X-Trace-ID')
    log_error(error, {'request_id': request_id, 'method': request.method, 'path': request.path})
    return jsonify(format_error(error, request_id, trace_id)), error.status_code

@app.errorhandler(400)
def handle_bad_request(error):
    e = APIError('VALIDATION_ERROR', 'Invalid request', 400)
    return jsonify(format_error(e, f'req_{int(datetime.utcnow().timestamp())}', None)), 400

@app.errorhandler(404)
def handle_not_found(error):
    e = APIError('NOT_FOUND', 'Resource not found', 404)
    return jsonify(format_error(e, f'req_{int(datetime.utcnow().timestamp())}', None)), 404

@app.errorhandler(500)
def handle_internal_error(error):
    logger.error(f'Internal error: {error}', exc_info=True)
    e = APIError('INTERNAL_ERROR', 'Internal server error', 500)
    return jsonify(format_error(e, f'req_{int(datetime.utcnow().timestamp())}', None)), 500

Monitoring & Tracking Patterns

Sentry Integration (Node.js)

import * as Sentry from "@sentry/node";

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 0.1,  // 10% of requests for performance tracking
});

// Capture structured error context
function trackError(error, context = {}) {
  Sentry.withScope((scope) => {
    scope.setTag("error.code", error.code);
    scope.setTag("error.status", error.statusCode);
    scope.setContext("request", {
      requestId: context.requestId,
      method: context.method,
      path: context.path,
      userId: context.userId,
    });
    Sentry.captureException(error);
  });
}

// Usage in global error middleware
if (error.statusCode >= 500) {
  trackError(error, { requestId, method: req.method, path: req.path, userId: req.user?.id });
}

Error Rate Metrics

Track error rates by code and status to detect spikes:

// In-memory error counter (production: use Redis or StatsD)
const errorMetrics = new Map();

function recordErrorMetric(code, statusCode) {
  const key = `${code}:${statusCode}`;
  const current = errorMetrics.get(key) || { count: 0, lastSeen: null };
  errorMetrics.set(key, {
    count: current.count + 1,
    lastSeen: new Date().toISOString(),
  });
}

// Call from global error handler
recordErrorMetric(err.code, err.statusCode);

// Expose /metrics/errors endpoint for dashboards
app.get("/metrics/errors", (req, res) => {
  const metrics = {};
  for (const [key, value] of errorMetrics.entries()) {
    metrics[key] = value;
  }
  res.json({
    timestamp: new Date().toISOString(),
    errors: metrics,
  });
});

StatsD / Datadog Integration

const StatsD = require("hot-shots");
const statsd = new StatsD({ host: "localhost", port: 8125 });

function recordErrorMetric(code, statusCode) {
  // Increment counter with tags for filtering
  statsd.increment("api.errors", 1, [`code:${code}`, `status:${statusCode}`]);

  // Track error rate by status family (4xx vs 5xx)
  const family = statusCode >= 500 ? "5xx" : "4xx";
  statsd.increment("api.errors.family", 1, [`family:${family}`]);
}

Alerting Strategy

Alert on rates, not raw counts, to avoid false positives from traffic spikes:

// Example: alert on 5xx rate over a rolling window
function checkErrorRateAlert(windowMs = 60_000) {
  const now = Date.now();
  const recent = recentErrors.filter((ts) => now - ts < windowMs);
  const recentTotal = recentRequests.filter((ts) => now - ts < windowMs).length;

  const rate = recentTotal > 0 ? recent.length / recentTotal : 0;

  if (rate > 0.05) {
    console.error(`[ALERT] 5xx error rate ${(rate * 100).toFixed(1)}% exceeds 5% threshold`);
    // notify PagerDuty / Slack / etc.
  }
}

Retry Strategies & Circuit Breaker

Exponential Backoff with Jitter

/**
 * Retry a function with exponential backoff.
 * Only retries on transient errors (5xx, network failures).
 * Never retries client errors (4xx) — they won't succeed on retry.
 */
async function retryWithBackoff(fn, options = {}) {
  const {
    maxRetries = 3,
    baseDelay = 1000,   // ms
    maxDelay = 30000,   // ms cap
    jitter = true,      // add randomness to avoid thundering herd
    retryOn = (err) => !err.statusCode || err.statusCode >= 500,
  } = options;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      const isLastAttempt = attempt === maxRetries - 1;
      const shouldRetry = retryOn(error);

      if (isLastAttempt || !shouldRetry) throw error;

      const exponential = Math.min(baseDelay * Math.pow(2, attempt), maxDelay);
      const delay = jitter
        ? exponential * (0.5 + Math.random() * 0.5)  // ±50% jitter
        : exponential;

      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

// Usage
const user = await retryWithBackoff(
  () => fetchUser(userId),
  { maxRetries: 3, baseDelay: 500 }
);

Circuit Breaker Pattern

Prevents cascade failures by short-circuiting calls to a failing dependency.

class CircuitBreaker {
  /**
   * States:
   *   CLOSED   — normal operation, requests pass through
   *   OPEN     — dependency is failing, requests short-circuit immediately
   *   HALF_OPEN — testing recovery, allows one probe request
   */
  constructor(options = {}) {
    this.failureThreshold = options.failureThreshold ?? 5;
    this.timeout = options.timeout ?? 60_000;     // ms before trying again
    this.successThreshold = options.successThreshold ?? 2;  // to close from HALF_OPEN

    this.failureCount = 0;
    this.successCount = 0;
    this.state = "CLOSED";
    this.nextAttempt = Date.now();
  }

  async execute(fn) {
    if (this.state === "OPEN") {
      if (Date.now() < this.nextAttempt) {
        throw new ApiError("SERVICE_UNAVAILABLE", "Circuit breaker is open", 503);
      }
      this.state = "HALF_OPEN";
    }

    try {
      const result = await fn();
      this._onSuccess();
      return result;
    } catch (error) {
      this._onFailure();
      throw error;
    }
  }

  _onSuccess() {
    this.failureCount = 0;
    if (this.state === "HALF_OPEN") {
      this.successCount++;
      if (this.successCount >= this.successThreshold) {
        this.state = "CLOSED";
        this.successCount = 0;
      }
    }
  }

  _onFailure() {
    this.failureCount++;
    this.successCount = 0;
    if (this.failureCount >= this.failureThreshold) {
      this.state = "OPEN";
      this.nextAttempt = Date.now() + this.timeout;
    }
  }

  get status() {
    return { state: this.state, failures: this.failureCount };
  }
}

// Usage — one breaker per downstream service
const paymentBreaker = new CircuitBreaker({ failureThreshold: 3, timeout: 30_000 });

app.post("/api/payments", asyncHandler(async (req, res) => {
  const result = await paymentBreaker.execute(() =>
    paymentService.charge(req.body)
  );
  res.json({ data: result });
}));

Combining Both Patterns

// Retry inside the circuit breaker for transient blips,
// but let the circuit trip on sustained failures.
async function resilientCall(fn) {
  return serviceBreaker.execute(() =>
    retryWithBackoff(fn, { maxRetries: 2, baseDelay: 200 })
  );
}

Validation Patterns

Input Validation (Node.js with Zod)

Validate at the boundary — before any business logic runs.

import { z } from "zod";

// Define schema with clear error messages
const CreateUserSchema = z.object({
  email:    z.string().email("Invalid email format"),
  name:     z.string().min(1, "Name is required").max(100),
  age:      z.number().int().min(0).max(150).optional(),
  role:     z.enum(["admin", "user", "viewer"]).default("user"),
});

// Validate and throw structured ApiError on failure
function validateInput(schema, data) {
  const result = schema.safeParse(data);
  if (!result.success) {
    const details = result.error.issues.map((issue) => ({
      field:   issue.path.join("."),
      message: issue.message,
      code:    issue.code.toUpperCase(),
    }));
    throw new ApiError("VALIDATION_ERROR", "Validation failed", 422, details);
  }
  return result.data;
}

// Route usage
app.post("/api/users", asyncHandler(async (req, res) => {
  const data = validateInput(CreateUserSchema, req.body);
  const user = await UserService.create(data);
  res.status(201).json({ data: user });
}));

Error response clients receive:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Validation failed",
    "statusCode": 422,
    "details": [
      { "field": "email",  "message": "Invalid email format", "code": "INVALID_STRING" },
      { "field": "name",   "message": "Name is required",     "code": "TOO_SMALL" }
    ]
  }
}

Schema Guards — Detecting Bad API Responses

Validate incoming responses from upstream services before trusting them:

const ExternalUserSchema = z.object({
  id:    z.string().uuid(),
  email: z.string().email(),
  name:  z.string(),
});

async function fetchExternalUser(userId) {
  const response = await fetch(`https://api.example.com/users/${userId}`);

  if (!response.ok) {
    throw new ApiError("SERVICE_UNAVAILABLE", "Upstream user service failed", 503);
  }

  const raw = await response.json();
  const result = ExternalUserSchema.safeParse(raw);

  if (!result.success) {
    // Log the schema violation for debugging — don't expose to client
    console.error("Upstream schema violation:", result.error.issues, { raw });
    throw new ApiError("INTERNAL_ERROR", "Unexpected response from user service", 500);
  }

  return result.data;
}

Validate Before Expensive Operations

Catch bad inputs before DB queries, file I/O, or API calls:

// ✅ Validate ID format before querying
app.get("/api/users/:id", asyncHandler(async (req, res) => {
  const id = req.params.id;

  // Guard: reject obviously invalid IDs before hitting the DB
  if (!/^[0-9a-f]{24}$/.test(id)) {
    throw new ApiError("NOT_FOUND", "User not found", 404);
    // Note: return 404, not 400 — don't reveal ID format details to clients
  }

  const user = await User.findById(id);
  if (!user) throw new ApiError("NOT_FOUND", "User not found", 404);

  res.json({ data: user });
}));

Python Input Validation (Pydantic)

from pydantic import BaseModel, EmailStr, validator
from typing import Optional, Literal

class CreateUserRequest(BaseModel):
    email: EmailStr
    name: str
    age: Optional[int] = None
    role: Literal["admin", "user", "viewer"] = "user"

    @validator("name")
    def name_must_not_be_empty(cls, v):
        if not v.strip():
            raise ValueError("Name must not be empty")
        return v.strip()

    @validator("age")
    def age_must_be_valid(cls, v):
        if v is not None and not (0 <= v <= 150):
            raise ValueError("Age must be between 0 and 150")
        return v

# In Flask route:
from flask import request, jsonify
from pydantic import ValidationError

@app.route("/api/users", methods=["POST"])
def create_user():
    try:
        data = CreateUserRequest(**request.json)
    except ValidationError as e:
        details = [
            {"field": ".".join(str(loc) for loc in err["loc"]),
             "message": err["msg"],
             "code": err["type"].upper()}
            for err in e.errors()
        ]
        raise APIError("VALIDATION_ERROR", "Validation failed", 422, details)

    user = UserService.create(data.dict())
    return jsonify({"data": user}), 201

Common Validation Anti-patterns

// ❌ DON'T — validate after the expensive operation
const user = await expensiveDbQuery(req.body.id);
if (!req.body.id) throw new Error("ID required");  // too late

// ❌ DON'T — return 200 with error in body
res.json({ success: false, error: "Invalid email" });  // breaks HTTP semantics

// ❌ DON'T — stop on first error (force client to fix one at a time)
if (!body.email) throw new ApiError(...);
if (!body.name)  throw new ApiError(...);

// ✅ DO — collect all field errors and return them together (see Zod example above)