Health Check

Health Check - Liveness, Readiness & Dependency Probes

Codifies the project's three-tier health probe architecture (shallow liveness, readiness, deep dependency checks), Docker container healthchecks, cron-based periodic monitoring, route discovery verification, and the comprehensive system health script. Patterns align with Kubernetes probe conventions even when running outside K8s.

Description

Codifies liveness, readiness, and deep dependency health endpoints for NodeJS-Starter-V1's Next.js and FastAPI services, covering three-tier probe architecture, Docker healthchecks, cron-based monitoring, route discovery verification, and the system health script.

---

When to Apply

Positive Triggers

• Adding new health check endpoints or probes
• Integrating new dependencies that need health verification
• Configuring Docker healthchecks for containers
• Setting up periodic health monitoring via cron
• Implementing startup, liveness, or readiness probes
• Adding service dependency checks to existing endpoints
• User mentions: "health check", "liveness", "readiness", "probe", "heartbeat", "service health", "dependency check"

Negative Triggers

• Collecting application metrics (use metrics-collector instead)
• Adding structured log statements (use structured-logging instead)
• Designing dashboard UI for health status (use dashboard-patterns instead)
• Implementing graceful shutdown (use graceful-shutdown when available)

Core Directives

The Three Laws of Health Checks

1. Three tiers, not one: Separate liveness (am I alive?), readiness (can I serve traffic?), and deep (are all dependencies healthy?). Never combine them. 2. Parallel dependency checks: Check all dependencies concurrently via Promise.all or asyncio.gather. Never check sequentially — a slow database should not delay the Redis check. 3. 503 for unhealthy: Return HTTP 200 for healthy/degraded, HTTP 503 for unhealthy. Load balancers and orchestrators use status codes, not response bodies.

---

Existing Project Infrastructure

Backend (FastAPI)

Frontend (Next.js)

Docker

System Script

scripts/health-check.ps1 — 6-phase comprehensive health check (prerequisites, database, backend, frontend, integration, summary) with exit code 0 (healthy) or 1 (unhealthy).

---

Health Status Model

All health endpoints use a three-state status:

Aggregation Rule

if any dependency is unhealthy → overall = unhealthy (503)
else if any dependency is degraded → overall = degraded (200)
else → overall = healthy (200)

---

Probe Patterns

Tier 1: Liveness (Shallow)

Returns immediately with minimal computation. Used by load balancers and orchestrators to confirm the process is alive.

Backend (/health):

@router.get("/health")
async def health_check() -> dict[str, str]:
    return {
        "status": "healthy",
        "timestamp": datetime.now().isoformat(),
        "version": "0.1.0",
    }

Frontend (/api/health):

interface HealthResponse {
  status: "healthy" | "degraded" | "unhealthy";
  timestamp: string;
  version: string;
  uptime: number;
  environment: string;
}

Rules: No database calls, no external service checks, no computation. Must respond in < 50ms.

Tier 2: Readiness

Confirms the service can accept and process requests. Checks that critical dependencies are reachable.

Backend (/ready):

@router.get("/ready")
async def readiness_check() -> dict[str, str]:
    # Check database connectivity
    # Check Redis connectivity
    # Check AI provider availability
    return {"status": "ready", "timestamp": datetime.now().isoformat()}

Rules: Check only fast, critical dependencies (database, cache). Timeout each check at 2–5 seconds. Do not check optional or slow services.

Tier 3: Deep Dependency Check

Checks all dependencies in parallel with latency measurement. Used for debugging and monitoring dashboards, not for load balancer probes (too slow).

Frontend (/api/health/deep):

interface DependencyCheck {
  name: string;
  status: "healthy" | "degraded" | "unhealthy" | "unchecked";
  latency_ms: number | null;
  error: string | null;
  last_checked: string;
}

Each dependency checker follows this pattern:

1. Record start time 2. Attempt operation with timeout (AbortSignal.timeout(5000)) 3. Measure latency (Date.now() - start) 4. Classify: healthy (success), degraded (slow or partial), unhealthy (error/timeout)

Checks run in parallel via Promise.all:

const [database, backend, verification] = await Promise.all([
  checkDatabase(),
  checkBackend(),
  checkVerificationSystem(),
]);

The summary aggregates results:

const summary = {
  total_checks: checks.length,
  passed: checks.filter(c => c.status === "healthy").length,
  failed: checks.filter(c => c.status === "unhealthy").length,
  degraded: checks.filter(c => c.status === "degraded").length,
};

---

Docker Healthcheck Pattern

Docker Compose healthchecks use CMD-SHELL with service-native commands:

services:
  postgres:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U starter_user -d starter_db"]
      interval: 10s
      timeout: 5s
      retries: 5
  redis:
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

For application containers, use curl or wget against the liveness endpoint:

backend:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 40s

start_period gives the application time to initialise before healthchecks begin. Use depends_on with condition: service_healthy to sequence container startup.

---

Cron-Based Monitoring

The project's /api/cron/health-check runs every 5 minutes, pings the backend, and logs results. Secured with CRON_SECRET bearer token.

Pattern for adding new periodic checks:

export async function GET(request: Request) {
  // 1. Verify CRON_SECRET
  const authHeader = request.headers.get("authorization");
  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {
    return new NextResponse("Unauthorized", { status: 401 });
  }
  // 2. Run checks with latency measurement
  const start = Date.now();
  const response = await fetch(`${backendUrl}/health`, {
    signal: AbortSignal.timeout(5000),
  });
  const latency = Date.now() - start;
  // 3. Log results
  logger.info("Health check cron", { backend: response.ok, latency });
  // 4. Alert if unhealthy
  if (!response.ok) { logger.error("Backend unhealthy"); }
  // 5. Return results
  return NextResponse.json({ status: response.ok ? "healthy" : "unhealthy" });
}

---

Route Health Verification

The /api/health/routes endpoint discovers all API routes by scanning the filesystem and optionally verifies each GET endpoint:

1. Discovery: Recursively scan app/api/ for route.ts files 2. Method detection: Parse file content for exported HTTP methods (GET, POST, PUT, PATCH, DELETE) 3. Verification (optional ?verify=true): Send GET request to each endpoint with 5-second timeout 4. Status: verified (200 OK), error (non-200 or timeout), unverified (not tested)

---

Adding a New Dependency Check

When integrating a new service, add a checker following this template:

async function checkNewService(): Promise<DependencyCheck> {
  const start = Date.now();
  const result: DependencyCheck = {
    name: "service_name",
    status: "unchecked",
    latency_ms: null,
    error: null,
    last_checked: new Date().toISOString(),
  };
  try {
    // Service-specific check (e.g., ping, SELECT 1, PING)
    const response = await fetch(serviceUrl, {
      signal: AbortSignal.timeout(5000),
    });
    result.latency_ms = Date.now() - start;
    result.status = response.ok ? "healthy" : "degraded";
    if (!response.ok) result.error = `HTTP ${response.status}`;
  } catch (e) {
    result.latency_ms = Date.now() - start;
    result.status = "unhealthy";
    result.error = e instanceof Error ? e.message : "Unknown error";
  }
  return result;
}

Then add it to the Promise.all array in the deep health endpoint.

---

Anti-Patterns

---

Checklist for New Health Endpoints

Structure

• [ ] Three-tier separation (liveness, readiness, deep)
• [ ] healthy / degraded / unhealthy status values
• [ ] HTTP 200 for healthy/degraded, 503 for unhealthy
• [ ] Response includes timestamp and version

Dependencies

• [ ] Parallel checking via Promise.all or asyncio.gather
• [ ] 5-second timeout per dependency check
• [ ] Latency measurement per dependency
• [ ] Graceful handling of missing environment variables

Docker

• [ ] Container healthcheck using service-native command
• [ ] interval, timeout, retries configured
• [ ] start_period covers application boot time
• [ ] condition: service_healthy for dependent services

Monitoring

• [ ] Cron-based periodic checks for production
• [ ] CRON_SECRET authentication on cron endpoints
• [ ] Health check latency instrumented via metrics-collector
• [ ] Failures logged via structured-logging

---

Response Format

[AGENT_ACTIVATED]: Health Check
[PHASE]: {Design | Implementation | Review}
[STATUS]: {in_progress | complete}

{health check analysis or implementation guidance}

[NEXT_ACTION]: {what to do next}

Integration Points

Metrics Collector

• health_check_duration_ms histogram per dependency
• health_check_status gauge (1=healthy, 0.5=degraded, 0=unhealthy)

Structured Logging

• Info-level health check results (dependency, status, latency)
• Error-level alerts when dependencies become unhealthy

Error Taxonomy

• SYS_HEALTH_DEPENDENCY_UNAVAILABLE (503) — critical dependency unreachable
• SYS_HEALTH_TIMEOUT (504) — dependency check exceeded timeout

Cron Scheduler

• /api/cron/health-check runs every 5 minutes with CRON_SECRET auth
• Results can trigger alerting via notification system

Dashboard Patterns

• StatusPulse component for live dependency status indicators
• DataStrip for health check latency metrics
• Connection status mapped to spectral colours (emerald=healthy, amber=degraded, red=unhealthy)

Australian Localisation (en-AU)

• Spelling: initialise, serialise, analyse, optimise, colour, behaviour
• Date: ISO 8601 in responses; DD/MM/YYYY in dashboard display
• Timezone: AEST/AEDT — timestamps stored as UTC

Health Check -- Before/After Examples

Concrete transformations from anti-patterns to three-tier health architecture.

---

Example 1: Single Endpoint to Three Tiers

Before

@router.get("/health")
async def health():
    """One endpoint that checks everything sequentially."""
    try:
        await db.execute(text("SELECT 1"))
        db_ok = True
    except Exception:
        db_ok = False

    try:
        await redis.ping()
        redis_ok = True
    except Exception:
        redis_ok = False

    all_ok = db_ok and redis_ok
    return {"status": "ok" if all_ok else "error", "database": db_ok, "redis": redis_ok}

Problems: Liveness check includes database (kills healthy process during DB maintenance). Sequential checks compound latency. Returns 200 even when unhealthy. Single tier conflates three concerns.

After

@router.get("/health")
async def liveness():
    """Tier 1: Process alive. No deps. < 50ms."""
    return {"status": "healthy", "timestamp": datetime.now().isoformat(), "version": "0.1.0"}


@router.get("/ready")
async def readiness():
    """Tier 2: Can serve traffic. Critical deps with timeout."""
    db, redis = await asyncio.gather(_check_database(), _check_redis())
    checks = [db, redis]
    any_unhealthy = any(c["status"] == "unhealthy" for c in checks)

    return JSONResponse(
        status_code=503 if any_unhealthy else 200,
        content={"status": "unhealthy" if any_unhealthy else "ready", "dependencies": checks},
    )


@router.get("/health/deep")
async def deep():
    """Tier 3: All deps in parallel. For dashboards and debugging."""
    checks = await asyncio.gather(
        _check_database(), _check_redis(), _check_ai_provider(),
    )
    any_unhealthy = any(c["status"] == "unhealthy" for c in checks)
    return JSONResponse(
        status_code=503 if any_unhealthy else 200,
        content={"status": "unhealthy" if any_unhealthy else "healthy", "dependencies": checks},
    )

---

Example 2: Sequential to Parallel

Before

async function deepHealth() {
  const db = await checkDatabase();       // 500ms
  const redis = await checkRedis();       // 200ms
  const backend = await checkBackend();   // 800ms
  // Total: 1500ms sequential
  return [db, redis, backend];
}

After

async function deepHealth() {
  const [db, redis, backend] = await Promise.all([
    checkDatabase(),    // 500ms
    checkRedis(),       // 200ms
    checkBackend(),     // 800ms
  ]);
  // Total: 800ms (max of all)
  return [db, redis, backend];
}

---

Example 3: 200 OK When Unhealthy to Correct Status Codes

Before

export async function GET() {
  const checks = await runAllChecks();
  // Always returns 200, even when database is down
  return NextResponse.json({ status: 'ok', checks });
}

After

export async function GET() {
  const checks = await runAllChecks();
  const anyUnhealthy = checks.some((c) => c.status === 'unhealthy');

  return NextResponse.json(
    {
      status: anyUnhealthy ? 'unhealthy' : 'healthy',
      checks,
    },
    { status: anyUnhealthy ? 503 : 200 },
  );
}

---

Example 4: No Timeout to Bounded Checks

Before

async function checkDatabase(): Promise<DependencyCheck> {
  const response = await fetch(dbUrl);  // May hang indefinitely
  return { name: 'database', status: response.ok ? 'healthy' : 'unhealthy' };
}

After

async function checkDatabase(): Promise<DependencyCheck> {
  const start = Date.now();
  try {
    const response = await fetch(dbUrl, {
      signal: AbortSignal.timeout(5000),  // 5-second hard limit
    });
    return {
      name: 'database',
      status: response.ok ? 'healthy' : 'degraded',
      latency_ms: Date.now() - start,
      error: response.ok ? null : `HTTP ${response.status}`,
    };
  } catch (e) {
    return {
      name: 'database',
      status: 'unhealthy',
      latency_ms: Date.now() - start,
      error: e instanceof Error ? e.message : 'Unknown error',
    };
  }
}

---

Example 5: Sensitive Details to Safe Responses

Before

{
  "database": {
    "error": "connection refused: postgresql://admin:s3cret@10.0.1.5:5432/mydb",
    "stack": "Traceback (most recent call last):\n  File \"/srv/app/..."
  }
}

After

{
  "database": {
    "name": "database",
    "status": "unhealthy",
    "latency_ms": null,
    "error": "Connection failed"
  }
}

Full error details logged server-side only:

logger.error("Database health check failed", error=str(exc), host=db_host)

Python Health Routes Template -- Generic

Framework-agnostic three-tier health endpoints. Adaptable to FastAPI, Flask, or any ASGI/WSGI framework.

---

Health Routes

import asyncio
import time
from datetime import datetime
from fastapi import APIRouter
from fastapi.responses import JSONResponse

router = APIRouter(tags=["health"])


@router.get("/health")
async def liveness() -> dict:
    """Tier 1: Liveness. No dependencies. < 50ms."""
    return {
        "status": "healthy",
        "timestamp": datetime.now().isoformat(),
        "version": "1.0.0",
    }


@router.get("/ready")
async def readiness() -> JSONResponse:
    """Tier 2: Readiness. Critical dependencies with timeout."""
    checks = await asyncio.gather(
        check_dependency("database", _ping_database),
        check_dependency("cache", _ping_cache),
    )
    any_unhealthy = any(c["status"] == "unhealthy" for c in checks)
    return JSONResponse(
        status_code=503 if any_unhealthy else 200,
        content={
            "status": "unhealthy" if any_unhealthy else "ready",
            "timestamp": datetime.now().isoformat(),
            "dependencies": checks,
        },
    )


@router.get("/health/deep")
async def deep() -> JSONResponse:
    """Tier 3: Deep check. All dependencies in parallel."""
    checks = await asyncio.gather(
        check_dependency("database", _ping_database),
        check_dependency("cache", _ping_cache),
        check_dependency("external_api", _ping_external),
    )
    any_unhealthy = any(c["status"] == "unhealthy" for c in checks)
    any_degraded = any(c["status"] == "degraded" for c in checks)
    overall = "unhealthy" if any_unhealthy else "degraded" if any_degraded else "healthy"

    return JSONResponse(
        status_code=503 if any_unhealthy else 200,
        content={
            "status": overall,
            "timestamp": datetime.now().isoformat(),
            "dependencies": checks,
            "summary": {
                "total": len(checks),
                "passed": sum(1 for c in checks if c["status"] == "healthy"),
                "failed": sum(1 for c in checks if c["status"] == "unhealthy"),
                "degraded": sum(1 for c in checks if c["status"] == "degraded"),
            },
        },
    )


async def check_dependency(name: str, check_fn) -> dict:
    """Run a dependency check with 5-second timeout and latency measurement."""
    start = time.perf_counter()
    try:
        await asyncio.wait_for(check_fn(), timeout=5.0)
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        return {"name": name, "status": "healthy", "latency_ms": latency_ms, "error": None}
    except asyncio.TimeoutError:
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        return {"name": name, "status": "unhealthy", "latency_ms": latency_ms, "error": "Timeout"}
    except Exception as e:
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        return {"name": name, "status": "unhealthy", "latency_ms": latency_ms, "error": str(e)}


async def _ping_database():
    """Replace with your database ping logic."""
    pass


async def _ping_cache():
    """Replace with your cache ping logic."""
    pass


async def _ping_external():
    """Replace with your external API ping logic."""
    pass

TypeScript Health Routes Template -- Generic

Framework-agnostic three-tier health endpoints. Adaptable to Next.js, Express, Fastify, or any Node.js framework.

---

Types

interface DependencyCheck {
  name: string;
  status: 'healthy' | 'degraded' | 'unhealthy' | 'unchecked';
  latency_ms: number | null;
  error: string | null;
  last_checked: string;
}

interface HealthResponse {
  status: 'healthy' | 'degraded' | 'unhealthy';
  timestamp: string;
  version: string;
}

interface DeepHealthResponse extends HealthResponse {
  dependencies: DependencyCheck[];
  summary: {
    total: number;
    passed: number;
    failed: number;
    degraded: number;
  };
}

---

Liveness (Tier 1)

export function livenessHandler(): HealthResponse {
  return {
    status: 'healthy',
    timestamp: new Date().toISOString(),
    version: process.env.APP_VERSION ?? '1.0.0',
  };
}

---

Deep Health (Tier 3)

export async function deepHealthHandler(): Promise<{
  response: DeepHealthResponse;
  statusCode: number;
}> {
  const checks = await Promise.all([
    checkDependency('database', DATABASE_URL),
    checkDependency('cache', CACHE_URL),
    checkDependency('external_api', EXTERNAL_API_URL),
  ]);

  const anyUnhealthy = checks.some((c) => c.status === 'unhealthy');
  const anyDegraded = checks.some((c) => c.status === 'degraded');
  const overall = anyUnhealthy ? 'unhealthy' : anyDegraded ? 'degraded' : 'healthy';

  return {
    statusCode: anyUnhealthy ? 503 : 200,
    response: {
      status: overall,
      timestamp: new Date().toISOString(),
      version: process.env.APP_VERSION ?? '1.0.0',
      dependencies: checks,
      summary: {
        total: checks.length,
        passed: checks.filter((c) => c.status === 'healthy').length,
        failed: checks.filter((c) => c.status === 'unhealthy').length,
        degraded: checks.filter((c) => c.status === 'degraded').length,
      },
    },
  };
}

async function checkDependency(name: string, url: string): Promise<DependencyCheck> {
  const start = Date.now();
  const result: DependencyCheck = {
    name,
    status: 'unchecked',
    latency_ms: null,
    error: null,
    last_checked: new Date().toISOString(),
  };

  try {
    const response = await fetch(url, {
      signal: AbortSignal.timeout(5000),
    });
    result.latency_ms = Date.now() - start;
    result.status = response.ok ? 'healthy' : 'degraded';
    if (!response.ok) result.error = `HTTP ${response.status}`;
  } catch (e) {
    result.latency_ms = Date.now() - start;
    result.status = 'unhealthy';
    result.error = e instanceof Error ? e.message : 'Unknown error';
  }

  return result;
}

FastAPI Health Routes Template -- Scientific Luxury

Three-tier health endpoints for the NodeJS-Starter-V1 FastAPI backend.

---

Health Routes

# apps/backend/src/api/routes/health.py
import asyncio
from datetime import datetime
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from src.utils import get_logger

logger = get_logger(__name__)
router = APIRouter(tags=["health"])


@router.get("/health")
async def liveness() -> dict:
    """Tier 1: Liveness probe. No dependencies. Must respond in < 50ms."""
    return {
        "status": "healthy",
        "timestamp": datetime.now().isoformat(),
        "version": "0.1.0",
    }


@router.get("/ready")
async def readiness() -> JSONResponse:
    """Tier 2: Readiness probe. Checks critical dependencies with timeout."""
    checks = await asyncio.gather(
        _check_database(),
        _check_redis(),
    )

    any_unhealthy = any(c["status"] == "unhealthy" for c in checks)
    status_code = 503 if any_unhealthy else 200
    overall = "unhealthy" if any_unhealthy else "ready"

    return JSONResponse(
        status_code=status_code,
        content={
            "status": overall,
            "timestamp": datetime.now().isoformat(),
            "dependencies": checks,
        },
    )


@router.get("/health/deep")
async def deep_health() -> JSONResponse:
    """Tier 3: Deep dependency check. All deps in parallel. For dashboards."""
    checks = await asyncio.gather(
        _check_database(),
        _check_redis(),
        _check_ai_provider(),
    )

    any_unhealthy = any(c["status"] == "unhealthy" for c in checks)
    any_degraded = any(c["status"] == "degraded" for c in checks)

    overall = "unhealthy" if any_unhealthy else "degraded" if any_degraded else "healthy"
    status_code = 503 if any_unhealthy else 200

    summary = {
        "total_checks": len(checks),
        "passed": sum(1 for c in checks if c["status"] == "healthy"),
        "failed": sum(1 for c in checks if c["status"] == "unhealthy"),
        "degraded": sum(1 for c in checks if c["status"] == "degraded"),
    }

    return JSONResponse(
        status_code=status_code,
        content={
            "status": overall,
            "timestamp": datetime.now().isoformat(),
            "dependencies": checks,
            "summary": summary,
        },
    )


async def _check_database() -> dict:
    """Check database connectivity with 5-second timeout."""
    import time
    start = time.perf_counter()
    try:
        from src.db.session import async_session
        async with async_session() as session:
            await asyncio.wait_for(
                session.execute(text("SELECT 1")),
                timeout=5.0,
            )
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        return {
            "name": "database",
            "status": "healthy",
            "latency_ms": latency_ms,
            "error": None,
        }
    except Exception as e:
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        logger.error("Database health check failed", error=str(e))
        return {
            "name": "database",
            "status": "unhealthy",
            "latency_ms": latency_ms,
            "error": "Connection failed",
        }


async def _check_redis() -> dict:
    """Check Redis connectivity with 5-second timeout."""
    import time
    start = time.perf_counter()
    try:
        from src.cache import redis_client
        await asyncio.wait_for(redis_client.ping(), timeout=5.0)
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        return {
            "name": "redis",
            "status": "healthy",
            "latency_ms": latency_ms,
            "error": None,
        }
    except Exception as e:
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        logger.error("Redis health check failed", error=str(e))
        return {
            "name": "redis",
            "status": "unhealthy",
            "latency_ms": latency_ms,
            "error": "Connection failed",
        }


async def _check_ai_provider() -> dict:
    """Check AI provider availability with 5-second timeout."""
    import time
    start = time.perf_counter()
    try:
        # Lightweight check -- ping or model list endpoint
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        return {
            "name": "ai_provider",
            "status": "healthy",
            "latency_ms": latency_ms,
            "error": None,
        }
    except Exception as e:
        latency_ms = round((time.perf_counter() - start) * 1000, 2)
        logger.error("AI provider health check failed", error=str(e))
        return {
            "name": "ai_provider",
            "status": "unhealthy",
            "latency_ms": latency_ms,
            "error": "Provider unavailable",
        }

Next.js Health Routes Template -- Scientific Luxury

Three-tier health endpoints for the NodeJS-Starter-V1 Next.js frontend.

---

Tier 1: Liveness

// apps/web/app/api/health/route.ts
import { NextResponse } from 'next/server';

interface HealthResponse {
  status: 'healthy' | 'degraded' | 'unhealthy';
  timestamp: string;
  version: string;
  uptime: number;
  environment: string;
}

const startTime = Date.now();

export async function GET() {
  const response: HealthResponse = {
    status: 'healthy',
    timestamp: new Date().toISOString(),
    version: process.env.APP_VERSION ?? '0.1.0',
    uptime: Math.floor((Date.now() - startTime) / 1000),
    environment: process.env.NODE_ENV ?? 'development',
  };

  return NextResponse.json(response);
}

---

Tier 3: Deep Dependency Check

// apps/web/app/api/health/deep/route.ts
import { NextResponse } from 'next/server';

interface DependencyCheck {
  name: string;
  status: 'healthy' | 'degraded' | 'unhealthy' | 'unchecked';
  latency_ms: number | null;
  error: string | null;
  last_checked: string;
}

export async function GET() {
  const [database, backend, verification] = await Promise.all([
    checkDatabase(),
    checkBackend(),
    checkVerificationSystem(),
  ]);

  const checks = [database, backend, verification];
  const anyUnhealthy = checks.some((c) => c.status === 'unhealthy');
  const anyDegraded = checks.some((c) => c.status === 'degraded');

  const overall = anyUnhealthy ? 'unhealthy' : anyDegraded ? 'degraded' : 'healthy';

  const summary = {
    total_checks: checks.length,
    passed: checks.filter((c) => c.status === 'healthy').length,
    failed: checks.filter((c) => c.status === 'unhealthy').length,
    degraded: checks.filter((c) => c.status === 'degraded').length,
  };

  return NextResponse.json(
    { status: overall, timestamp: new Date().toISOString(), dependencies: checks, summary },
    { status: anyUnhealthy ? 503 : 200 },
  );
}

async function checkDatabase(): Promise<DependencyCheck> {
  return checkDependency('database', `${process.env.BACKEND_URL}/ready`);
}

async function checkBackend(): Promise<DependencyCheck> {
  return checkDependency('backend_api', `${process.env.BACKEND_URL}/health`);
}

async function checkVerificationSystem(): Promise<DependencyCheck> {
  return checkDependency('verification', `${process.env.BACKEND_URL}/api/agents/verifier/health`);
}

async function checkDependency(name: string, url: string): Promise<DependencyCheck> {
  const start = Date.now();
  const result: DependencyCheck = {
    name,
    status: 'unchecked',
    latency_ms: null,
    error: null,
    last_checked: new Date().toISOString(),
  };

  try {
    const response = await fetch(url, {
      signal: AbortSignal.timeout(5000),
    });
    result.latency_ms = Date.now() - start;
    result.status = response.ok ? 'healthy' : 'degraded';
    if (!response.ok) result.error = `HTTP ${response.status}`;
  } catch (e) {
    result.latency_ms = Date.now() - start;
    result.status = 'unhealthy';
    result.error = e instanceof Error ? e.message : 'Unknown error';
  }

  return result;
}

---

Cron Health Check

// apps/web/app/api/cron/health-check/route.ts
import { NextResponse } from 'next/server';
import { logger } from '@/lib/logger';

export async function GET(request: Request) {
  // 1. Verify CRON_SECRET
  const authHeader = request.headers.get('authorization');
  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {
    return new NextResponse('Unauthorised', { status: 401 });
  }

  // 2. Check backend with latency measurement
  const start = Date.now();
  try {
    const response = await fetch(`${process.env.BACKEND_URL}/health`, {
      signal: AbortSignal.timeout(5000),
    });
    const latency = Date.now() - start;

    logger.info('Health check cron', { backend: response.ok, latency });

    if (!response.ok) {
      logger.error('Backend unhealthy', { status: response.status, latency });
    }

    return NextResponse.json({
      status: response.ok ? 'healthy' : 'unhealthy',
      latency,
    });
  } catch (e) {
    const latency = Date.now() - start;
    logger.error('Backend unreachable', { error: e instanceof Error ? e.message : 'Unknown', latency });
    return NextResponse.json({ status: 'unhealthy', latency }, { status: 503 });
  }
}

Health Check -- Anti-Patterns

Banned patterns extracted from the health-check skill. Every violation causes false positives, traffic routing to broken instances, or blocked health responses.

---

AP-1: Single-Tier Health Check

Severity: High -- conflates liveness, readiness, and dependency checks into one endpoint.

# BANNED: One endpoint that does everything
@router.get("/health")
async def health():
    db_ok = await check_database()
    redis_ok = await check_redis()
    ai_ok = await check_ai_provider()
    return {"status": "ok" if all([db_ok, redis_ok, ai_ok]) else "error"}

# CORRECT: Three separate tiers
@router.get("/health")
async def liveness():
    """Tier 1: Process alive. No dependencies. < 50ms."""
    return {"status": "healthy", "timestamp": datetime.now().isoformat()}

@router.get("/ready")
async def readiness():
    """Tier 2: Can serve traffic. Critical deps only. 2-5s timeout."""
    db_ok = await check_database(timeout=5)
    return {"status": "ready" if db_ok else "unhealthy"}

@router.get("/health/deep")
async def deep():
    """Tier 3: All dependencies in parallel. For debugging/dashboards."""
    results = await asyncio.gather(check_database(), check_redis(), check_ai())
    return {"dependencies": results}

Why it fails: Load balancers need a fast liveness check (< 50ms). If liveness includes a slow database check, a healthy process gets killed because the probe times out. Kubernetes restarts pods unnecessarily. Separate tiers let orchestrators make correct decisions.

---

AP-2: Sequential Dependency Checks

Severity: High -- total latency equals sum of all checks.

# BANNED: Sequential checks
async def deep_health():
    db = await check_database()      # 500ms
    redis = await check_redis()      # 200ms
    ai = await check_ai_provider()   # 1000ms
    # Total: 1700ms
    return [db, redis, ai]

# CORRECT: Parallel checks
async def deep_health():
    db, redis, ai = await asyncio.gather(
        check_database(),      # 500ms
        check_redis(),         # 200ms
        check_ai_provider(),   # 1000ms
    )
    # Total: 1000ms (max of all)
    return [db, redis, ai]

// CORRECT (frontend)
const [database, backend, verification] = await Promise.all([
  checkDatabase(),
  checkBackend(),
  checkVerificationSystem(),
]);

Why it fails: A slow database check should not delay the Redis check. Sequential checks compound latency. With 5 dependencies at 1 second each, sequential takes 5 seconds while parallel takes 1 second.

---

AP-3: HTTP 200 When Unhealthy

Severity: Critical -- load balancers route traffic to broken instances.

# BANNED: 200 OK with unhealthy status in body
@router.get("/health")
async def health():
    db_ok = await check_database()
    return {"status": "unhealthy", "database": "down"}  # HTTP 200!

# CORRECT: 503 for unhealthy
@router.get("/health/deep")
async def deep_health():
    checks = await run_all_checks()
    any_unhealthy = any(c.status == "unhealthy" for c in checks)

    status_code = 503 if any_unhealthy else 200
    return JSONResponse(
        status_code=status_code,
        content={"status": "unhealthy" if any_unhealthy else "healthy", "checks": checks},
    )

Why it fails: Load balancers and Kubernetes read HTTP status codes, not response bodies. A 200 with "status": "unhealthy" tells the load balancer the instance is healthy. Traffic continues flowing to a broken instance.

---

AP-4: No Timeout on Dependency Checks

Severity: High -- a single hung dependency blocks the entire health response.

// BANNED: No timeout
async function checkDatabase(): Promise<DependencyCheck> {
  const response = await fetch(dbUrl);  // May hang forever
  return { status: response.ok ? 'healthy' : 'unhealthy' };
}

// CORRECT: 5-second timeout per check
async function checkDatabase(): Promise<DependencyCheck> {
  const start = Date.now();
  try {
    const response = await fetch(dbUrl, {
      signal: AbortSignal.timeout(5000),
    });
    return {
      status: response.ok ? 'healthy' : 'degraded',
      latency_ms: Date.now() - start,
    };
  } catch (e) {
    return {
      status: 'unhealthy',
      latency_ms: Date.now() - start,
      error: e instanceof Error ? e.message : 'Unknown error',
    };
  }
}

Why it fails: A database connection that hangs (TCP half-open, firewall drop) causes the health endpoint to never respond. The load balancer eventually marks the instance as unhealthy, but only after its own timeout (typically 30 seconds), during which time the instance serves no traffic at all.

---

AP-5: Sensitive Details in Health Response

Severity: Medium -- leaks internal architecture to anyone who hits the endpoint.

// BANNED: Internal error details exposed
{
  "status": "unhealthy",
  "database": {
    "error": "connection refused: postgresql://admin:s3cret@10.0.1.5:5432/mydb",
    "stack": "Traceback (most recent call last):\n  File \"/srv/app/..."
  }
}

// CORRECT: Status and latency only
{
  "status": "unhealthy",
  "database": {
    "status": "unhealthy",
    "latency_ms": null,
    "error": "Connection failed"
  }
}

Why it fails: Health endpoints are often unauthenticated (for load balancer access). Internal error messages reveal database hostnames, credentials in connection strings, file paths, and library versions. Log the full error server-side; return only the status classification.

---

AP-6: Database Query in Liveness Probe

Severity: High -- kills healthy processes when the database is slow.

# BANNED: DB check in liveness
@router.get("/health")
async def liveness():
    await db.execute(text("SELECT 1"))  # Fails if DB is slow
    return {"status": "healthy"}

# CORRECT: Liveness = process alive only
@router.get("/health")
async def liveness():
    return {
        "status": "healthy",
        "timestamp": datetime.now().isoformat(),
        "version": "0.1.0",
    }

Why it fails: Liveness probes answer "is the process alive?" -- not "are dependencies available?" If the database is slow (maintenance, high load), the liveness probe fails, and Kubernetes restarts the pod. Restarting does not fix the database. The pod enters a restart loop. Database checks belong in the readiness probe.

Health Check -- Generic Standards

Portable health check standards applicable to any project. Framework-agnostic, design-system-agnostic. Aligns with Kubernetes probe conventions.

---

Principle

Health checks must be separated into three tiers: liveness (is the process alive?), readiness (can it serve traffic?), and deep (are all dependencies healthy?). Never combine them into a single endpoint.

---

Three-Tier Architecture

---

Status Model

Aggregation

• Any unhealthy -> overall unhealthy (503)
• Any degraded -> overall degraded (200)
• All healthy -> overall healthy (200)

---

Dependency Check Pattern

For each dependency:

1. Record start time 2. Attempt operation with a timeout (5 seconds max) 3. Measure latency 4. Classify result: healthy, degraded, or unhealthy 5. Return structured result with name, status, latency, and error

Run all checks in parallel (Promise.all / asyncio.gather).

---

Response Format

Liveness

{
  "status": "healthy",
  "timestamp": "2026-02-13T09:30:00.000Z",
  "version": "1.0.0"
}

Deep

{
  "status": "healthy",
  "dependencies": [
    { "name": "database", "status": "healthy", "latency_ms": 12, "error": null },
    { "name": "cache", "status": "healthy", "latency_ms": 3, "error": null }
  ],
  "summary": { "total": 2, "passed": 2, "failed": 0, "degraded": 0 }
}

---

Docker Healthcheck Pattern

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:PORT/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s

• Use service-native commands where possible (e.g., pg_isready, redis-cli ping)
• Set start_period to cover application boot time
• Use depends_on with condition: service_healthy for startup ordering

---

Checklist

• [ ] Three separate endpoints: liveness, readiness, deep
• [ ] Liveness has zero dependency checks and responds in < 50ms
• [ ] Readiness checks only critical dependencies with timeouts
• [ ] Deep checks all dependencies in parallel
• [ ] HTTP 503 for unhealthy, 200 for healthy/degraded
• [ ] Latency measured per dependency
• [ ] No sensitive details (credentials, internal paths) in responses
• [ ] Docker healthchecks configured with interval, timeout, retries, and start_period

Health Check -- Scientific Luxury Standards

Domain-specific standards for the three-tier health probe architecture in NodeJS-Starter-V1. Liveness < 50ms, readiness 2-5s, deep checks all dependencies in parallel.

---

Three-Tier Architecture

---

Health Status Model

All health endpoints use a three-state status:

Aggregation Rule

if any dependency is unhealthy -> overall = unhealthy (503)
else if any dependency is degraded -> overall = degraded (200)
else -> overall = healthy (200)

---

Existing Endpoint Inventory

Backend (FastAPI)

Frontend (Next.js)

---

Dependency Check Interface

interface DependencyCheck {
  name: string;
  status: 'healthy' | 'degraded' | 'unhealthy' | 'unchecked';
  latency_ms: number | null;
  error: string | null;
  last_checked: string;
}

Check Pattern

1. Record start time 2. Attempt operation with timeout (AbortSignal.timeout(5000)) 3. Measure latency (Date.now() - start) 4. Classify: healthy (success), degraded (slow or partial), unhealthy (error/timeout)

---

Liveness Response Format

{
  "status": "healthy",
  "timestamp": "2026-02-13T09:30:00.000Z",
  "version": "0.1.0"
}

Rules: No database calls, no external service checks, no computation. Must respond in < 50ms.

---

Deep Health Response Format

{
  "status": "healthy",
  "timestamp": "2026-02-13T09:30:00.000Z",
  "dependencies": [
    { "name": "database", "status": "healthy", "latency_ms": 12, "error": null },
    { "name": "redis", "status": "healthy", "latency_ms": 3, "error": null },
    { "name": "backend_api", "status": "healthy", "latency_ms": 45, "error": null }
  ],
  "summary": {
    "total_checks": 3,
    "passed": 3,
    "failed": 0,
    "degraded": 0
  }
}

---

Docker Healthcheck Standards

start_period covers application boot time. Use depends_on with condition: service_healthy to sequence container startup.

---

Cron-Based Monitoring

• Endpoint: /api/cron/health-check
• Frequency: Every 5 minutes
• Authentication: CRON_SECRET bearer token
• Actions: Ping backend, measure latency, log results, alert on failure

---

Dashboard Colour Mapping

---

Error Taxonomy Integration