Nginx C Module Debug

nginx.org C Module Debugging Best Practices

Comprehensive debugging guide for nginx C modules, derived from the official nginx development documentation and production debugging experience. Contains 45 rules across 8 categories, prioritized by impact to guide systematic diagnosis of crashes, memory bugs, and behavioral issues in nginx modules.

Companion skills: This skill complements nginx-c-modules (correctness) and nginx-c-module-perf-reliability (performance). This skill covers debugging and diagnosis.

When to Apply

Reference these guidelines when:

• Diagnosing nginx worker crashes (segfaults, SIGABRT, SIGSEGV)
• Finding memory bugs (use-after-free, leaks, pool corruption, buffer overruns)
• Setting up GDB and core dump analysis for nginx
• Tracing request flow through phases, subrequests, and filter chains
• Instrumenting nginx modules with debug logging and dynamic tracing tools

Rule Categories by Priority

Quick Reference

1. Crash Diagnosis & Signals (CRITICAL)

• `crash-segfault-signature` - Identify Segfault Crash Signature from Signal and Address
• `crash-null-deref-pattern` - Recognize NULL Pointer Dereference Patterns in nginx Modules
• `crash-double-free-finalize` - Diagnose Double Finalize Crashes from Request Reference Count
• `crash-stack-overflow` - Detect Stack Overflow from Recursive Subrequest or Filter Chains
• `crash-worker-exit-log` - Extract Crash Context from Worker Exit Log Messages
• `crash-error-page-redirect` - Avoid Crashes from error_page Internal Redirect Context Invalidation

2. Memory Bug Detection (CRITICAL)

• `memdbg-use-after-free` - Detect Use-After-Free from Pool Destruction Timing
• `memdbg-pool-leak-pattern` - Identify Pool Memory Leak Patterns from Growing Worker RSS
• `memdbg-slab-corruption` - Diagnose Shared Memory Slab Corruption from Multi-Worker Crashes
• `memdbg-cleanup-handler-leak` - Detect Resource Leaks from Missing Pool Cleanup Handlers
• `memdbg-buffer-overrun` - Find Buffer Overrun from ngx_pnalloc Size Miscalculation
• `memdbg-temp-pool-misuse` - Avoid Storing Long-Lived Pointers in Temporary Pools
• `memdbg-valgrind-pool-trace` - Use Valgrind Pool-Level Tracing to Find Leaked Allocations

3. GDB & Core Dump Analysis (HIGH)

• `gdb-coredump-setup` - Configure Core Dump Generation for nginx Worker Crashes
• `gdb-attach-worker` - Attach GDB to a Running nginx Worker Process
• `gdb-backtrace-read` - Read nginx Backtrace to Identify Crash Module and Phase
• `gdb-inspect-request` - Inspect ngx_http_request_t Fields in GDB for Request State
• `gdb-memory-buffer-extract` - Extract Debug Log from Memory Buffer Using GDB Script
• `gdb-watchpoint-corruption` - Use GDB Watchpoints to Catch Memory Corruption at Write Time

4. Request Flow Tracing (HIGH)

• `trace-phase-handler-flow` - Trace Request Through HTTP Phase Handlers
• `trace-subrequest-tree` - Map Subrequest Parent-Child Relationships for Debugging
• `trace-filter-chain-order` - Trace Filter Chain Execution Order and Data Flow
• `trace-upstream-callback-seq` - Trace Upstream Callback Sequence for Proxy Debugging
• `trace-event-handler-chain` - Trace Event Handler Execution for Connection Debugging
• `trace-config-inheritance` - Trace Configuration Inheritance Through Server and Location Blocks

5. Debug Logging Patterns (MEDIUM-HIGH)

• `dbglog-debug-mask` - Use Correct Debug Mask for Targeted Log Filtering
• `dbglog-debug-connection` - Use debug_connection to Isolate Single-Client Debug Output
• `dbglog-memory-buffer` - Use Memory Buffer Logging to Capture Debug Output Without Disk I/O
• `dbglog-log-action-string` - Set Log Action String for Context in Error Messages
• `dbglog-format-ngx-str` - Format ngx_str_t Correctly in Debug Log Messages

6. State & Lifecycle Debugging (MEDIUM)

• `state-connection-lifecycle` - Track Connection State Transitions for Lifecycle Debugging
• `state-upstream-state-machine` - Debug Upstream Module State by Logging Transition Points
• `state-timer-leak` - Detect Timer Leaks from Events Not Removed Before Pool Destruction
• `state-event-flag-debug` - Inspect Event Flags to Debug Unexpected Handler Invocation
• `state-request-count-track` - Track Request Reference Count to Debug Premature Destruction

7. Dynamic Tracing Tools (MEDIUM)

• `probe-strace-syscall` - Use strace to Trace System Call Patterns in nginx Workers
• `probe-dtrace-request` - Trace Request Processing with DTrace pid Provider
• `probe-systemtap-pool` - Trace Memory Pool Allocations with SystemTap
• `probe-ebpf-latency` - Measure Per-Function Latency with eBPF Probes
• `probe-strace-fd-leak` - Detect File Descriptor Leaks with strace and /proc

8. Build & Sanitizer Configuration (LOW-MEDIUM)

• `build-debug-flags` - Compile nginx with Full Debug Symbols and No Optimization
• `build-asan-configure` - Build nginx with AddressSanitizer for Memory Error Detection
• `build-single-process` - Use Single-Process Mode for Simplified Debugging
• `build-valgrind-suppressions` - Use nginx Valgrind Suppressions to Reduce False Positives
• `build-debug-palloc` - Enable NGX_DEBUG_PALLOC for Fine-Grained Pool Allocation Tracking

How to Use

Read individual reference files for detailed explanations and code examples:

• Section definitions - Category structure and impact levels
• Rule template - Template for adding new rules

Reference Files

Rule Title Here

1-3 sentences explaining WHY this matters. Focus on the consequences of missing or misusing this debugging technique in nginx module development.

Incorrect (what's wrong):

/* code example showing the anti-pattern or wrong approach */
/* comments explain the cost/consequence */

Correct (what's right):

/* code example showing the correct debugging approach */
/* comments explain the benefit */

Note: Optional additional context, exceptions, or related patterns.

Reference: Reference Title

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

---

1. Crash Diagnosis & Signals (crash)

Impact: CRITICAL Description: Segfaults and worker crashes are the most urgent debugging scenario — identifying the crash signature and signal type determines the entire debugging strategy.

2. Memory Bug Detection (memdbg)

Impact: CRITICAL Description: Use-after-free, pool corruption, and leaks are the #1 root cause of nginx module bugs — pool-allocated memory disappears silently when the wrong pool is destroyed.

3. GDB & Core Dump Analysis (gdb)

Impact: HIGH Description: GDB with coredumps is the primary post-mortem tool — correct setup and nginx-specific inspection commands cut diagnosis time from hours to minutes.

4. Request Flow Tracing (trace)

Impact: HIGH Description: Wrong-behavior bugs require tracing a request through phases, subrequests, and filter chains — missing a callback or phase transition hides the root cause.

5. Debug Logging Patterns (dbglog)

Impact: MEDIUM-HIGH Description: Structured debug logging with ngx_log_debug macros and debug_connection filtering produces targeted diagnostics without drowning in noise.

6. State & Lifecycle Debugging (state)

Impact: MEDIUM Description: Connection state machines, upstream callback sequences, and event handler transitions have strict ordering — state mismatches cause intermittent failures that resist reproduction.

7. Dynamic Tracing Tools (probe)

Impact: MEDIUM Description: DTrace, SystemTap, and strace enable production debugging without recompilation — tracing live worker processes reveals timing-dependent bugs invisible in test environments.

8. Build & Sanitizer Configuration (build)

Impact: LOW-MEDIUM Description: Correct --with-debug flags, AddressSanitizer, and Valgrind configuration catches memory errors at development time before they become production crashes.

Build nginx with AddressSanitizer for Memory Error Detection

AddressSanitizer (ASan) instruments every memory access at compile time, catching use-after-free, buffer overflows, and stack overflows immediately when they occur -- not later when corrupted data is read. Build nginx with -fsanitize=address in both CFLAGS and LDFLAGS. ASan prints a detailed report with the exact allocation and free call stacks. Overhead is ~2x which is acceptable for development.

Incorrect (building with ASan in CFLAGS only, linker fails or runtime missing):

/*
 * Partial ASan build — only sets CFLAGS, omits LDFLAGS.
 *
 * ./configure \
 *     --with-debug \
 *     --with-cc-opt='-fsanitize=address -g'
 * make -j$(nproc)
 *
 * Possible failures:
 *   1. Linker error: undefined reference to '__asan_report_load8'
 *      (libasan not linked)
 *   2. If it links by accident (distro auto-links), nginx forks
 *      workers that inherit the ASan state incorrectly, causing
 *      false positives on fork.
 *   3. Missing -fno-omit-frame-pointer produces truncated stack
 *      traces in ASan reports — you see the crash but not the
 *      full call chain leading to the allocation.
 */

/*
 * ASan report with missing frame pointers (truncated):
 *
 * ==4827==ERROR: AddressSanitizer: heap-use-after-free
 *   READ of size 8 at 0x60b000001a40 thread T0
 *     #0 0x4a3f20 in ngx_http_mymodule_handler (nginx+0x4a3f20)
 *     #1 0x43210a (<unknown module>)
 *     #2 0x42100b (<unknown module>)
 *
 * freed by thread T0 here:
 *     #0 0x7f8b2c001a40 in free (<unknown module>)
 *     #1 0x410023 (<unknown module>)
 *
 * (no function names, no source lines — useless for diagnosis)
 */

Correct (full ASan build with frame pointers and single-process mode):

/*
 * Complete ASan build — instruments compiler AND linker, includes
 * frame pointers for full stack traces. -O1 is optional (ASan works
 * with -O0) but produces slightly better stack traces.
 *
 * ./configure \
 *     --with-debug \
 *     --with-cc-opt='-fsanitize=address -O1 -g -fno-omit-frame-pointer' \
 *     --with-ld-opt='-fsanitize=address'
 * make -j$(nproc)
 * make install
 *
 * nginx.conf for ASan debugging:
 *   master_process off;
 *   worker_processes 1;
 *   daemon off;
 *
 * Environment variables for detailed reports:
 *   export ASAN_OPTIONS="detect_leaks=1:log_path=/tmp/asan:abort_on_error=1"
 */

/*
 * ASan report with full frame pointers (actionable):
 *
 * ==4827==ERROR: AddressSanitizer: heap-use-after-free
 *   READ of size 8 at 0x60b000001a40 thread T0
 *     #0 ngx_http_mymodule_log_handler ngx_http_mymodule.c:87
 *     #1 ngx_http_log_request ngx_http_request.c:3612
 *     #2 ngx_http_free_request ngx_http_request.c:3562
 *     #3 ngx_http_close_request ngx_http_request.c:3517
 *
 * freed by thread T0 here:
 *     #0 free (libasan.so)
 *     #1 ngx_destroy_pool ngx_palloc.c:74
 *     #2 ngx_http_free_request ngx_http_request.c:3558
 *
 * previously allocated by thread T0 here:
 *     #0 malloc (libasan.so)
 *     #1 ngx_alloc ngx_alloc.c:22
 *     #2 ngx_palloc_large ngx_palloc.c:186
 *     #3 ngx_http_mymodule_handler ngx_http_mymodule.c:45
 *
 * (exact source lines for allocation, free, and use-after-free)
 */

Reference: nginx Development Guide

Compile nginx with Full Debug Symbols and No Optimization

The default nginx build uses -O2 optimization which inlines functions, reorders code, and eliminates variables -- making GDB debugging nearly impossible. Build with --with-debug --with-cc-opt='-O0 -g' for full debug symbols and no optimization. Variables show their actual values, breakpoints hit on the correct lines, and stepping works as expected.

Incorrect (building with --with-debug only, still uses -O2 optimization):

/*
 * Build command that only enables debug logging but
 * leaves compiler optimization at -O2.
 *
 * ./configure --with-debug
 * make -j$(nproc)
 * make install
 *
 * Result: debug log macros are compiled in, but GDB sessions
 * are unusable — variables show "<optimized out>",
 * breakpoints land on wrong lines, stepping skips code.
 */

/* Attempt to inspect a request context in GDB: */
/* (gdb) break ngx_http_mymodule_handler         */
/* (gdb) print *ctx                               */
/* $1 = <optimized out>                           */
/* (gdb) print ctx->state                         */
/* $2 = <optimized out>                           */
/* (gdb) next                                     */
/* (jumps over 3 source lines due to inlining)    */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* GDB cannot inspect these values with -O2 */
    ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: state=%d flags=0x%xd",
                   ctx->state, ctx->flags);

    return NGX_OK;
}

Correct (building with -O0 -g for complete debugging support):

/*
 * Build command that disables optimization and includes
 * full DWARF debug symbols for GDB.
 *
 * ./configure \
 *     --with-debug \
 *     --with-cc-opt='-O0 -g' \
 *     --prefix=/usr/local/nginx-debug
 * make -j$(nproc)
 * make install
 *
 * --with-debug   → enables ngx_log_debug macros and debug_connection
 * -O0            → no optimization: no inlining, no reordering
 * -g             → full DWARF debug symbols for GDB
 *
 * For even more debug info (macro expansion in GDB):
 *     --with-cc-opt='-O0 -g3 -gdwarf-4'
 */

/* GDB session with -O0 -g build: */
/* (gdb) break ngx_http_mymodule_handler          */
/* Breakpoint 1 at 0x4a3f20: file ngx_http_mymodule.c, line 142. */
/* (gdb) run                                       */
/* (gdb) print *ctx                                */
/* $1 = {state = 2, flags = 0x3, upstream = 0x7f8b2c001a40} */
/* (gdb) print ctx->state                          */
/* $2 = 2                                          */
/* (gdb) next                                      */
/* 143     ngx_log_debug2(...)                      */
/* (single-line stepping works correctly)           */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* GDB shows exact values with -O0 */
    ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: state=%d flags=0x%xd",
                   ctx->state, ctx->flags);

    return NGX_OK;
}

Reference: nginx Debugging Guide

Enable NGX_DEBUG_PALLOC for Fine-Grained Pool Allocation Tracking

nginx pools allocate large blocks and sub-allocate from them, making individual allocations invisible to Valgrind and ASan. Defining NGX_DEBUG_PALLOC=1 forces ngx_palloc to use malloc for every allocation instead of sub-allocating from pool blocks. This makes every allocation visible to memory debugging tools with full call stacks, at the cost of changed allocation patterns and higher overhead.

Incorrect (running ASan on standard nginx build, per-allocation bugs invisible):

/*
 * Standard build — pools sub-allocate from 4096-byte blocks:
 *
 * ./configure --with-debug --with-cc-opt='-fsanitize=address -O1 -g \
 *     -fno-omit-frame-pointer' --with-ld-opt='-fsanitize=address'
 * make -j$(nproc)
 *
 * Pool allocation pattern (standard):
 *
 *   ngx_create_pool(4096)
 *     → malloc(4096)              ← ASan sees this one block
 *       ngx_palloc(pool, 64)     ← sub-allocated, invisible to ASan
 *       ngx_palloc(pool, 128)    ← sub-allocated, invisible to ASan
 *       ngx_palloc(pool, 32)     ← sub-allocated, invisible to ASan
 *
 * When the pool is destroyed, ASan reports one free for the entire
 * block. If the module reads from the second allocation after pool
 * destruction, ASan may not detect it — the whole block was freed
 * at once, and ASan cannot distinguish sub-allocations within it.
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t   *ctx;
    ngx_str_t  *cached;

    ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    /* allocate a string buffer from the request pool */
    cached = ngx_palloc(r->pool, sizeof(ngx_str_t));
    cached->data = ngx_pnalloc(r->pool, r->uri.len);
    cached->len = r->uri.len;
    ngx_memcpy(cached->data, r->uri.data, r->uri.len);

    /* save pointer to connection-scoped storage */
    ctx->saved_ref = cached;
    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    /*
     * BUG: after request pool is destroyed, ctx->saved_ref->data
     * is invalid. But ASan only sees one large block free — it
     * cannot pinpoint which ngx_palloc call created the dangling
     * pointer. ASan report shows:
     *
     *   freed by: ngx_destroy_pool (ngx_palloc.c:74)
     *   (no info about which specific allocation within the pool)
     */

    return NGX_OK;
}

Correct (building with NGX_DEBUG_PALLOC=1 for per-allocation tracking):

/*
 * Debug build — every ngx_palloc becomes a separate malloc:
 *
 * ./configure --with-debug \
 *     --with-cc-opt='-fsanitize=address -O1 -g \
 *         -fno-omit-frame-pointer -DNGX_DEBUG_PALLOC=1' \
 *     --with-ld-opt='-fsanitize=address'
 * make -j$(nproc)
 *
 * Pool allocation pattern (with NGX_DEBUG_PALLOC):
 *
 *   ngx_create_pool(4096)
 *     → malloc(sizeof(ngx_pool_t))     ← ASan tracks this
 *       ngx_palloc(pool, 64)
 *         → malloc(64)                 ← ASan tracks this separately
 *       ngx_palloc(pool, 128)
 *         → malloc(128)                ← ASan tracks this separately
 *       ngx_palloc(pool, 32)
 *         → malloc(32)                 ← ASan tracks this separately
 *
 * When the pool is destroyed, each sub-allocation is freed
 * individually. ASan can now detect use-after-free for any
 * specific allocation with its exact allocation call stack.
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t   *ctx;
    ngx_str_t  *cached;

    ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    /* each ngx_palloc is now a separate malloc — ASan tracks it */
    cached = ngx_palloc(r->pool, sizeof(ngx_str_t));
    cached->data = ngx_pnalloc(r->pool, r->uri.len);
    cached->len = r->uri.len;
    ngx_memcpy(cached->data, r->uri.data, r->uri.len);

    ctx->saved_ref = cached;
    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    /*
     * With NGX_DEBUG_PALLOC, ASan now reports the exact allocation:
     *
     * ==4827==ERROR: AddressSanitizer: heap-use-after-free
     *   READ of size 8 at 0x60200000efa0 thread T0
     *     #0 ngx_http_mymodule_log ngx_http_mymodule.c:112
     *
     * freed by thread T0 here:
     *     #0 free (libasan.so)
     *     #1 ngx_pfree ngx_palloc.c:142
     *     #2 ngx_destroy_pool ngx_palloc.c:88
     *
     * previously allocated by thread T0 here:
     *     #0 malloc (libasan.so)
     *     #1 ngx_palloc ngx_palloc.c:130   ← exact palloc call
     *     #2 ngx_http_mymodule_handler ngx_http_mymodule.c:89
     *
     * (pinpoints the exact ngx_palloc that created the pointer)
     */

    return NGX_OK;
}

Reference: nginx Development Guide — Memory Pools

Use Single-Process Mode for Simplified Debugging

Multi-process nginx makes debugging harder: you must identify the right worker, attach GDB to it, and signals from the master can interfere. Set master_process off; and worker_processes 1; in nginx.conf to run everything in a single process. GDB attaches directly, breakpoints work without race conditions, and there is no master process sending signals.

Incorrect (debugging with default multi-worker config, must chase PIDs and signals):

/*
 * nginx.conf — default multi-process configuration.
 *
 * worker_processes auto;   (spawns N workers)
 * daemon on;               (backgrounds the process)
 *
 * Debugging this requires multiple steps:
 *
 * Step 1: find the correct worker PID
 *   $ ps aux | grep 'nginx: worker'
 *   www-data 4827 ... nginx: worker process
 *   www-data 4828 ... nginx: worker process
 *   www-data 4829 ... nginx: worker process
 *   www-data 4830 ... nginx: worker process
 *   (which one handles your test request?)
 *
 * Step 2: attach GDB to one worker
 *   $ sudo gdb -p 4827
 *   (gdb) break ngx_http_mymodule_handler
 *   (gdb) continue
 *   (request hits worker 4829 instead — wrong PID)
 *
 * Step 3: master sends SIGCHLD/SIGALRM that interrupts GDB
 *   Program received signal SIGCHLD, Child exited.
 *   (gdb) continue
 *   Program received signal SIGALRM, Alarm clock.
 *   (gdb) continue
 *   (constant signal interruptions break debugging flow)
 *
 * Step 4: if you set follow-fork-mode child, GDB
 *   detaches from master and may miss the fork entirely.
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    /* breakpoint never triggers because request goes
     * to a different worker process */
    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: handler entry");

    return NGX_DECLINED;
}

Correct (single-process mode for direct GDB attachment):

/*
 * nginx.conf — single-process debug configuration:
 *
 * master_process off;      # no master: single process only
 * worker_processes 1;      # one worker (redundant but explicit)
 * daemon off;              # stay in foreground for GDB/valgrind
 * error_log /dev/stderr debug;  # debug log to terminal
 *
 * events {
 *     worker_connections 64;   # low limit is fine for debugging
 * }
 *
 * http {
 *     access_log off;         # reduce noise during debug sessions
 *     server {
 *         listen 8080;
 *         location /test {
 *             mymodule;
 *         }
 *     }
 * }
 *
 * Launch under GDB directly — no PID hunting, no signal noise:
 *
 *   $ gdb --args ./objs/nginx -c /path/to/debug.conf
 *   (gdb) break ngx_http_mymodule_handler
 *   (gdb) run
 *   (every request hits this single process)
 *
 * Or launch under Valgrind:
 *
 *   $ valgrind --leak-check=full \
 *       ./objs/nginx -c /path/to/debug.conf
 *   (all allocations tracked in one process)
 *
 * Or launch under ASan (just run the ASan-compiled binary):
 *
 *   $ ASAN_OPTIONS="abort_on_error=1" \
 *       ./objs/nginx -c /path/to/debug.conf
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    /* breakpoint always triggers — single process handles all requests */
    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: handler entry");

    return NGX_DECLINED;
}

Reference: nginx Debugging Guide

Use nginx Valgrind Suppressions to Reduce False Positives

Running nginx under Valgrind produces many false positives from nginx core's intentional uninitialized reads (e.g., in the hash table implementation) and OpenSSL internals. Use Valgrind suppressions to filter these out so real bugs in your module are visible. Compile with -DNGX_DEBUG_PALLOC=1 to make pool allocations visible to Valgrind, and run with --suppressions=nginx.supp to silence known false positives.

Incorrect (running Valgrind without suppressions, real bugs hidden in noise):

/*
 * Running Valgrind with no suppressions:
 *
 * $ valgrind --leak-check=full ./objs/nginx -c debug.conf
 *
 * Output is flooded with false positives:
 *
 * ==4827== Conditional jump depends on uninitialised value(s)
 * ==4827==    at 0x44B2C1: ngx_hash_find (ngx_hash.c:24)
 * ==4827==    by 0x455A3F: ngx_hash_find_combined (ngx_hash.c:47)
 * ==4827==
 * ==4827== Conditional jump depends on uninitialised value(s)
 * ==4827==    at 0x44B2E8: ngx_hash_find (ngx_hash.c:26)
 * ==4827==    by 0x455A3F: ngx_hash_find_combined (ngx_hash.c:47)
 * ==4827==
 * ==4827== Use of uninitialised value of size 8
 * ==4827==    at 0x6D3F2A1: ssl3_read_bytes (ssl3_record.c:1462)
 * ==4827==    by 0x6D42103: ssl3_read (ssl3_lib.c:162)
 * ==4827==
 * (50+ reports from nginx core and OpenSSL before any module code)
 * (developer gives up reading output — real module bugs are buried)
 *
 * Also: nginx pools sub-allocate from large blocks, so Valgrind
 * only sees the outer ngx_alloc call — individual ngx_palloc
 * calls that cause bugs are invisible.
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    /* BUG: use-after-free hidden in 50+ false positive reports */
    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "state: %d", ctx->state);

    return NGX_OK;
}

Correct (creating suppressions file and building with NGX_DEBUG_PALLOC):

/*
 * Step 1: Create a suppressions file (nginx.supp):
 *
 * --- nginx.supp ---
 * {
 *    nginx_hash_find_uninit
 *    Memcheck:Cond
 *    fun:ngx_hash_find
 * }
 * {
 *    nginx_hash_find_value
 *    Memcheck:Value8
 *    fun:ngx_hash_find
 * }
 * {
 *    nginx_init_cycle_hostname
 *    Memcheck:Param
 *    socketcall.getsockname(addr)
 *    fun:getsockname
 *    fun:ngx_connection_local_sockaddr
 * }
 * {
 *    openssl_ssl3_read
 *    Memcheck:Cond
 *    ...
 *    obj:*libssl*
 * }
 * {
 *    openssl_crypto_internal
 *    Memcheck:Value8
 *    ...
 *    obj:*libcrypto*
 * }
 * --- end ---
 *
 * Step 2: Build with NGX_DEBUG_PALLOC to expose pool allocations:
 *
 * ./configure \
 *     --with-debug \
 *     --with-cc-opt='-O0 -g -DNGX_DEBUG_PALLOC=1'
 * make -j$(nproc)
 *
 * Step 3: Run Valgrind with suppressions and gen-suppressions:
 *
 * $ valgrind \
 *     --leak-check=full \
 *     --show-reachable=yes \
 *     --track-origins=yes \
 *     --suppressions=nginx.supp \
 *     --gen-suppressions=all \
 *     ./objs/nginx -c debug.conf
 *
 * --gen-suppressions=all prints suppression blocks for any new
 * false positives so you can add them to nginx.supp incrementally.
 *
 * Clean output — only real module bugs remain:
 *
 * ==4827== Invalid read of size 4
 * ==4827==    at 0x4A3F20: ngx_http_mymodule_handler (mymodule.c:87)
 * ==4827==  Address 0x60b000001a48 is 8 bytes inside a block of size 32 free'd
 * ==4827==    at 0x483CA3F: free (vg_replace_malloc.c:540)
 * ==4827==    by 0x44F2A1: ngx_destroy_pool (ngx_palloc.c:74)
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "mymodule: ctx is NULL");
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* Valgrind now reports only real bugs in module code */
    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "state: %d", ctx->state);

    return NGX_OK;
}

Reference: nginx Development Guide

Diagnose Double Finalize Crashes from Request Reference Count

Calling ngx_http_finalize_request twice on the same request double-decrements r->main->count, which can drive it to zero prematurely. When count reaches zero, nginx destroys the request pool. Any subsequent access to the request or its pool memory triggers a use-after-free. Under high concurrency this manifests as intermittent segfaults in unrelated code because the freed pool memory gets reallocated to another request. The crash signature typically shows access to apparently valid but incorrect data structures.

Incorrect (finalize called in both error path and normal callback, double-decrement under error):

static void
ngx_http_mymodule_upstream_callback(ngx_http_request_t *r,
    ngx_http_upstream_t *u)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    if (u->headers_in.status_n != NGX_HTTP_OK) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "upstream returned %ui", u->headers_in.status_n);
        /* first finalize on error */
        ngx_http_finalize_request(r, NGX_HTTP_BAD_GATEWAY);
        /* BUG: does not return — falls through */
    }

    /* BUG: if error path ran, r->pool may already be destroyed.
     * accessing ctx is use-after-free; second finalize
     * double-decrements count */
    ctx->response_status = u->headers_in.status_n;
    ngx_http_finalize_request(r, NGX_OK);
}

Correct (single finalize per code path with immediate return):

static void
ngx_http_mymodule_upstream_callback(ngx_http_request_t *r,
    ngx_http_upstream_t *u)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    if (u->headers_in.status_n != NGX_HTTP_OK) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "upstream returned %ui", u->headers_in.status_n);
        ngx_http_finalize_request(r, NGX_HTTP_BAD_GATEWAY);
        return;  /* immediate return — no further access to r */
    }

    if (ctx == NULL) {
        ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
    }

    ctx->response_status = u->headers_in.status_n;

    /* exactly one finalize per execution path */
    ngx_http_finalize_request(r, NGX_OK);
}

Note: When debugging double-finalize crashes, add a temporary ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "finalize count=%d", r->main->count) before each ngx_http_finalize_request call. If you see count=0 in the log before a finalize, the request has already been finalized elsewhere.

Reference: nginx Development Guide

Avoid Crashes from error_page Internal Redirect Context Invalidation

When error_page triggers an internal redirect, nginx resets the request's module context array. Module context pointers obtained via ngx_http_get_module_ctx before the redirect become stale — they point to memory that may be freed or reallocated during the redirect. This is a common source of intermittent production crashes, especially under SSL with long-lived connections where the timing allows the pool to be partially reused. The crash signature shows access to data that looks structurally valid but contains wrong values, because the memory has been repurposed.

Incorrect (saves ctx pointer before an operation that may trigger error_page, uses it after redirect):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t   *ctx;
    ngx_int_t   rc;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
        if (ctx == NULL) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }
        ctx->original_uri = r->uri;
        ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);
    }

    /* this may return 404, triggering error_page 404 /fallback */
    rc = ngx_http_mymodule_lookup(r);

    if (rc == NGX_HTTP_NOT_FOUND) {
        return rc;  /* nginx processes error_page internally */
    }

    /* BUG: if error_page triggered an internal redirect in a
     * previous call, ctx now points to stale/freed memory.
     * The internal redirect reset the module context array. */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: original_uri=\"%V\"",
                   &ctx->original_uri);

    return NGX_OK;
}

Correct (re-fetches module context after any operation that may trigger an internal redirect):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t   *ctx;
    ngx_int_t   rc;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
        if (ctx == NULL) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }
        ctx->original_uri = r->uri;
        ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);
    }

    rc = ngx_http_mymodule_lookup(r);

    if (rc == NGX_HTTP_NOT_FOUND) {
        return rc;
    }

    /* re-fetch context after any operation that may have
     * triggered an internal redirect via error_page */
    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        /* context was lost during redirect — re-establish it */
        ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
        if (ctx == NULL) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }
        ctx->original_uri = r->uri;
        ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);
    }

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: original_uri=\"%V\"",
                   &ctx->original_uri);

    return NGX_OK;
}

Note: An alternative defensive pattern is to store critical state on the main request's context (r->main) rather than on the subrequest or redirected request context, since r->main survives internal redirects. Use ngx_http_get_module_ctx(r->main, ...) when the state must persist across redirects.

Reference: Coredump Debug Story About Nginx

Recognize NULL Pointer Dereference Patterns in nginx Modules

NULL pointer dereferences are the most common crash in nginx modules. They arise from three recurring patterns: unchecked allocation returns from ngx_palloc/ngx_pcalloc when pools are exhausted, accessing request fields after ngx_http_finalize_request has destroyed the request, and dereferencing module context pointers obtained via ngx_http_get_module_ctx without verifying that the context was previously set. Under memory pressure or unusual request flows, any of these NULL pointers will crash the worker process.

Incorrect (uses module context without NULL check, crashes on first request without context):

static ngx_int_t
ngx_http_mymodule_access_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    /* BUG: ctx is NULL if no prior phase set it — segfault */
    if (ctx->authenticated) {
        return NGX_OK;
    }

    /* BUG: allocation not checked either */
    ctx = ngx_palloc(r->pool, sizeof(my_ctx_t));
    ctx->authenticated = 0;
    ctx->attempts = 0;

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    return NGX_HTTP_FORBIDDEN;
}

Correct (checks every pointer before dereference, handles allocation failure):

static ngx_int_t
ngx_http_mymodule_access_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    if (ctx != NULL) {
        /* context exists from a prior phase — safe to read */
        if (ctx->authenticated) {
            return NGX_OK;
        }

        return NGX_HTTP_FORBIDDEN;
    }

    /* first invocation: allocate and zero-initialize context */
    ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
    if (ctx == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* pcalloc zeros all fields: authenticated=0, attempts=0 */
    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    return NGX_HTTP_FORBIDDEN;
}

Note: Use ngx_pcalloc instead of ngx_palloc for context structs so all pointer fields start as NULL and all integer fields as zero. This prevents reading uninitialized memory if a code path accesses a field before it is explicitly set.

Reference: nginx Development Guide

Identify Segfault Crash Signature from Signal and Address

When a worker process crashes, the error log records the signal number and the faulting memory address. This single line of information determines the entire debugging strategy: SIGSEGV at address 0x0 is a NULL pointer dereference, a small hex address like 0x8 or 0x18 indicates struct member access through a NULL pointer (the address is the member offset), and a large invalid address points to use-after-free or heap corruption where the pointer was overwritten with garbage.

Incorrect (ignoring crash signature and searching code blindly):

/*
 * Worker crash log shows:
 *   worker process 4827 exited on signal 11
 *   signal 11 (SIGSEGV), code 1, fault address 0x18
 *
 * BAD approach: grep the entire codebase for segfaults, add
 * printf debugging everywhere, rebuild, and hope to reproduce.
 *
 * This wastes hours because the fault address already tells
 * you the crash category.
 */

/* developer adds random NULL checks everywhere */
static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    /* scattered printf debugging with no strategy */
    fprintf(stderr, "DEBUG: handler called\n");
    fprintf(stderr, "DEBUG: ctx = %p\n", (void *) ctx);
    fprintf(stderr, "DEBUG: ctx->state = %d\n", ctx->state);

    return NGX_OK;
}

Correct (mapping fault address to crash category for targeted diagnosis):

/*
 * Worker crash log shows:
 *   worker process 4827 exited on signal 11
 *   signal 11 (SIGSEGV), code 1, fault address 0x18
 *
 * Step 1: Signal 11 = SIGSEGV (invalid memory access)
 *
 * Step 2: Classify by fault address:
 *   0x0         → NULL pointer dereference
 *   0x1 - 0xFFF → struct member access on NULL pointer
 *                  (0x18 = offset of a field in a struct)
 *   large addr  → use-after-free or heap corruption
 *
 * Step 3: fault address 0x18 = small offset → NULL struct access
 *         Find which struct has a member at offset 0x18:
 */

/* Use GDB to identify the struct layout */
/* (gdb) ptype /o my_ctx_t */
/*   offset  |  size  |  type   | field          */
/*     0x00  |     8  |  void*  | data           */
/*     0x08  |     8  |  void*  | handler        */
/*     0x10  |     4  |  int    | state          */
/*     0x14  |     4  |  int    | flags          */
/*     0x18  |     8  |  void*  | upstream  <--- fault offset */

/* Conclusion: ctx->upstream accessed when ctx == NULL */
/* Fix: add NULL check before accessing ctx->upstream */
static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "mymodule: ctx is NULL in handler");
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* safe: ctx is validated before member access */
    if (ctx->upstream == NULL) {
        return NGX_HTTP_BAD_GATEWAY;
    }

    return NGX_OK;
}

Reference: nginx Development Guide

Detect Stack Overflow from Recursive Subrequest or Filter Chains

Stack overflow in nginx occurs when subrequests recurse past NGX_HTTP_MAX_SUBREQUESTS (default 50) or when a body filter inadvertently triggers itself in a loop. The crash presents as SIGSEGV with a fault address near the stack boundary (typically at the bottom of the mapped stack region). The backtrace in GDB shows hundreds of repeating function frames. Unlike heap-related segfaults, the fault address is far from the heap range and falls at a page boundary.

Incorrect (body filter issues subrequest that passes through the same filter, infinite recursion):

static ngx_int_t
ngx_http_mymodule_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_http_request_t  *sr;
    ngx_int_t            rc;

    /* BUG: every invocation creates a subrequest to /internal,
     * which generates a response that passes through this same
     * body filter, creating another subrequest — infinite loop */
    rc = ngx_http_subrequest(r, &ngx_http_mymodule_uri, NULL,
                             &sr, NULL, 0);
    if (rc != NGX_OK) {
        return NGX_ERROR;
    }

    return ngx_http_next_body_filter(r, in);
}

Correct (guards against re-entrant filter processing and respects subrequest limit):

static ngx_int_t
ngx_http_mymodule_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_http_request_t  *sr;
    ngx_int_t            rc;
    my_ctx_t            *ctx;

    /* skip processing for subrequests — only act on main request */
    if (r != r->main) {
        return ngx_http_next_body_filter(r, in);
    }

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);
    if (ctx == NULL) {
        return ngx_http_next_body_filter(r, in);
    }

    /* guard: only issue the subrequest once */
    if (ctx->subrequest_issued) {
        return ngx_http_next_body_filter(r, in);
    }

    /* verify subrequest budget before issuing */
    if (r->subrequests < 1) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "mymodule: subrequest limit reached");
        return NGX_ERROR;
    }

    ctx->subrequest_issued = 1;

    rc = ngx_http_subrequest(r, &ngx_http_mymodule_uri, NULL,
                             &sr, NULL, 0);
    if (rc != NGX_OK) {
        return NGX_ERROR;
    }

    return ngx_http_next_body_filter(r, in);
}

Note: When the backtrace shows repeating frames, count the depth and identify the two or three functions forming the cycle. The fix always involves breaking the cycle with a guard flag, a subrequest counter check, or a r != r->main test.

Reference: nginx Development Guide

Extract Crash Context from Worker Exit Log Messages

When a worker process crashes, the master process logs worker process PID exited on signal N. This message alone identifies the crash type (signal 11 = SIGSEGV, signal 6 = SIGABRT) but not the location. The critical technique is to correlate the exit signal with the last debug log entries written before the crash. These entries reveal the request URI, the processing phase, and the module active at crash time. Enabling debug_connection for the offending client IP captures a complete per-request trace without flooding the log for all traffic.

Incorrect (looking only at the exit message and guessing the crash location):

/*
 * Error log shows only:
 *   2024/01/15 10:23:45 [alert] 1234#0: worker process 5678
 *                       exited on signal 11
 *
 * BAD approach: no debug logging enabled, no way to know
 * which request or module was active when the crash occurred.
 *
 * nginx.conf has only:
 */

/* error_log logs/error.log error; */

/*
 * At "error" level, nginx only logs severe errors.
 * The last lines before the crash show nothing about the
 * request being processed — the developer has no context
 * and must try to reproduce blindly.
 */

Correct (correlating exit signal with targeted debug log entries to identify crash context):

/*
 * Step 1: Enable debug logging for the problem client only.
 *
 * nginx.conf:
 *   error_log logs/error.log error;
 *
 *   events {
 *       debug_connection 192.168.1.50;
 *   }
 *
 * Step 2: Reproduce the crash. The log now shows:
 *
 *   10:23:45 [debug] 5678#0: *42 http request "/api/data?"
 *   10:23:45 [debug] 5678#0: *42 rewrite phase: 0
 *   10:23:45 [debug] 5678#0: *42 access phase: 1
 *   10:23:45 [debug] 5678#0: *42 content phase: 2
 *   10:23:45 [debug] 5678#0: *42 http mymodule handler
 *   10:23:45 [debug] 5678#0: *42 mymodule: fetching ctx
 *   10:23:45 [alert] 1234#0: worker process 5678
 *                    exited on signal 11
 *
 * Step 3: The last debug entry before the crash tells us:
 *   - Request: /api/data (connection 42)
 *   - Phase: content phase
 *   - Module: mymodule, inside "fetching ctx" logic
 *   - Signal 11: SIGSEGV — likely NULL deref after get_ctx
 *
 * Step 4: Add targeted debug logging in the module:
 */

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: handler entry, uri=\"%V\"", &r->uri);

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: ctx=%p", ctx);

    if (ctx == NULL) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "mymodule: NULL ctx for \"%V\"", &r->uri);
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: ctx->state=%d", ctx->state);

    return NGX_OK;
}

Reference: Debugging nginx

Use debug_connection to Isolate Single-Client Debug Output

In production, enabling full debug logging generates gigabytes per minute and imposes a 5-10x latency penalty on every connection. The debug_connection directive in the events block enables debug-level logging only for connections originating from specific IP addresses or CIDR blocks. All other connections continue using the configured log level. This enables production debugging of a single client's request flow without impacting performance for other users. Requires nginx built with --with-debug.

Incorrect (enabling debug globally in production, flooding disk and degrading all clients):

# nginx.conf — BAD: global debug logging in production
# This generates 10-50 GB/day under moderate traffic.
# Every connection gets debug-level output, including
# event loop internals, SSL handshake details, and
# upstream negotiation for ALL clients.

error_log /var/log/nginx/error.log debug;

events {
    worker_connections  4096;
    # No filtering — every client triggers full debug output
}

http {
    server {
        listen 80;
        server_name example.com;
        # ...
    }
}

# Result: disk fills up, latency spikes for all users,
# the one trace you need is buried in millions of lines.
# In high-traffic scenarios the worker may block on I/O
# waiting to flush log buffers, causing request timeouts.

Correct (using debug_connection to enable debug only for the test client IP):

# nginx.conf — debug only for specific test client
# Non-matching connections use the 'error' log level.
# Performance impact is limited to the matched client only.

error_log /var/log/nginx/error.log error;

events {
    worker_connections  4096;

    # Enable debug logging for the test machine only.
    # Accepts IP addresses and CIDR blocks.
    debug_connection  192.168.1.100;
    debug_connection  10.0.0.0/24;

    # The debug output for matched connections goes to
    # the per-server or global error_log, but only those
    # connections produce debug-level messages.
}

http {
    server {
        listen 80;
        server_name example.com;

        # Optional: separate log file for debug output
        # so production error logs remain clean
        error_log /var/log/nginx/debug-trace.log debug;
        error_log /var/log/nginx/error.log error;
        # ...
    }
}

# Usage:
# 1. SSH to the test machine at 192.168.1.100
# 2. curl http://example.com/problematic-endpoint
# 3. Only this request produces debug output
# 4. tail -f /var/log/nginx/debug-trace.log

Reference: nginx Debugging Log

Use Correct Debug Mask for Targeted Log Filtering

The ngx_log_debug macros accept a mask parameter (NGX_LOG_DEBUG_HTTP, NGX_LOG_DEBUG_EVENT, NGX_LOG_DEBUG_ALLOC, etc.) that categorizes debug output by subsystem. These masks are C-level constants checked via bitwise AND against log->log_level. From nginx.conf, error_log ... debug enables ALL debug masks — there is no config-level way to selectively enable only HTTP or event debug messages. However, using the correct mask per subsystem lets you filter debug output via grep after capture, separating HTTP request traces from event loop noise and allocation tracking.

Incorrect (using NGX_LOG_DEBUG_CORE for HTTP module output, defeating grep filtering):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_http_mymodule_conf_t  *conf;

    conf = ngx_http_get_module_loc_conf(r, ngx_http_mymodule_module);

    /* BAD: NGX_LOG_DEBUG_CORE makes this indistinguishable from
     * core debug output. When grepping logs by subsystem, these
     * HTTP-related messages mix with event loop and core traces,
     * making it impossible to isolate module behavior. */
    ngx_log_debug1(NGX_LOG_DEBUG_CORE, r->connection->log, 0,
                   "mymodule: processing uri \"%V\"", &r->uri);

    ngx_log_debug1(NGX_LOG_DEBUG_CORE, r->connection->log, 0,
                   "mymodule: conf value = %d", conf->value);

    /* Even memory-related debug output uses CORE */
    ngx_log_debug2(NGX_LOG_DEBUG_CORE, r->connection->log, 0,
                   "mymodule: allocated %uz bytes at %p",
                   sizeof(ngx_http_mymodule_ctx_t),
                   ngx_pcalloc(r->pool,
                               sizeof(ngx_http_mymodule_ctx_t)));

    return NGX_DECLINED;
}

Correct (using subsystem-specific masks for post-hoc grep filtering):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_http_mymodule_ctx_t   *ctx;
    ngx_http_mymodule_conf_t  *conf;

    conf = ngx_http_get_module_loc_conf(r, ngx_http_mymodule_module);

    /* NGX_LOG_DEBUG_HTTP — categorizes output as HTTP subsystem.
     * grep for "http" in debug logs to isolate request traces. */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: processing uri \"%V\"", &r->uri);

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: conf value = %d", conf->value);

    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_mymodule_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    /* NGX_LOG_DEBUG_ALLOC — categorizes output as allocation
     * subsystem; use for leak-hunting sessions where you grep
     * for allocation traces specifically */
    ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, r->connection->log, 0,
                   "mymodule: allocated ctx %uz bytes at %p",
                   sizeof(ngx_http_mymodule_ctx_t), ctx);

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    return NGX_DECLINED;
}

Reference: nginx Debugging Log

Format ngx_str_t Correctly in Debug Log Messages

ngx_str_t strings are NOT null-terminated. They consist of a .data pointer and a .len field. Using %s to log them reads past the buffer into uninitialized memory, producing garbage output, truncated strings, or crashes. Use %V for ngx_str_t * (reads .len bytes from .data). For raw u_char * buffers with a known length, use %*s with the length as the preceding argument. Incorrect format specifiers in debug logging create misleading output that sends you down the wrong debugging path.

Incorrect (using %s with ngx_str_t data, reading past the buffer boundary):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_str_t                  token;
    ngx_http_mymodule_conf_t  *conf;

    conf = ngx_http_get_module_loc_conf(r, ngx_http_mymodule_module);

    /* BAD: %s expects a null-terminated C string.
     * r->uri.data is NOT null-terminated — %s reads past .len
     * into adjacent memory (next header, pool metadata, etc.)
     * producing garbage appended to the real URI. */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: request uri: %s", r->uri.data);

    /* BAD: same problem with header values */
    if (ngx_http_arg(r, (u_char *) "token", 5, &token) == NGX_OK) {
        /* token.data points into the query string buffer,
         * not null-terminated. %s reads garbage after the value. */
        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "mymodule: token param: %s", token.data);
    }

    /* BAD: %s with method name — method_name is ngx_str_t */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: method: %s", r->method_name.data);

    return NGX_DECLINED;
}

*Correct (using %V for ngx_str_t and %s for u_char with length):*

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_str_t                  token;
    u_char                    *raw_data;
    size_t                     raw_len;
    ngx_http_mymodule_conf_t  *conf;

    conf = ngx_http_get_module_loc_conf(r, ngx_http_mymodule_module);

    /* %V takes a pointer to ngx_str_t and reads exactly .len bytes.
     * Pass the ADDRESS of the ngx_str_t, not .data */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: request uri: %V", &r->uri);

    /* %V works for all ngx_str_t: headers, args, method, etc. */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: method: %V", &r->method_name);

    /* For extracted args, %V works the same way */
    if (ngx_http_arg(r, (u_char *) "token", 5, &token) == NGX_OK) {
        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "mymodule: token param: %V", &token);
    }

    /* For raw u_char* with known length, use %*s.
     * The length argument comes BEFORE the pointer. */
    raw_data = r->header_in->pos;
    raw_len = r->header_in->last - r->header_in->pos;
    if (raw_len > 64) {
        raw_len = 64;  /* truncate for readability */
    }

    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: raw input (%uz bytes): \"%*s\"",
                   raw_len, raw_len, raw_data);

    return NGX_DECLINED;
}

Reference: nginx Development Guide — Logging

Set Log Action String for Context in Error Messages

The log action string (log->action) is automatically appended to error messages by nginx's logging system, showing what operation was in progress when the error occurred. Setting this pointer before each major operation phase creates a breadcrumb trail through your module's execution. Without it, error messages like "recv() failed (104: Connection reset by peer)" lack any indication of which module phase was running, forcing you to correlate timestamps and line numbers manually.

Incorrect (not setting log->action, producing context-free error messages):

static ngx_int_t
ngx_http_myauth_handler(ngx_http_request_t *r)
{
    ngx_int_t   rc;
    ngx_buf_t  *b;

    /* Phase 1: validate the auth token header */
    rc = ngx_http_myauth_parse_token(r);
    if (rc != NGX_OK) {
        /* Log says: "auth token parse failed"
         * but no automatic context about WHERE in the module */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "auth token parse failed");
        return NGX_HTTP_UNAUTHORIZED;
    }

    /* Phase 2: query the auth backend */
    rc = ngx_http_myauth_backend_query(r);
    if (rc != NGX_OK) {
        /* If the backend connection fails, nginx core logs:
         * "recv() failed (104: Connection reset by peer)"
         * No way to tell this came from the auth backend vs
         * the upstream, proxy, or any other module */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "auth backend query failed");
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* Phase 3: inject auth headers into the request */
    rc = ngx_http_myauth_inject_headers(r);
    if (rc != NGX_OK) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "failed to inject auth headers");
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    return NGX_DECLINED;
}

Correct (setting log->action before each phase for automatic context in all messages):

static ngx_int_t
ngx_http_myauth_handler(ngx_http_request_t *r)
{
    ngx_int_t   rc;
    ngx_buf_t  *b;

    /* Phase 1: validate the auth token header
     * Pointer assignment only — zero runtime cost */
    r->connection->log->action = "parsing auth token in myauth module";

    rc = ngx_http_myauth_parse_token(r);
    if (rc != NGX_OK) {
        /* Now logs: "auth token parse failed while parsing
         * auth token in myauth module, client: 10.0.0.1" */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "auth token parse failed");
        return NGX_HTTP_UNAUTHORIZED;
    }

    /* Phase 2: query the auth backend */
    r->connection->log->action = "querying auth backend in myauth module";

    rc = ngx_http_myauth_backend_query(r);
    if (rc != NGX_OK) {
        /* Core errors now show context automatically:
         * "recv() failed (104: Connection reset by peer) while
         * querying auth backend in myauth module"
         * Immediately distinguishable from upstream/proxy errors */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "auth backend query failed");
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* Phase 3: inject auth headers */
    r->connection->log->action = "injecting auth headers in myauth module";

    rc = ngx_http_myauth_inject_headers(r);
    if (rc != NGX_OK) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "failed to inject auth headers");
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    return NGX_DECLINED;
}

Reference: nginx Development Guide

Use Memory Buffer Logging to Capture Debug Output Without Disk I/O

Writing debug logs to disk introduces significant I/O overhead and can change timing behavior, hiding race conditions and making Heisenbugs disappear. Memory buffer logging (error_log memory:SIZE debug) writes to a cyclic in-memory buffer with negligible overhead. The buffer wraps around when full, always retaining the most recent messages. Extract the buffer content via GDB after a crash or on-demand from a running process. This is the preferred approach for debugging timing-sensitive issues.

Incorrect (using file-based debug logging when investigating timing-sensitive bugs):

/*
 * Scenario: intermittent race condition between upstream
 * response handling and client disconnect. Happens once
 * every ~10,000 requests under load.
 *
 * nginx.conf:
 *   error_log /var/log/nginx/debug.log debug;
 *
 * Problem: disk I/O from debug logging adds 50-200μs per
 * log line. Under load, this changes the timing enough
 * that the race condition never triggers. You're debugging
 * for hours with debug enabled but the bug vanishes.
 *
 * Additional problems:
 * - Debug log grows at 500MB+/min under load
 * - write() syscalls contend with sendfile() for I/O bandwidth
 * - Log rotation under debug load risks losing the crash trace
 */

static void
ngx_http_mymodule_upstream_read(ngx_http_request_t *r)
{
    /* These debug calls each trigger a write() syscall */
    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: upstream read handler entered");

    /* The I/O delay from the debug log above changes the
     * window between this check and the actual read below,
     * preventing the race from occurring */
    if (r->connection->error) {
        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "mymodule: client already disconnected");
        return;
    }

    /* ... rest of handler ... */
}

Correct (using memory buffer logging to preserve timing while capturing debug output):

# nginx.conf — memory buffer logging (no disk I/O)
# The cyclic buffer keeps the most recent 32MB of debug output.
# Overhead is a memcpy to the buffer — no syscalls, no disk.
error_log memory:32m debug;

# Extract the memory buffer from a running or crashed process
# using GDB. The buffer is stored in log->wdata (set by
# ngx_log_memory_writer during initialization).

# Method 1: Attach to running worker and dump buffer
$ gdb -batch -p $(pgrep -f 'nginx: worker') \
    -ex 'set $log = ngx_cycle->log' \
    -ex 'while $log && $log->writer != ngx_log_memory_writer' \
    -ex 'set $log = $log->next' \
    -ex 'end' \
    -ex 'set $buf = (ngx_log_memory_buf_t *) $log->wdata' \
    -ex 'dump binary memory /tmp/nginx-debug.log $buf->start $buf->end'

# Method 2: From a core dump after a crash
$ gdb /usr/local/nginx/sbin/nginx /tmp/core.12345
(gdb) set $log = ngx_cycle->log
(gdb) while $log && $log->writer != ngx_log_memory_writer
 > set $log = $log->next
 > end
(gdb) set $buf = (ngx_log_memory_buf_t *) $log->wdata
(gdb) dump binary memory /tmp/crash-debug.log $buf->start $buf->end
(gdb) quit

# View the extracted log (may contain binary zeros at the
# wrap point — filter with tr or strings)
$ strings /tmp/crash-debug.log | tail -500

Reference: nginx Debugging Log — Logging to a Cyclic Memory Buffer

Attach GDB to a Running nginx Worker Process

Attaching GDB to a specific worker PID enables live debugging of hangs, infinite loops, or incorrect behavior in real time. The critical distinction is that the master process does not handle requests; only worker processes do. During development, use single-process mode (master_process off; worker_processes 1;) to simplify attaching. In multi-worker setups, identify which worker handles the problematic connection before attaching.

Incorrect (attaching to the master process, which never processes requests):

# Find nginx PIDs
$ ps aux | grep nginx
root      3201  ...  nginx: master process /usr/local/nginx/sbin/nginx
www-data  3202  ...  nginx: worker process
www-data  3203  ...  nginx: worker process

# BAD: attaching to the master PID
$ gdb -p 3201

# (gdb) bt
# #0  sigsuspend () at ...
# #1  ngx_master_process_cycle ()
#
# The master is blocked in sigsuspend() waiting for signals.
# It never processes HTTP requests, so you will never see
# your module's handler code in the backtrace.
# Setting breakpoints on your handler will never trigger.

# (gdb) break ngx_http_mymodule_handler
# Breakpoint 1 at 0x...
# ... (nothing happens, master doesn't call handlers)

Correct (attaching to the worker that handles the connection):

# Step 1: Use single-process mode for development
# nginx.conf:
#   master_process off;
#   worker_processes 1;
#   daemon off;

# Step 2: Find the worker PID (not master)
$ ps aux | grep 'nginx: worker'
www-data  3202  ...  nginx: worker process
www-data  3203  ...  nginx: worker process

# Step 3: For multi-worker, find which worker handles
# your test connection (check connection count or use
# a single worker during debugging)
$ gdb -p 3202

# Step 4: Set breakpoints on your module's functions
(gdb) break ngx_http_mymodule_handler
(gdb) continue

# Step 5: Send the test request from another terminal
# $ curl http://localhost/test

# GDB breaks at your handler:
# Breakpoint 1, ngx_http_mymodule_handler (r=0x...)
(gdb) print (char *)r->uri.data
# $1 = "/test"

# Step 6: When done, detach cleanly (don't kill the worker)
(gdb) detach

Reference: nginx Development Guide

Read nginx Backtrace to Identify Crash Module and Phase

An nginx backtrace reveals the crash location through the function call chain. Key functions identify the execution context: ngx_http_*_handler functions indicate the request phase, ngx_http_*_filter functions show filter chain position, ngx_http_upstream_* functions reveal upstream proxy state, and ngx_http_finalize_request indicates cleanup. Frame 0 shows the immediate crash site, but frames further up show why that code path was reached, which is often more important for determining the fix.

Incorrect (looking only at frame 0 and fixing the symptom):

/*
 * Backtrace from coredump:
 *
 * (gdb) bt
 * #0  0x00005612a3c1f8a0 in ngx_http_mymodule_process ()
 * #1  0x00005612a3b8e200 in ngx_http_upstream_process_body_in_memory ()
 * #2  0x00005612a3b8a140 in ngx_http_upstream_handler ()
 * #3  0x00005612a3b71c00 in ngx_epoll_process_events ()
 * #4  0x00005612a3b68400 in ngx_process_events_and_timers ()
 * #5  0x00005612a3b6f800 in ngx_worker_process_cycle ()
 *
 * BAD approach: look only at frame 0, see a NULL dereference
 * in ngx_http_mymodule_process, add a NULL check, and move on.
 */

static ngx_int_t
ngx_http_mymodule_process(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    /* "Fix" — add NULL check (treats symptom, not cause) */
    if (ctx == NULL) {
        return NGX_ERROR;  /* why is ctx NULL here? unknown */
    }

    /* the real question: why did the upstream body callback
     * reach this module when context was never initialized?  */
    return ctx->handler(r, ctx);
}

Correct (reading the full backtrace to identify phase and root cause):

/*
 * Read the backtrace bottom-to-top for context:
 *
 * (gdb) bt full
 * #5  ngx_worker_process_cycle   → worker event loop
 * #4  ngx_process_events_and_timers → epoll returned an event
 * #3  ngx_epoll_process_events   → socket became readable
 * #2  ngx_http_upstream_handler  → upstream event dispatched
 * #1  ngx_http_upstream_process_body_in_memory → reading body
 * #0  ngx_http_mymodule_process  → OUR CODE — crash here
 *
 * Analysis: the crash happens during upstream body processing
 * (frame #1), NOT during normal phase handling. This means:
 * - The request is in upstream state, not phase engine state
 * - ctx must be set BEFORE upstream is initiated
 *
 * (gdb) frame 1
 * (gdb) print u->input_filter_ctx
 * $1 = (void *) 0x0
 *
 * Root cause: input_filter_ctx was not set when upstream
 * was initialized. The module's create_request callback
 * forgot to assign u->input_filter_ctx = ctx.
 */

/* Fix: set input_filter_ctx during upstream initialization */
static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t            *ctx;
    ngx_http_upstream_t *u;

    ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
    if (ctx == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    if (ngx_http_upstream_create(r) != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    u = r->upstream;
    u->input_filter_init = ngx_http_mymodule_filter_init;
    u->input_filter = ngx_http_mymodule_filter;
    u->input_filter_ctx = ctx;  /* ROOT CAUSE FIX */

    return NGX_OK;
}

Reference: nginx Development Guide

Configure Core Dump Generation for nginx Worker Crashes

Without proper core dump configuration, worker crashes leave no artifact for analysis and you are left guessing. Core dumps require coordinated setup across three layers: nginx.conf directives (working_directory, worker_rlimit_core), kernel core pattern (/proc/sys/kernel/core_pattern), and OS-level resource limits (ulimit). The most common failure mode is that the nginx worker runs as the nobody or www-data user and the dump directory lacks write permissions for that user, so coredumps are silently discarded.

Incorrect (incomplete setup that silently discards coredumps):

/*
 * nginx.conf — only kernel core_pattern is set,
 * missing nginx.conf directives and directory permissions.
 *
 * The worker process has no working_directory set,
 * so it tries to write the core to '/' (no permission).
 * worker_rlimit_core is unset, defaulting to 0 (no dump).
 */

/* System setup (as root): */
/* echo '/tmp/cores/core.%e.%p' > /proc/sys/kernel/core_pattern */

/* nginx.conf — MISSING critical directives: */
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}

/*
 * Result: worker crashes with signal 11, error log says
 * "exited on signal 11" but NO coredump file is created.
 * /tmp/cores/ is empty or doesn't exist.
 */

Correct (full setup across nginx.conf, kernel, and filesystem):

/*
 * Step 1: Create dump directory with correct ownership
 */
/* mkdir -p /var/coredumps                         */
/* chown www-data:www-data /var/coredumps           */
/* chmod 0700 /var/coredumps                        */

/*
 * Step 2: Set kernel core pattern and limits (as root)
 */
/* echo '/var/coredumps/core.%e.%p.%t' > /proc/sys/kernel/core_pattern */
/* echo 0 > /proc/sys/kernel/core_uses_pid                             */
/* ulimit -c unlimited                                                  */

/*
 * Step 3: nginx.conf — enable coredumps in nginx itself
 */
worker_processes  1;
working_directory /var/coredumps;    /* where worker writes the dump  */
worker_rlimit_core 500m;            /* max core size (enough for heap) */

/* Build nginx with debug symbols for useful backtraces:               */
/* ./configure --with-debug --with-cc-opt='-O0 -g'                     */

events {
    worker_connections  1024;
}

http {
    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}

/*
 * Step 4: Verify after a crash
 */
/* ls -la /var/coredumps/core.nginx.*               */
/* gdb /usr/local/nginx/sbin/nginx /var/coredumps/core.nginx.4827  */
/* (gdb) bt full                                                    */

Reference: nginx Debugging Guide

Inspect ngx_http_request_t Fields in GDB for Request State

The ngx_http_request_t struct contains all state needed to understand what was happening at crash time. Key fields include r->uri, r->phase_handler, r->count (reference count), r->main (main vs subrequest), r->internal (internal redirect), and r->connection->destroyed. The main challenge is that ngx_str_t fields consist of a pointer and length, so printing them with p r->uri shows raw struct data instead of a readable string. Use p *r->uri.data@r->uri.len to display the actual string content.

Incorrect (printing ngx_str_t fields as raw structs, unreadable output):

# Attached to coredump or live worker
(gdb) print r->uri
# $1 = {len = 5, data = 0x7f2a3c001230 "/test\004\200..."}
#
# The raw print includes garbage after the string because
# ngx_str_t is NOT null-terminated. The output is misleading.

(gdb) print r->args
# $2 = {len = 8, data = 0x7f2a3c001236 "foo=bar\001\377..."}
#
# Again, garbage after the string data.

(gdb) print r->headers_in.host
# $3 = (ngx_table_elt_t *) 0x7f2a3c002100
#
# Just a pointer — no useful information visible.

# Developer gives up trying to read request state from GDB,
# adds printf debugging, rebuilds, and loses the crash context.

Correct (using proper GDB commands to inspect nginx request fields):

# Define a helper for ngx_str_t fields (add to ~/.gdbinit)
# define ngx_str
#   set $str = $arg0
#   set print elements $str.len
#   printf "%.*s\n", (int)$str.len, $str.data
# end

# Inspect request URI and arguments
(gdb) p *r->uri.data@r->uri.len
# $1 = "/test"

(gdb) p *r->args.data@r->args.len
# $2 = "foo=bar"

# Inspect request phase and state
(gdb) p r->phase_handler
# $3 = 7    (index into the phase handler array)

(gdb) p r->count
# $4 = 1    (reference count; >1 means active subrequests)

(gdb) p r->main == r
# $5 = 1    (this IS the main request)

(gdb) p r->internal
# $6 = 0    (no internal redirect has occurred)

# Inspect connection state
(gdb) p r->connection->fd
# $7 = 12   (socket file descriptor)

(gdb) p r->connection->destroyed
# $8 = 0    (connection is still alive)

# Inspect host header
(gdb) p *r->headers_in.host->value.data@r->headers_in.host->value.len
# $9 = "localhost"

# Inspect HTTP method
(gdb) p *r->method_name.data@r->method_name.len
# $10 = "GET"

Reference: nginx Development Guide

Extract Debug Log from Memory Buffer Using GDB Script

When using error_log memory:32m debug;, nginx writes the debug log to a cyclic memory buffer within the worker process. This buffer survives in the coredump even after a crash, providing the complete debug trace leading up to the failure. File-based debug logging at the debug level imposes a 5-10x performance penalty and fills disk rapidly under load, making it unsuitable for production. Memory buffer logging has near-zero overhead and captures the same information, recoverable post-crash via a GDB script.

Incorrect (using file-based debug logging in production, destroying performance):

/*
 * nginx.conf — file-based debug logging
 *
 * Under production load (10,000+ req/s), this configuration:
 * - Reduces throughput by 5-10x due to synchronous disk writes
 * - Generates 1-10 GB/hour of log data
 * - Fills disk and causes cascading failures
 * - May not even capture the crash if disk becomes full
 */

error_log /var/log/nginx/debug.log debug;

worker_processes  4;

events {
    worker_connections  4096;
}

http {
    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}

/*
 * Result under load:
 *   $ du -sh /var/log/nginx/debug.log
 *   4.2G    /var/log/nginx/debug.log   (after 30 minutes)
 *   $ nginx -s reload  → disk full, reload fails
 */

Correct (memory buffer logging with GDB extraction script):

/*
 * nginx.conf — memory buffer debug logging (near-zero overhead)
 */
error_log memory:32m debug;

worker_processes  1;
working_directory /var/coredumps;
worker_rlimit_core 500m;

events {
    worker_connections  4096;
}

http {
    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}

/*
 * After a crash, extract the debug log from the coredump:
 *
 * Save this as extract_debug_log.gdb:
 */

/*
 * set $log = ngx_cycle->log
 * while $log
 *   if $log->writer == ngx_log_memory_writer
 *     set $buf = (ngx_log_memory_buf_t *) $log->wdata
 *     set $start = $buf->start
 *     set $pos = $buf->pos
 *     set $end = $buf->end
 *
 *     # Dump from write position to end (older entries)
 *     if $pos < $end
 *       dump binary memory /tmp/debug_tail.log $pos $end
 *     end
 *     # Dump from start to write position (newer entries)
 *     dump binary memory /tmp/debug_head.log $start $pos
 *
 *     printf "Extracted debug log to /tmp/debug_head.log"
 *     printf " and /tmp/debug_tail.log\n"
 *     printf "Concatenate: cat tail head > full_debug.log\n"
 *   end
 *   set $log = $log->next
 * end
 */

/* Usage:
 * $ gdb /usr/local/nginx/sbin/nginx /var/coredumps/core.nginx.4827
 * (gdb) source extract_debug_log.gdb
 * $ cat /tmp/debug_tail.log /tmp/debug_head.log > full_debug.log
 * $ grep 'http request\|upstream\|finalize' full_debug.log
 */

Reference: nginx Debugging Guide

Use GDB Watchpoints to Catch Memory Corruption at Write Time

When a struct field contains a corrupted value but the crash only manifests later when the value is read, the crash site is the symptom, not the cause. A GDB hardware watchpoint on the memory address of the corrupted field breaks execution at the exact instruction that writes the bad data, even if the write comes from a completely unrelated function or module. Hardware watchpoints are implemented by the CPU debug registers (limited to 4 on x86-64), so they have zero performance impact and do not slow down execution.

Incorrect (setting breakpoints at the read site where the crash occurs):

# Crash backtrace shows:
# #0  ngx_http_mymodule_send_response (r=0x...) at my_module.c:145
#
# (gdb) frame 0
# (gdb) print ctx->buffer_size
# $1 = -559038737   (0xDEADBEEF — clearly corrupted)

# BAD: set breakpoints where the corrupted value is READ
(gdb) break my_module.c:145
(gdb) continue

# When it breaks, ctx->buffer_size is already corrupted.
# The corruption happened earlier in the request lifecycle.
# We can see THAT it's corrupted, but not WHO corrupted it.

# Developer tries adding more breakpoints at every function
# that touches ctx, spending hours on trial and error:
(gdb) break my_module.c:80
(gdb) break my_module.c:95
(gdb) break my_module.c:110
# ... this doesn't scale and misses cross-module corruption

Correct (using a hardware watchpoint to catch the exact corruption write):

# Step 1: From coredump or live session, find the address of
# the corrupted field
(gdb) print &ctx->buffer_size
# $1 = (size_t *) 0x7f2a3c004a18

(gdb) print ctx->buffer_size
# $2 = 3735928559   (0xDEADBEEF — corrupted)

# Step 2: Restart nginx in single-process mode and attach GDB
# nginx.conf: master_process off; worker_processes 1; daemon off;
$ gdb --args /usr/local/nginx/sbin/nginx -c /etc/nginx/debug.conf

# Step 3: Set a breakpoint at context creation to get ctx address
(gdb) break ngx_http_mymodule_handler
(gdb) run

# When handler fires, get the live ctx address
(gdb) print ctx
# $3 = (my_ctx_t *) 0x7f2a3c008200

# Step 4: Set hardware watchpoint on the field
(gdb) watch *(size_t *)&ctx->buffer_size
# Hardware watchpoint 2: *(size_t *)&ctx->buffer_size
(gdb) continue

# Step 5: GDB stops at the EXACT write instruction
# Hardware watchpoint 2: *(size_t *)&ctx->buffer_size
# Old value = 8192
# New value = 3735928559
# ngx_http_other_module_cleanup (data=0x...) at other_module.c:203
# 203     ngx_memset(pool_block, 0xDEAD, block_size);
#
# ROOT CAUSE FOUND: another module's cleanup handler is
# overwriting memory that our ctx occupies — a use-after-free
# because ctx was allocated from a pool that was destroyed.

(gdb) bt
# Full backtrace shows exactly how we got here

Reference: nginx Development Guide

Find Buffer Overrun from ngx_pnalloc Size Miscalculation

When calculating buffer size for string operations (concatenation, formatting), off-by-one errors or missing space for terminators cause writes past the allocated buffer. Since pool allocations are contiguous, this corrupts the next allocation silently. The crash appears later in unrelated code that reads the corrupted data, making the root cause extremely difficult to trace without AddressSanitizer.

Incorrect (allocates exactly str1.len + str2.len bytes, missing separator and null terminator):

static ngx_int_t
ngx_http_mymodule_build_key(ngx_http_request_t *r, ngx_str_t *key,
    ngx_str_t *prefix, ngx_str_t *uri)
{
    u_char  *p;

    /* BUG: needs prefix->len + 1 (separator ':') + uri->len + 1 (null term)
     * but only allocates prefix->len + uri->len — overruns by 2 bytes */
    key->len = prefix->len + uri->len;
    key->data = ngx_pnalloc(r->pool, key->len);
    if (key->data == NULL) {
        return NGX_ERROR;
    }

    p = key->data;
    p = ngx_cpymem(p, prefix->data, prefix->len);
    *p++ = ':';                    /* writes 1 byte past allocation */
    p = ngx_cpymem(p, uri->data, uri->len);
    *p = '\0';                     /* writes 2 bytes past allocation */

    /* next ngx_pnalloc from this pool may return memory that
     * overlaps the overrun — silent corruption */

    return NGX_OK;
}

Correct (calculates exact required size including separator, prefix, and null terminator):

static ngx_int_t
ngx_http_mymodule_build_key(ngx_http_request_t *r, ngx_str_t *key,
    ngx_str_t *prefix, ngx_str_t *uri)
{
    u_char  *p;

    /* separator ':' = 1 byte, null terminator = 1 byte for C library calls */
    key->len = prefix->len + 1 + uri->len;
    key->data = ngx_pnalloc(r->pool, key->len + 1);
    if (key->data == NULL) {
        return NGX_ERROR;
    }

    p = key->data;
    p = ngx_cpymem(p, prefix->data, prefix->len);
    *p++ = ':';
    p = ngx_cpymem(p, uri->data, uri->len);
    *p = '\0';

    /* SAFE: all writes are within the allocated (prefix + 1 + uri + 1) region.
     * key->len does not include the null terminator (nginx convention),
     * but the allocation does to support passing key->data to C string APIs. */

    return NGX_OK;
}

Reference: nginx Development Guide — Strings

Detect Resource Leaks from Missing Pool Cleanup Handlers

nginx pool destruction frees memory but does NOT close file descriptors, network connections, or free external library resources. Without pool cleanup handlers, these resources leak on every request. Monitor open file descriptors per worker (ls /proc/PID/fd | wc -l on Linux, lsof -p PID | wc -l on macOS) — steady growth indicates missing cleanup handlers.

Incorrect (opens a file in handler without registering a cleanup handler):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_fd_t    fd;
    ngx_str_t   path = ngx_string("/var/data/cache.dat");
    u_char      buf[4096];
    ssize_t     n;

    fd = ngx_open_file(path.data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, ngx_errno,
                      ngx_open_file_n " \"%V\" failed", &path);
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    n = ngx_read_fd(fd, buf, sizeof(buf));

    /* BUG: if the code returns early or an error occurs below,
     * fd is never closed. Even on the normal path, fd leaks because
     * r->pool destruction only frees memory, not file descriptors. */

    /* ... process buf, may return NGX_ERROR early ... */

    return NGX_OK;
    /* fd leaked — pool cleanup would have caught this */
}

Correct (registers ngx_pool_cleanup_add with a handler that calls ngx_close_file):

typedef struct {
    ngx_fd_t    fd;
} my_cleanup_ctx_t;

static void
ngx_http_mymodule_cleanup(void *data)
{
    my_cleanup_ctx_t  *ctx = data;

    if (ctx->fd != NGX_INVALID_FILE) {
        ngx_close_file(ctx->fd);
        ctx->fd = NGX_INVALID_FILE;
    }
}

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_fd_t             fd;
    ngx_str_t            path = ngx_string("/var/data/cache.dat");
    u_char               buf[4096];
    ssize_t              n;
    ngx_pool_cleanup_t  *cln;
    my_cleanup_ctx_t    *ctx;

    fd = ngx_open_file(path.data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, ngx_errno,
                      ngx_open_file_n " \"%V\" failed", &path);
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    /* SAFE: register cleanup immediately after opening the resource */
    cln = ngx_pool_cleanup_add(r->pool, sizeof(my_cleanup_ctx_t));
    if (cln == NULL) {
        ngx_close_file(fd);
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    ctx = cln->data;
    ctx->fd = fd;
    cln->handler = ngx_http_mymodule_cleanup;

    n = ngx_read_fd(fd, buf, sizeof(buf));

    /* SAFE: even if we return NGX_ERROR, pool destruction calls
     * ngx_http_mymodule_cleanup which closes fd */

    /* ... process buf ... */

    return NGX_OK;
}

Reference: nginx Development Guide — Pool Cleanup

Identify Pool Memory Leak Patterns from Growing Worker RSS

nginx pools do not support individual frees — memory grows until the pool is destroyed. If a handler allocates from the connection pool (c->pool) on every request instead of the request pool (r->pool), memory accumulates across requests on long-lived connections (keepalive, HTTP/2 multiplexed streams). Monitor worker RSS over time; linear growth that correlates with request rate indicates a pool-level leak.

Incorrect (allocates per-request data from connection pool, which persists across keepalive requests):

typedef struct {
    ngx_str_t    response_body;
} my_req_data_t;

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_connection_t  *c = r->connection;
    my_req_data_t     *data;

    /* BUG: c->pool survives across keepalive requests on this connection.
     * Each request adds ~4KB here that is never freed until connection close.
     * With keepalive_requests=1000 (default), this leaks ~4MB per connection. */
    data = ngx_pcalloc(c->pool, sizeof(my_req_data_t));
    if (data == NULL) {
        return NGX_ERROR;
    }

    data->response_body.data = ngx_pnalloc(c->pool, 4096);
    if (data->response_body.data == NULL) {
        return NGX_ERROR;
    }
    data->response_body.len = 4096;

    /* ... use data to build response ... */

    return NGX_OK;
}

Correct (allocates per-request data from request pool, which is destroyed when request finishes):

typedef struct {
    ngx_str_t    response_body;
} my_req_data_t;

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_req_data_t  *data;

    /* SAFE: r->pool is destroyed by ngx_http_free_request at end of request.
     * No accumulation across keepalive requests on the same connection. */
    data = ngx_pcalloc(r->pool, sizeof(my_req_data_t));
    if (data == NULL) {
        return NGX_ERROR;
    }

    data->response_body.data = ngx_pnalloc(r->pool, 4096);
    if (data->response_body.data == NULL) {
        return NGX_ERROR;
    }
    data->response_body.len = 4096;

    /* ... use data to build response ... */

    return NGX_OK;
}

Reference: nginx Development Guide — Memory Pools

Diagnose Shared Memory Slab Corruption from Multi-Worker Crashes

Modifying shared data structures (rbtrees, counters, hash tables) in a slab zone without holding the mutex corrupts those structures and the slab metadata they reference. ngx_slab_alloc() self-locks internally, so the allocation itself is safe — but the surrounding operations (rbtree inserts, counter increments) are not. The corruption manifests as crashes in ngx_slab_alloc or ngx_slab_free in any worker process, not just the one that caused the corruption. The crash backtrace points to slab internals rather than the offending module, making root-cause analysis difficult. The fix is to lock externally with ngx_shmtx_lock and use ngx_slab_alloc_locked() so all operations share a single critical section.

Incorrect (updates shared memory counter without acquiring the shmtx lock):

typedef struct {
    ngx_rbtree_t      rbtree;
    ngx_rbtree_node_t sentinel;
    ngx_uint_t        total_entries;
} my_shm_data_t;

static ngx_int_t
ngx_http_mymodule_add_entry(ngx_http_request_t *r, ngx_shm_zone_t *zone,
    ngx_str_t *key)
{
    ngx_slab_pool_t    *shpool;
    my_shm_data_t      *data;
    ngx_rbtree_node_t  *node;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;
    data = (my_shm_data_t *) shpool->data;

    /* BUG: ngx_slab_alloc() self-locks so the alloc is safe, but
     * ngx_rbtree_insert and total_entries++ below have NO lock —
     * concurrent workers corrupt the rbtree structure, causing
     * crashes in ngx_slab_alloc or ngx_slab_free later */
    node = ngx_slab_alloc(shpool, sizeof(ngx_rbtree_node_t) + key->len);
    if (node == NULL) {
        return NGX_ERROR;
    }

    node->key = ngx_crc32_short(key->data, key->len);
    ngx_rbtree_insert(&data->rbtree, node);
    data->total_entries++;

    return NGX_OK;
}

Correct (locks before slab operations; uses atomic operations for simple counters):

typedef struct {
    ngx_rbtree_t      rbtree;
    ngx_rbtree_node_t sentinel;
    ngx_atomic_t      total_entries;  /* atomic for lock-free reads */
} my_shm_data_t;

static ngx_int_t
ngx_http_mymodule_add_entry(ngx_http_request_t *r, ngx_shm_zone_t *zone,
    ngx_str_t *key)
{
    ngx_slab_pool_t    *shpool;
    my_shm_data_t      *data;
    ngx_rbtree_node_t  *node;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;
    data = (my_shm_data_t *) shpool->data;

    /* SAFE: lock protects both slab allocator metadata and rbtree structure */
    ngx_shmtx_lock(&shpool->mutex);

    node = ngx_slab_alloc_locked(shpool,
                                 sizeof(ngx_rbtree_node_t) + key->len);
    if (node == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    node->key = ngx_crc32_short(key->data, key->len);
    ngx_rbtree_insert(&data->rbtree, node);

    ngx_shmtx_unlock(&shpool->mutex);

    /* counter can be updated atomically outside the lock */
    ngx_atomic_fetch_add(&data->total_entries, 1);

    return NGX_OK;
}

Reference: nginx Development Guide — Shared Memory

Avoid Storing Long-Lived Pointers in Temporary Pools

nginx destroys temporary pools at specific lifecycle points: r->pool is reset during internal redirects, and request body temporary pools are freed after body processing completes. Saving pointers from these pools and accessing them after the lifecycle event causes use-after-free. This is especially dangerous with r->request_body->temp_file and with subrequests that share the parent request body.

Incorrect (caches pointer to request body data from temp pool, reads it in a later phase):

typedef struct {
    u_char      *body_data;    /* dangling after body temp pool freed */
    size_t       body_len;
} my_ctx_t;

static void
ngx_http_mymodule_body_read(ngx_http_request_t *r)
{
    my_ctx_t     *ctx;
    ngx_chain_t  *cl;
    ngx_buf_t    *buf;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    cl = r->request_body->bufs;
    if (cl == NULL) {
        ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST);
        return;
    }

    buf = cl->buf;

    /* BUG: body buffer may be in a temporary pool or temp file mapping
     * that is freed after this callback returns */
    ctx->body_data = buf->pos;
    ctx->body_len = buf->last - buf->pos;

    /* later phase reads ctx->body_data — use-after-free */
    ngx_http_finalize_request(r, ngx_http_mymodule_process(r));
}

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }
    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    r->request_body_in_single_buf = 1;
    r->request_body_in_persistent_file = 0;

    return ngx_http_read_client_request_body(r,
                                             ngx_http_mymodule_body_read);
}

Correct (copies required data to stable pool before lifecycle boundary):

typedef struct {
    u_char      *body_data;    /* deep copy in r->pool */
    size_t       body_len;
} my_ctx_t;

static void
ngx_http_mymodule_body_read(ngx_http_request_t *r)
{
    my_ctx_t     *ctx;
    ngx_chain_t  *cl;
    ngx_buf_t    *buf;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    cl = r->request_body->bufs;
    if (cl == NULL) {
        ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST);
        return;
    }

    buf = cl->buf;

    /* SAFE: deep-copy body into r->pool before the body temp pool is freed */
    ctx->body_len = buf->last - buf->pos;
    ctx->body_data = ngx_pnalloc(r->pool, ctx->body_len);
    if (ctx->body_data == NULL) {
        ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
    }
    ngx_memcpy(ctx->body_data, buf->pos, ctx->body_len);

    /* SAFE: ctx->body_data now lives in r->pool, independent of
     * the request body temporary pool lifecycle */
    ngx_http_finalize_request(r, ngx_http_mymodule_process(r));
}

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    my_ctx_t  *ctx;

    ctx = ngx_pcalloc(r->pool, sizeof(my_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }
    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    r->request_body_in_single_buf = 1;
    r->request_body_in_persistent_file = 0;

    return ngx_http_read_client_request_body(r,
                                             ngx_http_mymodule_body_read);
}

Reference: nginx Development Guide — HTTP Request Body

Detect Use-After-Free from Pool Destruction Timing

When a connection or request pool is destroyed, all memory allocated from it becomes invalid. Saving a pointer from a request pool and accessing it after ngx_http_finalize_request completes triggers use-after-free. The bug is intermittent because the freed memory may still contain valid-looking data until it is reallocated by a subsequent request on the same connection.

Incorrect (stores request pool pointer in connection context, reads it after request completes):

typedef struct {
    ngx_str_t    cached_uri;   /* points into r->pool memory */
} my_conn_ctx_t;

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_connection_t  *c = r->connection;
    my_conn_ctx_t     *ctx;

    ctx = ngx_pcalloc(c->pool, sizeof(my_conn_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    /* BUG: cached_uri.data points into r->pool — freed when request ends */
    ctx->cached_uri.len = r->uri.len;
    ctx->cached_uri.data = r->uri.data;

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    return NGX_DECLINED;
}

static ngx_int_t
ngx_http_mymodule_log_handler(ngx_http_request_t *r)
{
    my_conn_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    /* BUG: if this runs on a keepalive request, ctx->cached_uri.data
     * from the previous request's pool is already freed */
    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "previous uri: %V", &ctx->cached_uri);

    return NGX_OK;
}

Correct (copies data to connection pool so it survives request destruction):

typedef struct {
    ngx_str_t    cached_uri;   /* owns its own copy in c->pool */
} my_conn_ctx_t;

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_connection_t  *c = r->connection;
    my_conn_ctx_t     *ctx;

    ctx = ngx_pcalloc(c->pool, sizeof(my_conn_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    /* SAFE: deep-copy URI data into c->pool, which outlives the request */
    ctx->cached_uri.len = r->uri.len;
    ctx->cached_uri.data = ngx_pnalloc(c->pool, r->uri.len);
    if (ctx->cached_uri.data == NULL) {
        return NGX_ERROR;
    }
    ngx_memcpy(ctx->cached_uri.data, r->uri.data, r->uri.len);

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    return NGX_DECLINED;
}

static ngx_int_t
ngx_http_mymodule_log_handler(ngx_http_request_t *r)
{
    my_conn_ctx_t  *ctx;

    ctx = ngx_http_get_module_ctx(r, ngx_http_mymodule_module);

    /* SAFE: cached_uri.data lives in c->pool, valid across keepalive */
    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "previous uri: %V", &ctx->cached_uri);

    return NGX_OK;
}

Reference: nginx Development Guide — Memory Pools

Use Valgrind Pool-Level Tracing to Find Leaked Allocations

Standard Valgrind cannot track nginx pool allocations because ngx_palloc internally manages a large mmap'd block. All pool allocations appear as a single large block, making it impossible to identify which ngx_palloc call leaks. Recompile nginx with NGX_DEBUG_PALLOC defined to force each ngx_palloc call to use malloc directly, making every allocation visible to Valgrind with a full backtrace to the exact source line.

Incorrect (runs Valgrind on standard nginx build, showing one opaque allocation):

/*
 * Standard build — configure and run under Valgrind:
 *
 *   ./configure --with-debug
 *   make
 *   valgrind --leak-check=full ./objs/nginx -g 'daemon off; master_process off;'
 *
 * Valgrind output shows:
 *
 *   ==12345== 262,144 bytes in 1 blocks are possibly lost
 *      at 0x4C2FB0F: malloc (in /usr/lib/valgrind/...)
 *      by 0x44A3E2: ngx_alloc (ngx_alloc.c:22)
 *      by 0x44B1C7: ngx_palloc_block (ngx_palloc.c:132)
 *      by 0x44B0D3: ngx_palloc (ngx_palloc.c:98)
 *
 * BUG: no visibility into WHICH ngx_palloc call leaked —
 * all allocations collapse into the pool's internal block.
 * Cannot distinguish the leaking module allocation from
 * hundreds of legitimate pool allocations.
 */

Correct (compiles with NGX_DEBUG_PALLOC to make each allocation a separate malloc):

/*
 * Debug build — define NGX_DEBUG_PALLOC so each ngx_palloc calls malloc:
 *
 *   ./configure --with-debug --with-cc-opt='-DNGX_DEBUG_PALLOC=1'
 *   make
 *   valgrind --leak-check=full --show-leak-kinds=all \
 *     ./objs/nginx -g 'daemon off; master_process off;'
 *
 * Valgrind output now shows EACH allocation separately:
 *
 *   ==12345== 4,096 bytes in 1 blocks are definitely lost
 *      at 0x4C2FB0F: malloc (in /usr/lib/valgrind/...)
 *      by 0x44A3E2: ngx_alloc (ngx_alloc.c:22)
 *      by 0x44B0D3: ngx_palloc (ngx_palloc.c:68)
 *      by 0x4823AF: ngx_http_mymodule_handler (ngx_http_mymodule.c:147)
 *      by 0x457C21: ngx_http_core_content_phase (ngx_http_core_module.c:1261)
 *
 * SAFE: exact source file and line number of the leaking allocation.
 * Combine with --track-origins=yes to trace uninitialized value origins.
 *
 * IMPORTANT: NGX_DEBUG_PALLOC disables pool block reuse, so memory usage
 * is higher and performance is lower — use only for debugging, never
 * in production.
 */

See also: `build-debug-palloc` for the canonical NGX_DEBUG_PALLOC build configuration and ASan integration.

Reference: nginx Development Guide — Memory Pools

Trace Request Processing with DTrace pid Provider

DTrace's pid provider hooks into any function in a running process without recompilation or restart. On systems where DTrace is available (macOS, FreeBSD, Solaris, illumos), you can trace nginx request processing by probing function entries and returns, reading struct fields from function arguments, and computing per-function latency. This avoids rebuilding nginx with --with-debug (which changes timing and adds overhead), making it suitable for investigating production issues that only reproduce under real load.

Incorrect (rebuilding nginx with --with-debug just to trace request flow):

# BAD: full rebuild cycle to add tracing
$ ./configure --with-debug --add-module=../my_module
$ make && make install

# Restart nginx with debug logging
$ nginx -s stop
$ nginx

# Problems with this approach:
# 1. Requires service restart — drops active connections
# 2. --with-debug adds overhead to ALL code paths, not just
#    the functions you want to trace
# 3. Changes timing behavior — race conditions may disappear
# 4. Debug binary is ~30% larger, more instruction cache misses
# 5. Must revert and rebuild again for production
# 6. Cannot trace functions you didn't instrument with
#    ngx_log_debug calls at compile time

# Also BAD: manually adding printf/ngx_log_debug calls
static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    /* Temporary debug printf — must rebuild to add/remove */
    fprintf(stderr, "handler called for %.*s\n",
            (int) r->uri.len, r->uri.data);
    /* ... */
}

Correct (using DTrace pid provider to attach to running worker without rebuild):

# Step 1: Find the worker PID
$ ps aux | grep 'nginx: worker'
www       4521  ...  nginx: worker process

# Step 2: Create a DTrace script to trace request processing
$ cat > trace_request.d << 'DTRACE_SCRIPT'
#!/usr/sbin/dtrace -s

/* WARNING: struct offsets (0xb8, 0xc0 below) are examples only.
 * They change with every nginx version, compile flags, and platform.
 * Find YOUR offsets first:
 *   pahole -C ngx_http_request_s ./objs/nginx | grep -A1 uri
 *   gdb -batch ./objs/nginx -ex 'ptype /o ngx_http_request_s' */

/* Trace ngx_http_process_request — entry point for
 * every HTTP request after headers are parsed.
 * arg0 = ngx_http_request_t *r */
pid$target::ngx_http_process_request:entry
{
    self->req = arg0;
    self->start = timestamp;

    /* Replace 0xb8/0xc0 with YOUR offsets from pahole/gdb */
    this->uri_len = *(uint32_t *)copyin(arg0 + 0xb8, 4);
    this->uri_data = *(uintptr_t *)copyin(arg0 + 0xc0, 8);

    printf("REQUEST START: uri=%.*s\n",
           this->uri_len < 128 ? this->uri_len : 128,
           copyinstr(this->uri_data, this->uri_len < 128
                     ? this->uri_len : 128));
}

/* Trace your custom module handler */
pid$target::ngx_http_mymodule_handler:entry
{
    self->mod_start = timestamp;
    printf("  MYMODULE HANDLER ENTRY\n");
}

pid$target::ngx_http_mymodule_handler:return
{
    this->elapsed = (timestamp - self->mod_start) / 1000;
    printf("  MYMODULE HANDLER EXIT: %d μs, rc=%d\n",
           this->elapsed, arg1);
}

/* Trace request finalization */
pid$target::ngx_http_finalize_request:entry
{
    this->total = (timestamp - self->start) / 1000;
    printf("REQUEST FINALIZE: rc=%d total=%d μs\n",
           arg1, this->total);
    self->req = 0;
    self->start = 0;
}
DTRACE_SCRIPT

# Step 3: Run the DTrace script against the worker
$ sudo dtrace -s trace_request.d -p 4521

# Output (live, no restart required):
# REQUEST START: uri=/api/v1/users
#   MYMODULE HANDLER ENTRY
#   MYMODULE HANDLER EXIT: 42 μs, rc=0
# REQUEST FINALIZE: rc=200 total=187 μs
# REQUEST START: uri=/api/v1/health
#   MYMODULE HANDLER ENTRY
#   MYMODULE HANDLER EXIT: 3 μs, rc=-5
# REQUEST FINALIZE: rc=200 total=28 μs

Reference: DTrace pid Provider Documentation

Measure Per-Function Latency with eBPF Probes

eBPF (via bpftrace or bcc) attaches kernel-level probes to userspace functions with near-zero overhead when probes are not firing. Unlike strace (which intercepts every syscall via ptrace, adding microseconds per call), eBPF runs instrumentation code in-kernel and only activates on the specific functions you target. This makes it safe for production latency measurement. Use uprobes to attach to function entry/exit, compute duration in nanoseconds, and aggregate into histograms that show the full latency distribution, not just averages.

Incorrect (using strace -T for function-level latency measurement):

# BAD: strace measures SYSCALL latency, not function latency.
# It cannot time your module's handler function directly.
# It only shows time spent inside system calls.
$ strace -p 8402 -T -e trace=read,write 2>&1 | head -10
# read(12, "GET /api HTTP/1.1\r\n"..., 4096) = 285 <0.000013>
# write(12, "HTTP/1.1 200 OK\r\n"..., 512) = 512 <0.000009>

# Problems:
# 1. strace uses ptrace — adds ~10-50μs overhead PER syscall
# 2. Under load, this overhead accumulates: a worker doing
#    50,000 syscalls/sec gets 0.5-2.5 seconds of added latency
#    per second — fundamentally changing the behavior
# 3. Cannot measure time spent in user-space functions between
#    syscalls (your module's parsing, auth checking, etc.)
# 4. strace stops the process on every syscall (SIGSTOP/CONT),
#    disrupting epoll timing and masking race bugs
# 5. Cannot aggregate into histograms — only raw per-call times

# BAD: timing with clock_gettime in the module code
static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    struct timespec start, end;
    /* Requires code change and rebuild */
    clock_gettime(CLOCK_MONOTONIC, &start);
    /* ... handler logic ... */
    clock_gettime(CLOCK_MONOTONIC, &end);
    /* Overhead of two syscalls per request */
}

Correct (using bpftrace to measure function latency with near-zero overhead):

# Step 1: Find the nginx binary and worker PID
$ NGINX_BIN=$(which nginx || echo /usr/local/nginx/sbin/nginx)
$ WORKER_PID=$(pgrep -f 'nginx: worker' | head -1)

# Step 2: Measure handler latency distribution with bpftrace
# uprobes attach at function entry/exit without modifying the binary
$ sudo bpftrace -p $WORKER_PID -e '
uprobe:'"$NGINX_BIN"':ngx_http_mymodule_handler
{
    @start[tid] = nsecs;
}

uretprobe:'"$NGINX_BIN"':ngx_http_mymodule_handler
/@start[tid]/
{
    $duration_us = (nsecs - @start[tid]) / 1000;
    @handler_latency_us = hist($duration_us);
    @total_calls = count();
    delete(@start[tid]);
}

interval:s:10
{
    printf("\n--- Handler Latency Distribution (μs) ---\n");
    print(@handler_latency_us);
    printf("Total calls: ");
    print(@total_calls);
    clear(@handler_latency_us);
    clear(@total_calls);
}
'

# Output every 10 seconds:
# --- Handler Latency Distribution (μs) ---
# @handler_latency_us:
# [1]          1523 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  |
# [2, 4)        487 |@@@@@@@@@@@@                            |
# [4, 8)         93 |@@                                      |
# [8, 16)        12 |                                        |
# [16, 32)        3 |                                        |  <-- outliers
# [32, 64)        1 |                                        |
# Total calls: 2119

# Step 3: Compare two functions to find the bottleneck
$ sudo bpftrace -p $WORKER_PID -e '
uprobe:'"$NGINX_BIN"':ngx_http_mymodule_parse_headers {
    @parse_start[tid] = nsecs;
}
uretprobe:'"$NGINX_BIN"':ngx_http_mymodule_parse_headers
/@parse_start[tid]/ {
    @parse_us = hist((nsecs - @parse_start[tid]) / 1000);
    delete(@parse_start[tid]);
}

uprobe:'"$NGINX_BIN"':ngx_http_mymodule_auth_check {
    @auth_start[tid] = nsecs;
}
uretprobe:'"$NGINX_BIN"':ngx_http_mymodule_auth_check
/@auth_start[tid]/ {
    @auth_us = hist((nsecs - @auth_start[tid]) / 1000);
    delete(@auth_start[tid]);
}

interval:s:30 { exit(); }
'
# Compare the two histograms to identify which function
# contributes most to total handler latency.

Reference: bpftrace Reference Guide

Detect File Descriptor Leaks with strace and /proc

File descriptor leaks in nginx modules manifest as EMFILE errors ("too many open files") after prolonged runtime. Each leaked fd (socket, file, pipe) remains open until the worker process exits, gradually consuming the per-process fd limit. Diagnosing the leak requires three steps: (1) counting open fds per worker over time via /proc/PID/fd to confirm the leak exists and measure its rate, (2) attaching strace with -e trace=open,close,socket to find fds that are opened without a matching close, (3) correlating leaked fd numbers with /proc/PID/fdinfo entries to identify the type and target. Simply increasing worker_rlimit_nofile masks the symptom while the leak continues.

Incorrect (increasing worker_rlimit_nofile to hide the leak):

# nginx.conf — BAD: raising the limit to mask the leak
# The module leaks 1 fd per request to /tmp/mymodule_cache_*.
# At 100 req/s, this consumes 360,000 fds per hour.
# Raising the limit only delays the inevitable EMFILE.

worker_rlimit_nofile 1048576;  # was 65536, now 1M

events {
    worker_connections 65536;
}

# The worker will still eventually hit 1M open fds.
# Memory usage grows because each fd has kernel buffers.
# /proc/PID/fd listing becomes extremely slow.
# Performance degrades as the kernel manages 1M+ fds.

/* The leaking module code — opens a cache file but never closes it
 * when the request pool is destroyed without a cleanup handler */
static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_fd_t  fd;
    u_char    path[256];

    ngx_snprintf(path, sizeof(path), "/tmp/mymodule_cache_%V%Z",
                 &r->uri);

    fd = ngx_open_file(path, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        return NGX_DECLINED;
    }

    /* BUG: fd is stored but never closed.
     * No ngx_pool_cleanup_add to close the fd when the
     * request completes. */
    /* ... read from fd ... */

    return NGX_OK;
    /* Request finishes, pool is destroyed, but fd stays open */
}

Correct (diagnosing the fd leak with /proc and strace, then fixing with cleanup):

# Step 1: Confirm the leak exists — count fds over time
$ WORKER_PID=$(pgrep -f 'nginx: worker' | head -1)
$ while true; do
    count=$(ls /proc/$WORKER_PID/fd 2>/dev/null | wc -l)
    echo "$(date +%H:%M:%S) fds=$count"
    sleep 5
  done
# 14:00:00 fds=42
# 14:00:05 fds=47
# 14:00:10 fds=53   <-- growing = leak confirmed
# 14:00:15 fds=58

# Step 2: Identify what's being leaked via /proc/PID/fd
$ ls -la /proc/$WORKER_PID/fd | tail -20
# ... normal sockets and pipes ...
# lrwx------ 1 www www 64 ... 55 -> /tmp/mymodule_cache_api_users
# lrwx------ 1 www www 64 ... 56 -> /tmp/mymodule_cache_api_health
# lrwx------ 1 www www 64 ... 57 -> /tmp/mymodule_cache_api_items
# Pattern: /tmp/mymodule_cache_* files accumulating

# Step 3: Trace open/close to find unmatched opens
$ strace -p $WORKER_PID -e trace=openat,close -T \
    2>&1 | tee /tmp/fd-trace.log &
# Let it run for 30 seconds, then:
$ kill %1

# Step 4: Find fds opened but never closed
$ awk '
  /openat.*mymodule_cache/ && /= [0-9]/ {
    match($0, /= ([0-9]+)/, m); opened[m[1]]++
  }
  /^close\(([0-9]+)\)/ {
    match($0, /close\(([0-9]+)\)/, m); closed[m[1]]++
  }
  END {
    for (fd in opened)
      if (!(fd in closed))
        printf "LEAKED fd=%s (opened %d times, never closed)\n",
               fd, opened[fd]
  }
' /tmp/fd-trace.log
# LEAKED fd=58 (opened 1 times, never closed)
# LEAKED fd=59 (opened 1 times, never closed)

/* Fixed: register a cleanup handler to close the fd */
static void
ngx_http_mymodule_cleanup_fd(void *data)
{
    ngx_fd_t *fdp = data;

    if (*fdp != NGX_INVALID_FILE) {
        ngx_close_file(*fdp);
    }
}

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_fd_t            fd;
    ngx_fd_t           *fdp;
    ngx_pool_cleanup_t *cln;
    u_char              path[256];

    ngx_snprintf(path, sizeof(path), "/tmp/mymodule_cache_%V%Z",
                 &r->uri);

    fd = ngx_open_file(path, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        return NGX_DECLINED;
    }

    /* Register cleanup BEFORE using the fd */
    cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_fd_t));
    if (cln == NULL) {
        ngx_close_file(fd);
        return NGX_ERROR;
    }

    fdp = cln->data;
    *fdp = fd;
    cln->handler = ngx_http_mymodule_cleanup_fd;

    /* ... read from fd ... */
    /* fd is automatically closed when request pool is destroyed */

    return NGX_OK;
}

Reference: proc(5) — /proc/PID/fd

Use strace to Trace System Call Patterns in nginx Workers

strace attaches to a running worker process and shows every system call with arguments, return values, and timing. It is invaluable for diagnosing: unexpected blocking calls (disk I/O in an async path, synchronous DNS resolution), socket errors that nginx logs generically, and I/O patterns that reveal performance bottlenecks. The key is attaching to a worker PID (not master) and filtering with -e trace= to limit output to relevant syscall categories. Running strace on the master process shows only signal handling and waitpid -- no request processing.

Incorrect (running strace on the master process, which never handles requests):

# Find nginx processes
$ ps aux | grep nginx
root      8401  ...  nginx: master process /usr/sbin/nginx
www-data  8402  ...  nginx: worker process
www-data  8403  ...  nginx: worker process

# BAD: attaching to the master PID
$ strace -p 8401

# Output shows only master process activity:
# rt_sigsuspend([], 8)              = ? ERESTARTNOHAND
# --- SIGCHLD {si_signo=SIGCHLD, ...} ---
# waitpid(-1, [{WIFEXITED(s)}], WNOHANG) = 0
# rt_sigsuspend([], 8)              = ? ERESTARTNOHAND
# ...
#
# The master process sits in sigsuspend() waiting for signals.
# You will NEVER see accept(), recv(), send(), epoll_wait(),
# or any request-handling syscalls.
# Modules' file operations, socket I/O, and upstream connections
# are all invisible from here.

# BAD: no filtering — massive output even on the right PID
$ strace -p 8402
# Produces thousands of lines per second including
# clock_gettime, gettimeofday, epoll_wait polls, etc.
# The actual bug-relevant syscalls are buried in noise.

Correct (attaching to a worker PID with syscall filtering and timing):

# Step 1: Identify the correct worker PID
$ ps aux | grep 'nginx: worker'
www-data  8402  ...  nginx: worker process
www-data  8403  ...  nginx: worker process

# Step 2: Trace network syscalls with timing (-T shows duration)
# -f follows child threads, -e filters to network category
$ strace -p 8402 -e trace=network -T 2>&1 | head -30
# accept4(7, {sa_family=AF_INET, sin_port=htons(54321),
#   sin_addr=inet_addr("10.0.0.1")}, [128->16], SOCK_NONBLOCK) = 12
#   <0.000021>
# setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
# recvfrom(12, "GET /api/v1/users HTTP/1.1\r\n"..., 4096, 0,
#   NULL, NULL) = 285 <0.000013>
# connect(13, {sa_family=AF_INET, sin_port=htons(8080),
#   sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS
#   <0.000045>

# Step 3: Trace file I/O to find unexpected blocking disk access
$ strace -p 8402 -e trace=file -T 2>&1 | head -20
# openat(AT_FDCWD, "/var/cache/nginx/proxy/abc123",
#   O_RDONLY) = 14 <0.000350>
# --- A 350μs open in an event loop iteration is suspicious ---

# Step 4: Trace with full timestamps for correlation with logs
$ strace -p 8402 -e trace=network,write -tt -T \
    -o /tmp/nginx-worker-trace.log
# -tt: microsecond timestamps
# -T: syscall duration
# -o: write to file (avoids terminal I/O interfering with nginx)

# Step 5: Look for blocking patterns in the trace output
$ grep -E '<[0-9]+\.[0-9]{3,}>' /tmp/nginx-worker-trace.log
# Any syscall taking >1ms in an event-driven worker is a red flag.
# Common culprits: DNS resolution, disk reads, file locking

Reference: strace(1) Manual

Track Request Reference Count to Debug Premature Destruction

The r->main->count field tracks how many async operations reference a request. When ngx_http_finalize_request is called and r->main->count drops to zero, the request is destroyed and its memory pool freed. If count is too low, a finalize destroys the request while an async operation is still pending (use-after-free). If count is too high, the request is never freed (connection leak). Logging r->main->count at each transition reveals exactly where the count diverges from expectations.

Note: Functions like ngx_http_read_client_request_body() and ngx_http_subrequest() increment r->main->count internally. Do NOT manually increment before calling them — double-incrementing causes request leaks.

Incorrect (no r->count logging, count mismatch is invisible):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_http_mymodule_ctx_t  *ctx;

    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_mymodule_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    r->request_body_in_single_buf = 1;

    /* BUG: no logging of r->count — when body read completes
     * and the request is destroyed prematurely (or leaks),
     * there is no trace showing where count went wrong */
    ngx_http_read_client_request_body(r,
        ngx_http_mymodule_body_handler);

    return NGX_DONE;
}

static void
ngx_http_mymodule_body_handler(ngx_http_request_t *r)
{
    ngx_http_mymodule_process_body(r);
    ngx_http_finalize_request(r, NGX_OK);
}

Correct (logging r->count at each transition to trace count lifecycle):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_int_t                 rc;
    ngx_http_mymodule_ctx_t  *ctx;

    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_mymodule_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: handler entry, r->count=%d",
                   r->main->count);

    r->request_body_in_single_buf = 1;

    /* ngx_http_read_client_request_body increments r->main->count
     * internally — do not increment manually */
    rc = ngx_http_read_client_request_body(r,
             ngx_http_mymodule_body_handler);

    ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: after body read, r->count=%d rc=%i",
                   r->main->count, rc);

    if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
        return rc;
    }

    return NGX_DONE;
}

static void
ngx_http_mymodule_body_handler(ngx_http_request_t *r)
{
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: body handler entry, r->count=%d",
                   r->main->count);

    ngx_http_mymodule_process_body(r);

    /* ngx_http_finalize_request decrements r->main->count.
     * Log before finalizing to trace the decrement. */
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "mymodule: finalizing, r->count=%d (will decrement)",
                   r->main->count);

    ngx_http_finalize_request(r, NGX_OK);
}

Reference: nginx Development Guide — HTTP Request