Nginx C Module Perf

nginx.org C Module Performance & Reliability Best Practices

Comprehensive performance optimization and reliability guide for nginx C modules, derived from the official nginx development documentation and production engineering experience. Contains 43 rules across 8 categories, prioritized by impact to guide automated optimization and resilience improvements.

Companion skill: This skill complements nginx-c-modules which covers correctness (memory safety, request lifecycle, configuration). This skill covers performance optimization and operational reliability.

When to Apply

Reference these guidelines when:

• Optimizing nginx C module throughput and latency
• Reducing buffer copies and enabling zero-copy I/O paths
• Tuning connection pooling and socket options
• Minimizing shared memory lock contention across workers
• Implementing graceful error recovery and fallback responses
• Configuring upstream timeouts and retry strategies
• Building in-module response caches with shared memory
• Tuning worker process behavior under load

Rule Categories by Priority

Quick Reference

1. Buffer & Zero-Copy I/O (CRITICAL)

• `buf-chain-reuse` - Reuse Buffer Chain Links Instead of Allocating New Ones
• `buf-file-sendfile` - Use File Buffers for Static Content Instead of Reading into Memory
• `buf-avoid-copy` - Avoid Copying Buffers When Passing Through Filter Chain
• `buf-coalesce-small` - Coalesce Small Buffers Before Output
• `buf-shadow-reference` - Use Shadow Buffers for Derived Data Instead of Full Copies
• `buf-recycled-flag` - Mark Buffers as Recycled for Upstream Response Reuse

2. Connection Efficiency (CRITICAL)

• `conn-reusable-queue` - Mark Idle Connections as Reusable for Pool Recovery
• `conn-drain-pressure` - Handle Connection Drain Under Memory Pressure
• `conn-tcp-nodelay` - Control TCP_NODELAY for Latency-Sensitive Responses
• `conn-prealloc-pool` - Size Connection Pool to Avoid Runtime Reallocation
• `conn-close-linger` - Use Lingering Close for Graceful Connection Shutdown
• `conn-ssl-session-reuse` - Enable SSL Session Caching in Upstream Connections

3. Lock Contention & Atomics (HIGH)

• `lock-minimize-critical` - Minimize Critical Section Duration in Shared Memory
• `lock-atomic-counters` - Use Atomic Operations for Simple Counters Instead of Mutex
• `lock-trylock-fallback` - Use ngx_shmtx_trylock with Fallback to Avoid Worker Stalls
• `lock-per-worker-aggregate` - Aggregate Per-Worker Counters to Reduce Shared Memory Access
• `lock-alloc-outside` - Perform Slab Allocation Outside Hot Path
• `lock-rw-pattern` - Use Read-Copy-Update Pattern for Read-Heavy Shared Data

4. Error Recovery & Resilience (HIGH)

• `err-cache-errno` - Cache ngx_errno Immediately to Prevent Overwrite
• `err-fallback-response` - Return Fallback Response When Upstream Fails
• `err-resource-exhaustion` - Handle Pool and Slab Allocation Exhaustion Gracefully
• `err-blocked-counter` - Use Blocked Counter to Prevent Premature Request Destruction
• `err-connection-error-check` - Check Connection Error Flag Before I/O Operations
• `err-log-once-pattern` - Limit Repeated Error Logging to Prevent Log Storms

5. Timeout & Retry Strategy (MEDIUM-HIGH)

• `timeout-upstream-phases` - Set Separate Timeouts for Connect, Send, and Read Phases
• `timeout-retry-next-upstream` - Configure next_upstream Mask for Retriable Failures
• `timeout-backoff-reconnect` - Use Exponential Backoff for Upstream Reconnection Attempts
• `timeout-client-body-limit` - Set Client Body Timeout to Bound Slow-Client Resource Usage

6. Response Caching (MEDIUM)

• `cache-shm-lru` - Implement LRU Eviction in Shared Memory Cache Zones
• `cache-stampede-lock` - Prevent Cache Stampede with Single-Flight Pattern
• `cache-key-hash` - Use ngx_hash for Fixed Cache Key Lookups
• `cache-ttl-atomic` - Use Atomic Timestamp Comparison for TTL Expiry Checks
• `cache-conditional-store` - Cache Only Successful Responses to Avoid Negative Cache Pollution

7. Worker & Process Tuning (MEDIUM)

• `worker-accept-mutex` - Understand Accept Mutex Impact on Connection Distribution
• `worker-connection-prealloc` - Use Pre-Allocated Free List for Module Data Structures
• `worker-graceful-shutdown` - Handle Worker Shutdown Signal Without Data Loss
• `worker-single-process-debug` - Support Single-Process Mode for Debugging
• `worker-cycle-conf` - Access Configuration Through Cycle for Process-Level Operations

8. Logging & Metrics (LOW-MEDIUM)

• `log-level-guard` - Guard Expensive Debug Argument Computation Behind Level Check
• `log-connection-context` - Attach Module Context to Connection Log for Tracing
• `log-shared-metrics` - Collect Metrics via Shared Memory Counters
• `log-error-dedup` - Deduplicate Repeated Error Messages with Throttling
• `log-action-string` - Set Log Action String for Operation Context

How to Use

Read individual reference files for detailed explanations and code examples:

• Section definitions - Category structure and impact levels
• Rule template - Template for adding new rules

Reference Files

{Rule Title}

{1-3 sentences explaining WHY this matters for performance or reliability. Focus on measurable impact.}

Incorrect ({what's wrong}):

{Bad code example - production-realistic, not strawman}
{/* Comments explaining the cost */}

Correct ({what's right}):

{Good code example - minimal diff from incorrect}
{/* Comments explaining the benefit */}

Note: {Optional additional context, alternatives, or caveats.}

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

---

1. Buffer & Zero-Copy I/O (buf)

Impact: CRITICAL Description: Buffer chain reuse and zero-copy patterns are the #1 throughput multiplier — unnecessary memory copies waste CPU cycles on every byte of every request.

2. Connection Efficiency (conn)

Impact: CRITICAL Description: Connection pooling, reuse queue management, and socket tuning directly determine maximum concurrent connections per worker process.

3. Lock Contention & Atomics (lock)

Impact: HIGH Description: Shared memory lock contention is the primary scaling bottleneck in multi-worker deployments — minimizing critical sections yields linear worker scaling.

4. Error Recovery & Resilience (err)

Impact: HIGH Description: Graceful error propagation and resource exhaustion recovery prevent cascading failures that turn partial outages into full service loss.

5. Timeout & Retry Strategy (timeout)

Impact: MEDIUM-HIGH Description: Correct timeout tuning and retry patterns determine whether upstream failures cascade across the request pipeline or stay isolated.

6. Response Caching (cache)

Impact: MEDIUM Description: In-module caching with shared memory zones eliminates redundant upstream calls — cache stampede prevention is critical at scale.

7. Worker & Process Tuning (worker)

Impact: MEDIUM Description: Accept mutex control, connection pre-allocation, and graceful shutdown patterns determine system behavior under load spikes and deployments.

8. Logging & Metrics (log)

Impact: LOW-MEDIUM Description: Efficient logging avoids becoming a bottleneck — debug logging without level checks costs ~50ns per call even when disabled at runtime.

Avoid Copying Buffers When Passing Through Filter Chain

Filters that inspect but do not modify data should pass the original buffer chain through unchanged. Copying buffer contents into a newly allocated buffer at each filter stage multiplies memory usage and CPU cost by the number of filters in the chain. For a 1 MB response passing through 5 filters, unnecessary copies waste 5 MB of allocation and 5 MB of memcpy — per request.

Incorrect (copies buffer data into a new buffer before passing downstream):

static ngx_int_t
ngx_http_myfilter_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_chain_t  *cl, *out, **ll;
    ngx_buf_t    *b;
    size_t        len;

    out = NULL;
    ll = &out;

    for (cl = in; cl; cl = cl->next) {
        len = ngx_buf_size(cl->buf);
        if (len == 0) {
            continue;
        }

        /* SLOW: full copy of every buffer just to inspect it */
        b = ngx_create_temp_buf(r->pool, len);
        if (b == NULL) {
            return NGX_ERROR;
        }

        ngx_memcpy(b->pos, cl->buf->pos, len);
        b->last = b->pos + len;
        b->last_buf = cl->buf->last_buf;
        b->last_in_chain = cl->buf->last_in_chain;

        ngx_http_myfilter_inspect(b->pos, len);

        *ll = ngx_alloc_chain_link(r->pool);
        if (*ll == NULL) {
            return NGX_ERROR;
        }

        (*ll)->buf = b;
        (*ll)->next = NULL;
        ll = &(*ll)->next;
    }

    return ngx_http_next_body_filter(r, out);
}

Correct (inspects original buffers in-place and passes chain through unmodified):

static ngx_int_t
ngx_http_myfilter_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_chain_t  *cl;
    size_t        len;

    for (cl = in; cl; cl = cl->next) {
        if (cl->buf->in_file) {
            continue;
        }

        len = cl->buf->last - cl->buf->pos;
        if (len > 0) {
            /* read-only inspection — no copy needed */
            ngx_http_myfilter_inspect(cl->buf->pos, len);
        }
    }

    /* pass original chain through — zero allocation, zero memcpy */
    return ngx_http_next_body_filter(r, in);
}

Reuse Buffer Chain Links Instead of Allocating New Ones

Every ngx_chain_t link allocated with ngx_alloc_chain_link can be returned to the connection's free list via ngx_free_chain after use. Allocating fresh chain links in a loop — especially inside body filters called thousands of times per request — fragments the pool and wastes cycles on allocator bookkeeping. Reusing links from the pool's free list is an O(1) pointer swap with zero allocator overhead.

Incorrect (allocates a new chain link on every filter invocation):

static ngx_int_t
ngx_http_myfilter_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_chain_t  *cl, *out, **ll;

    out = NULL;
    ll = &out;

    for (cl = in; cl; cl = cl->next) {
        /* BUG: allocates from pool every iteration — never reused */
        *ll = ngx_palloc(r->pool, sizeof(ngx_chain_t));
        if (*ll == NULL) {
            return NGX_ERROR;
        }

        (*ll)->buf = cl->buf;
        (*ll)->next = NULL;
        ll = &(*ll)->next;
    }

    return ngx_http_next_body_filter(r, out);
}

Correct (uses ngx_alloc_chain_link and returns links via ngx_free_chain):

static ngx_int_t
ngx_http_myfilter_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_chain_t  *cl, *out, *ln, **ll;

    out = NULL;
    ll = &out;

    for (cl = in; cl; cl = cl->next) {
        /* O(1) — pops from connection free list, falls back to pool only if empty */
        ln = ngx_alloc_chain_link(r->pool);
        if (ln == NULL) {
            return NGX_ERROR;
        }

        ln->buf = cl->buf;
        ln->next = NULL;
        *ll = ln;
        ll = &ln->next;
    }

    ngx_int_t rc = ngx_http_next_body_filter(r, out);

    /* return all chain links to the free list for reuse */
    for (cl = out; cl; /* void */) {
        ln = cl->next;
        ngx_free_chain(r->pool, cl);
        cl = ln;
    }

    return rc;
}

Coalesce Small Buffers Before Output

Sending many small buffers individually causes one writev() or send() syscall per buffer — each syscall costs ~1-5 microseconds of kernel overhead. For a response built from 50 small fragments, that is 50 syscalls instead of 1. Coalescing adjacent memory buffers into a single output buffer before calling the output filter collapses the syscall count and allows the kernel to send a single large TCP segment.

Incorrect (sends each small fragment as a separate chain link):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_chain_t  *out, *cl, **ll;
    ngx_buf_t    *b;
    ngx_uint_t    i;
    ngx_int_t     rc;

    rc = ngx_http_send_header(r);
    if (rc == NGX_ERROR || rc > NGX_OK) {
        return rc;
    }

    out = NULL;
    ll = &out;

    /* SLOW: 50 tiny buffers = 50 potential syscalls */
    for (i = 0; i < r->headers_out.count; i++) {
        b = ngx_create_temp_buf(r->pool, 64);
        if (b == NULL) {
            return NGX_ERROR;
        }

        b->last = ngx_sprintf(b->pos, "item_%ui\n", i);
        b->memory = 1;

        cl = ngx_alloc_chain_link(r->pool);
        if (cl == NULL) {
            return NGX_ERROR;
        }

        cl->buf = b;
        cl->next = NULL;
        *ll = cl;
        ll = &cl->next;
    }

    if (out) {
        out->buf->last_buf = 1;
    }

    return ngx_http_output_filter(r, out);
}

Correct (pre-calculates total size and writes into a single coalesced buffer):

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_buf_t    *b;
    ngx_chain_t   out;
    ngx_uint_t    i, n;
    ngx_int_t     rc;
    u_char       *p;
    size_t        total;

    n = r->headers_out.count;

    /* first pass: calculate total output size */
    total = 0;
    for (i = 0; i < n; i++) {
        total += sizeof("item_\n") - 1 + NGX_INT_T_LEN;
    }

    rc = ngx_http_send_header(r);
    if (rc == NGX_ERROR || rc > NGX_OK) {
        return rc;
    }

    /* single allocation for entire response body */
    b = ngx_create_temp_buf(r->pool, total);
    if (b == NULL) {
        return NGX_ERROR;
    }

    /* second pass: write all fragments into one buffer */
    p = b->pos;
    for (i = 0; i < n; i++) {
        p = ngx_sprintf(p, "item_%ui\n", i);
    }

    b->last = p;
    b->memory = 1;
    b->last_buf = (r == r->main) ? 1 : 0;
    b->last_in_chain = 1;

    out.buf = b;
    out.next = NULL;

    /* single output filter call = single writev() syscall */
    return ngx_http_output_filter(r, &out);
}

Use File Buffers for Static Content Instead of Reading into Memory

When serving file-backed content, setting in_file = 1 with file_pos and file_last on ngx_buf_t lets nginx use the kernel's sendfile() syscall to transfer data directly from the page cache to the socket — zero CPU copies. Reading the file into a memory buffer first doubles memory usage and forces an extra kernel-to-userspace-to-kernel copy on every byte, destroying throughput for large files.

Incorrect (reads entire file into a memory buffer before sending):

static ngx_int_t
ngx_http_mymodule_send_file(ngx_http_request_t *r, ngx_str_t *path)
{
    ngx_buf_t         *b;
    ngx_chain_t        out;
    ngx_file_info_t    fi;
    ssize_t            n;
    ngx_fd_t           fd;
    size_t             size;

    fd = ngx_open_file(path->data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        return NGX_HTTP_NOT_FOUND;
    }

    ngx_fd_info(fd, &fi);
    size = (size_t) ngx_file_size(&fi);

    /* SLOW: allocates response-sized buffer and copies file into userspace */
    b = ngx_create_temp_buf(r->pool, size);
    if (b == NULL) {
        ngx_close_file(fd);
        return NGX_ERROR;
    }

    n = ngx_read_fd(fd, b->pos, size);
    ngx_close_file(fd);

    if (n == NGX_FILE_ERROR) {
        return NGX_ERROR;
    }

    b->last = b->pos + n;
    b->memory = 1;
    b->last_buf = 1;

    out.buf = b;
    out.next = NULL;

    return ngx_http_output_filter(r, &out);
}

Correct (sets file buffer fields for kernel-level sendfile zero-copy):

static ngx_int_t
ngx_http_mymodule_send_file(ngx_http_request_t *r, ngx_str_t *path)
{
    ngx_buf_t         *b;
    ngx_chain_t        out;
    ngx_open_file_info_t  of;

    ngx_memzero(&of, sizeof(ngx_open_file_info_t));
    of.read_ahead = 0;
    of.directio = NGX_OPEN_FILE_DIRECTIO_OFF;

    if (ngx_open_cached_file(r->connection->log, path, &of, r->pool)
        != NGX_OK)
    {
        return NGX_HTTP_NOT_FOUND;
    }

    b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
    if (b == NULL) {
        return NGX_ERROR;
    }

    /* zero-copy: kernel sends directly from page cache via sendfile() */
    b->in_file = 1;
    b->file_pos = 0;
    b->file_last = of.size;
    b->file = ngx_pcalloc(r->pool, sizeof(ngx_file_t));
    if (b->file == NULL) {
        return NGX_ERROR;
    }

    b->file->fd = of.fd;
    b->file->name = *path;
    b->file->log = r->connection->log;
    b->last_buf = (r == r->main) ? 1 : 0;
    b->last_in_chain = 1;

    r->headers_out.status = NGX_HTTP_OK;
    r->headers_out.content_length_n = of.size;

    out.buf = b;
    out.next = NULL;

    ngx_http_send_header(r);

    return ngx_http_output_filter(r, &out);
}

Mark Buffers as Recycled for Upstream Response Reuse

When proxying upstream responses, nginx maintains a buffer pool (p->free and p->busy lists) to avoid allocating new buffers for every chunk of upstream data. Setting b->recycled = 1 on buffers returned from input_filter tells the upstream machinery that this buffer can be returned to the free pool once its contents have been sent downstream. Without this flag, buffers accumulate on the busy list and are never reused — forcing a new allocation for every upstream read, which under sustained load exhausts the connection pool and triggers expensive pool growth.

Incorrect (upstream buffers never returned to free pool):

static ngx_int_t
ngx_http_myupstream_input_filter(void *data, ssize_t bytes)
{
    ngx_http_request_t  *r = data;
    ngx_buf_t           *b;
    ngx_chain_t         *cl;

    cl = ngx_chain_get_free_buf(r->pool, &r->upstream->free_bufs);
    if (cl == NULL) {
        return NGX_ERROR;
    }

    b = cl->buf;

    b->pos = b->start;
    b->last = b->pos + bytes;
    b->memory = 1;
    b->flush = 1;
    b->tag = r->upstream->output.tag;
    /* BUG: recycled not set — buffer stays on busy list permanently */

    cl->next = NULL;
    *r->upstream->busy_end = cl;
    r->upstream->busy_end = &cl->next;

    return NGX_OK;
}

Correct (marks buffers as recycled for pool reuse across upstream reads):

static ngx_int_t
ngx_http_myupstream_input_filter(void *data, ssize_t bytes)
{
    ngx_http_request_t  *r = data;
    ngx_buf_t           *b;
    ngx_chain_t         *cl;

    cl = ngx_chain_get_free_buf(r->pool, &r->upstream->free_bufs);
    if (cl == NULL) {
        return NGX_ERROR;
    }

    b = cl->buf;

    b->pos = b->start;
    b->last = b->pos + bytes;
    b->memory = 1;
    b->flush = 1;
    b->tag = r->upstream->output.tag;
    b->recycled = 1; /* enables return to free pool after downstream send */

    cl->next = NULL;
    *r->upstream->busy_end = cl;
    r->upstream->busy_end = &cl->next;

    return NGX_OK;
}

Note: The recycled flag works in conjunction with the buffer's tag field. When nginx scans the busy list after a downstream write completes, buffers with matching tags and recycled = 1 are moved back to the free list. Ensure the tag matches r->upstream->output.tag for proper recycling.

Use Shadow Buffers for Derived Data Instead of Full Copies

When a filter needs to output a sub-range or modified view of an existing buffer, a shadow buffer (b->shadow) points to the same underlying memory while allowing independent pos/last cursors. This avoids duplicating the entire buffer just to change the visible range. For a filter that splits a 64 KB buffer into 4 chunks, shadow buffers save 192 KB of allocation and memcpy per invocation compared to full copies.

Incorrect (copies entire buffer to output a sub-range):

static ngx_int_t
ngx_http_myfilter_split(ngx_http_request_t *r, ngx_buf_t *orig,
    size_t offset, size_t len)
{
    ngx_buf_t    *b;
    ngx_chain_t   out;

    /* SLOW: allocates and copies just to output a slice */
    b = ngx_create_temp_buf(r->pool, len);
    if (b == NULL) {
        return NGX_ERROR;
    }

    ngx_memcpy(b->pos, orig->pos + offset, len);
    b->last = b->pos + len;
    b->memory = 1;

    out.buf = b;
    out.next = NULL;

    return ngx_http_next_body_filter(r, &out);
}

Correct (creates a shadow buffer referencing the original memory range):

static ngx_int_t
ngx_http_myfilter_split(ngx_http_request_t *r, ngx_buf_t *orig,
    size_t offset, size_t len)
{
    ngx_buf_t    *b;
    ngx_chain_t   out;

    b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
    if (b == NULL) {
        return NGX_ERROR;
    }

    /* shadow buffer — points to same memory, different range */
    b->pos = orig->pos + offset;
    b->last = b->pos + len;
    b->memory = 1;
    b->shadow = orig;

    /* inherit file-backed properties if applicable */
    if (orig->in_file) {
        b->in_file = 1;
        b->file = orig->file;
        b->file_pos = orig->file_pos + offset;
        b->file_last = b->file_pos + len;
    }

    /* mark consumption progress on the original buffer */
    if (b->last == orig->last) {
        b->last_shadow = 1;
    }

    out.buf = b;
    out.next = NULL;

    return ngx_http_next_body_filter(r, &out);
}

Note: Set last_shadow = 1 on the final shadow buffer derived from the original. This tells downstream consumers that the original buffer's memory can be reclaimed once this shadow is consumed.

Cache Only Successful Responses to Avoid Negative Cache Pollution

Caching every upstream response regardless of HTTP status pollutes the cache with 5xx errors, 404s, and other transient failures. Once a 502 or 503 is stored, every subsequent request for that key receives the error response for the full TTL duration — turning a momentary upstream blip into sustained user-facing failures. Always check the upstream status code before committing a response to the cache, and only store responses that represent valid, serveable content.

Incorrect (caches all upstream responses including errors):

static ngx_int_t
my_cache_store_response(ngx_http_request_t *r, ngx_shm_zone_t *zone,
    uint32_t hash)
{
    ngx_slab_pool_t    *shpool;
    my_cache_entry_t   *entry;
    ngx_str_t           body;

    /* BAD: stores response without checking status — a 502 from upstream
     * gets cached and served to all users for the entire TTL */
    body.data = r->upstream->buffer.pos;
    body.len = r->upstream->buffer.last - r->upstream->buffer.pos;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    entry = my_cache_find_or_create_locked(shpool, zone, hash);
    if (entry == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    entry->value.data = ngx_slab_alloc_locked(shpool, body.len);
    if (entry->value.data == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    ngx_memcpy(entry->value.data, body.data, body.len);
    entry->value.len = body.len;
    entry->expires = ngx_time() + 60;
    entry->status = r->upstream->headers_in.status_n;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Correct (checks status code before caching, rejects error responses):

static ngx_int_t
my_cache_store_response(ngx_http_request_t *r, ngx_shm_zone_t *zone,
    uint32_t hash)
{
    ngx_slab_pool_t    *shpool;
    my_cache_entry_t   *entry;
    ngx_str_t           body;
    ngx_uint_t          status;

    status = r->upstream->headers_in.status_n;

    /* only cache 2xx responses — reject errors, redirects, and auth challenges */
    if (status < 200 || status >= 300) {
        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "cache skip: upstream returned status %ui", status);
        return NGX_DECLINED;
    }

    body.data = r->upstream->buffer.pos;
    body.len = r->upstream->buffer.last - r->upstream->buffer.pos;

    /* reject empty bodies even on 200 — likely an upstream misconfiguration */
    if (body.len == 0) {
        return NGX_DECLINED;
    }

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    entry = my_cache_find_or_create_locked(shpool, zone, hash);
    if (entry == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    /* free previous value if overwriting a stale entry */
    if (entry->value.data) {
        ngx_slab_free_locked(shpool, entry->value.data);
    }

    entry->value.data = ngx_slab_alloc_locked(shpool, body.len);
    if (entry->value.data == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    ngx_memcpy(entry->value.data, body.data, body.len);
    entry->value.len = body.len;
    entry->expires = ngx_time() + 60;
    entry->status = status;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Use ngx_hash for Fixed Cache Key Lookups

When the set of cache keys is known at configuration time (e.g., a whitelist of cacheable URIs or upstream host names), a pre-built ngx_hash_t provides O(1) lookups. Iterating through an array or linked list to match keys at request time costs O(n) per request, which becomes a measurable bottleneck as the key set grows. Since ngx_hash_t is immutable after ngx_hash_init, build it during configuration and use ngx_hash_find for lock-free, read-only lookups at runtime.

Incorrect (linear scan through array for cache key matching):

typedef struct {
    ngx_str_t    key;
    ngx_str_t    cached_value;
    time_t       expires;
} my_cache_kv_t;

typedef struct {
    my_cache_kv_t   *entries;
    ngx_uint_t       nentries;
} my_cache_loc_conf_t;

static ngx_str_t *
my_cache_find(ngx_http_request_t *r)
{
    my_cache_loc_conf_t  *lcf;
    ngx_uint_t            i;

    lcf = ngx_http_get_module_loc_conf(r, ngx_http_mycache_module);

    /* BAD: O(n) linear scan on every request — 500 entries = 500 comparisons */
    for (i = 0; i < lcf->nentries; i++) {
        if (lcf->entries[i].key.len == r->uri.len
            && ngx_strncmp(lcf->entries[i].key.data,
                           r->uri.data, r->uri.len) == 0)
        {
            if (lcf->entries[i].expires >= ngx_time()) {
                return &lcf->entries[i].cached_value;
            }
        }
    }

    return NULL;
}

Correct (pre-built ngx_hash_t for O(1) lookup at request time):

typedef struct {
    ngx_str_t    cached_value;
    time_t       expires;
} my_cache_val_t;

typedef struct {
    ngx_hash_t   cache_hash;     /* built once during config merge */
} my_cache_loc_conf_t;

/* build hash table during configuration — called once */
static char *
ngx_http_mycache_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
    my_cache_loc_conf_t  *conf = child;
    ngx_hash_init_t       hinit;
    ngx_array_t          *keys_array;

    /* keys_array populated by directive parsing (omitted for brevity) */
    keys_array = my_collect_cache_keys(cf, conf);
    if (keys_array == NULL) {
        return NGX_CONF_ERROR;
    }

    hinit.hash = &conf->cache_hash;
    hinit.key = ngx_hash_key_lc;
    hinit.max_size = 1024;
    hinit.bucket_size = ngx_align(64, ngx_cacheline_size);
    hinit.name = "mycache_keys";
    hinit.pool = cf->pool;
    hinit.temp_pool = cf->temp_pool;

    if (ngx_hash_init(&hinit,
                       (ngx_hash_key_t *) keys_array->elts,
                       keys_array->nelts) != NGX_OK)
    {
        return NGX_CONF_ERROR;
    }

    return NGX_CONF_OK;
}

/* O(1) lookup at request time — no locks needed, hash is immutable */
static ngx_str_t *
my_cache_find(ngx_http_request_t *r)
{
    my_cache_loc_conf_t  *lcf;
    my_cache_val_t       *val;
    ngx_uint_t            hash;

    lcf = ngx_http_get_module_loc_conf(r, ngx_http_mycache_module);

    hash = ngx_hash_key_lc(r->uri.data, r->uri.len);

    val = ngx_hash_find(&lcf->cache_hash, hash, r->uri.data, r->uri.len);

    if (val != NULL && val->expires >= ngx_time()) {
        return &val->cached_value;
    }

    return NULL;
}

Implement LRU Eviction in Shared Memory Cache Zones

Shared memory zones have fixed capacity. Without an eviction strategy, the slab allocator returns NULL once the zone is full, and every subsequent cache-miss request falls through to the upstream — effectively disabling caching for the rest of the process lifetime. Maintaining an LRU queue with ngx_queue_t lets you evict the oldest entry on allocation failure, keeping the cache effective under diverse key sets without requiring a restart or zone resize.

Incorrect (allocates without eviction — cache becomes permanently full):

typedef struct {
    ngx_rbtree_t         rbtree;
    ngx_rbtree_node_t    sentinel;
} my_cache_shm_t;

typedef struct {
    ngx_rbtree_node_t    node;
    ngx_str_t            value;
    uint32_t             key_hash;
} my_cache_entry_t;

static my_cache_entry_t *
my_cache_insert(ngx_shm_zone_t *zone, uint32_t hash, ngx_str_t *val,
    ngx_log_t *log)
{
    ngx_slab_pool_t    *shpool;
    my_cache_shm_t     *cache;
    my_cache_entry_t   *entry;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;
    cache = (my_cache_shm_t *) shpool->data;

    ngx_shmtx_lock(&shpool->mutex);

    /* BAD: when zone is full, alloc returns NULL and every miss hits upstream */
    entry = ngx_slab_alloc_locked(shpool, sizeof(my_cache_entry_t));
    if (entry == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        ngx_log_error(NGX_LOG_ERR, log, 0, "cache zone full, no eviction");
        return NULL;
    }

    entry->node.key = hash;
    entry->key_hash = hash;
    entry->value = *val;

    ngx_rbtree_insert(&cache->rbtree, &entry->node);

    ngx_shmtx_unlock(&shpool->mutex);

    return entry;
}

Correct (maintains LRU queue and evicts oldest entry on allocation failure):

typedef struct {
    ngx_rbtree_t         rbtree;
    ngx_rbtree_node_t    sentinel;
    ngx_queue_t          lru_queue;   /* LRU: head = newest, tail = oldest */
} my_cache_shm_t;

typedef struct {
    ngx_rbtree_node_t    node;
    ngx_queue_t          queue;       /* link into lru_queue */
    ngx_str_t            value;
    uint32_t             key_hash;
} my_cache_entry_t;

/* evicts LRU tail: ngx_queue_last → rbtree_delete → slab_free_locked */
static void my_cache_evict_oldest(ngx_slab_pool_t *, my_cache_shm_t *);

static my_cache_entry_t *
my_cache_insert(ngx_shm_zone_t *zone, uint32_t hash, ngx_str_t *val,
    ngx_log_t *log)
{
    ngx_slab_pool_t    *shpool;
    my_cache_shm_t     *cache;
    my_cache_entry_t   *entry;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;
    cache = (my_cache_shm_t *) shpool->data;

    ngx_shmtx_lock(&shpool->mutex);

    entry = ngx_slab_alloc_locked(shpool, sizeof(my_cache_entry_t));
    if (entry == NULL) {
        /* evict LRU entry and retry allocation */
        my_cache_evict_oldest(shpool, cache);
        entry = ngx_slab_alloc_locked(shpool, sizeof(my_cache_entry_t));
    }

    if (entry == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        ngx_log_error(NGX_LOG_ALERT, log, 0,
                      "cache zone full even after LRU eviction");
        return NULL;
    }

    entry->node.key = hash;
    entry->key_hash = hash;
    entry->value = *val;

    ngx_rbtree_insert(&cache->rbtree, &entry->node);
    ngx_queue_insert_head(&cache->lru_queue, &entry->queue);

    ngx_shmtx_unlock(&shpool->mutex);

    return entry;
}

Prevent Cache Stampede with Single-Flight Pattern

When a popular cache key expires, every concurrent request that encounters the miss will independently fetch from the upstream, multiplying backend load by the concurrency factor. Under high traffic this thundering-herd effect can overwhelm the upstream or saturate worker connections. A single-flight pattern uses a lock flag in the shared cache entry so that only the first request fetches while subsequent requests wait or return stale data until the entry is repopulated.

Incorrect (all concurrent requests independently fetch on cache miss):

typedef struct {
    ngx_rbtree_node_t    node;
    ngx_queue_t          queue;
    ngx_str_t            value;
    time_t               expires;
    uint32_t             key_hash;
} my_cache_entry_t;

static ngx_int_t
my_cache_lookup(ngx_http_request_t *r, ngx_shm_zone_t *zone,
    uint32_t hash, ngx_str_t *result)
{
    ngx_slab_pool_t    *shpool;
    my_cache_entry_t   *entry;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    entry = my_cache_find_locked(zone, hash);

    if (entry == NULL || entry->expires < ngx_time()) {
        ngx_shmtx_unlock(&shpool->mutex);
        /* BAD: every concurrent miss triggers a separate upstream fetch —
         * 100 concurrent requests = 100 upstream calls for the same key */
        return my_fetch_from_upstream(r, zone, hash, result);
    }

    *result = entry->value;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Correct (uses lock flag so only the first miss fetches, others wait or get stale):

typedef struct {
    ngx_rbtree_node_t    node;
    ngx_queue_t          queue;
    ngx_str_t            value;
    time_t               expires;
    uint32_t             key_hash;
    unsigned             fetching:1;    /* single-flight lock flag */
} my_cache_entry_t;

static ngx_int_t
my_cache_lookup(ngx_http_request_t *r, ngx_shm_zone_t *zone,
    uint32_t hash, ngx_str_t *result)
{
    ngx_slab_pool_t    *shpool;
    my_cache_entry_t   *entry;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    entry = my_cache_find_locked(zone, hash);

    if (entry != NULL && entry->expires >= ngx_time()) {
        /* cache hit — promote in LRU and return */
        *result = entry->value;
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_OK;
    }

    if (entry != NULL && entry->fetching) {
        /* another worker is already fetching — serve stale value */
        *result = entry->value;
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_OK;
    }

    /* first miss: claim the fetch lock */
    if (entry == NULL) {
        entry = my_cache_create_locked(shpool, zone, hash);
        if (entry == NULL) {
            ngx_shmtx_unlock(&shpool->mutex);
            return NGX_ERROR;
        }
    }

    entry->fetching = 1;

    ngx_shmtx_unlock(&shpool->mutex);

    /* only this worker fetches from upstream */
    ngx_int_t rc = my_fetch_from_upstream(r, zone, hash, result);

    ngx_shmtx_lock(&shpool->mutex);

    if (rc == NGX_OK) {
        entry->value = *result;
        entry->expires = ngx_time() + 60;
    }

    entry->fetching = 0;   /* release single-flight lock */

    ngx_shmtx_unlock(&shpool->mutex);

    return rc;
}

Use Atomic Timestamp Comparison for TTL Expiry Checks

Checking whether a cache entry has expired is a read-only operation — it only compares the entry's stored timestamp against the current time. Acquiring the shared memory mutex for this check serializes all workers on every cache probe, even when the entry is still valid and no mutation is needed. By storing the expiry as an ngx_atomic_t and reading it with ngx_atomic_fetch_add(&expires, 0) (a no-op add that returns the current value atomically), you skip the mutex on the expired-entry fast path. Cache hits still require the mutex to safely read the value pointer, but expired entries return immediately without locking.

Incorrect (locks mutex just to check expiry timestamp):

typedef struct {
    ngx_rbtree_node_t    node;
    ngx_queue_t          queue;
    ngx_str_t            value;
    time_t               expires;       /* plain time_t, needs mutex to read safely */
    uint32_t             key_hash;
} my_cache_entry_t;

static ngx_int_t
my_cache_check_ttl(ngx_shm_zone_t *zone, my_cache_entry_t *entry,
    ngx_str_t *result)
{
    ngx_slab_pool_t  *shpool;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    /* BAD: full mutex round-trip just to compare two timestamps */
    ngx_shmtx_lock(&shpool->mutex);

    if (entry->expires < ngx_time()) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_DECLINED;    /* expired */
    }

    *result = entry->value;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Correct (atomic read of expiry timestamp avoids mutex for TTL check):

typedef struct {
    ngx_rbtree_node_t    node;
    ngx_queue_t          queue;
    ngx_str_t            value;
    ngx_atomic_t         expires;       /* atomic — safe to read without mutex */
    uint32_t             key_hash;
} my_cache_entry_t;

static ngx_int_t
my_cache_check_ttl(ngx_shm_zone_t *zone, my_cache_entry_t *entry,
    ngx_str_t *result)
{
    ngx_slab_pool_t  *shpool;
    time_t            exp;

    /* lock-free TTL check: atomic read of expiry, compare with cached time */
    exp = (time_t) ngx_atomic_fetch_add(&entry->expires, 0);

    if (exp < ngx_time()) {
        return NGX_DECLINED;    /* expired — caller can lock and evict */
    }

    /* cache hit — still need mutex to read the value safely
     * (value.data is a pointer that could be freed concurrently) */
    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);
    *result = entry->value;
    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Use Lingering Close for Graceful Connection Shutdown

When a module finishes sending an error response (e.g., 400 or 413) but the client is still transmitting request body data, calling ngx_close_connection() immediately causes the kernel to send a TCP RST. The RST arrives before the client reads the response, so the client's TCP stack discards the buffered response data and reports a "connection reset" error instead of showing the server's error page. Lingering close drains the remaining client data before performing the final close, ensuring the response is fully received.

Incorrect (immediate close sends RST while client is still sending):

static void
ngx_http_mymodule_reject_request(ngx_http_request_t *r)
{
    ngx_connection_t  *c = r->connection;
    ngx_buf_t         *b;
    ngx_chain_t        out;

    /* send 413 response */
    r->headers_out.status = NGX_HTTP_REQUEST_ENTITY_TOO_LARGE;
    r->headers_out.content_length_n = sizeof("Request too large") - 1;
    ngx_http_send_header(r);

    b = ngx_calloc_buf(r->pool);
    b->pos = (u_char *) "Request too large";
    b->last = b->pos + sizeof("Request too large") - 1;
    b->memory = 1;
    b->last_buf = 1;
    out.buf = b;
    out.next = NULL;

    ngx_http_output_filter(r, &out);

    /* BUG: client is still uploading body data — immediate close
     * sends RST, client never sees the 413 response */
    ngx_http_close_connection(c);
}

Correct (lingering close drains client data so response is fully received):

static void
ngx_http_mymodule_reject_request(ngx_http_request_t *r)
{
    ngx_buf_t    *b;
    ngx_chain_t   out;

    /* send 413 response */
    r->headers_out.status = NGX_HTTP_REQUEST_ENTITY_TOO_LARGE;
    r->headers_out.content_length_n = sizeof("Request too large") - 1;
    ngx_http_send_header(r);

    b = ngx_calloc_buf(r->pool);
    b->pos = (u_char *) "Request too large";
    b->last = b->pos + sizeof("Request too large") - 1;
    b->memory = 1;
    b->last_buf = 1;
    out.buf = b;
    out.next = NULL;

    ngx_http_output_filter(r, &out);

    /* use lingering close — nginx reads and discards remaining
     * client data before final close, preventing RST;
     * r->lingering_close triggers the lingering close state machine */
    r->lingering_close = 1;
    ngx_http_finalize_request(r, NGX_HTTP_REQUEST_ENTITY_TOO_LARGE);
}

Note: ngx_http_finalize_request with r->lingering_close = 1 enters nginx's built-in lingering close state machine, which reads and discards incoming data for up to lingering_timeout seconds. For raw connections outside the HTTP layer, use ngx_http_set_lingering_close() or implement a manual drain loop with a timer. The lingering_close and lingering_timeout directives in nginx.conf control the global behavior.

Handle Connection Drain Under Memory Pressure

Each nginx worker pre-allocates a fixed-size connection array sized by worker_connections. When free_connection_n drops near zero, the worker cannot accept new clients -- accept() calls fail silently and requests queue in the kernel backlog until they time out. A module that opens its own connections (upstream health checks, async backends) must monitor the free pool and proactively close its own idle connections before the pool is fully exhausted. Note: ngx_drain_connections() is an internal static function — modules cannot call it directly, but nginx calls it automatically inside ngx_get_connection() to reclaim connections marked reusable.

Incorrect (opens connections without checking free pool capacity):

static ngx_int_t
ngx_http_mymodule_connect_backend(ngx_http_request_t *r)
{
    ngx_peer_connection_t  pc;
    ngx_int_t              rc;

    ngx_memzero(&pc, sizeof(ngx_peer_connection_t));
    pc.sockaddr = r->upstream->peer.sockaddr;
    pc.socklen = r->upstream->peer.socklen;
    pc.name = r->upstream->peer.name;
    pc.get = ngx_event_get_peer;
    pc.log = r->connection->log;
    pc.rcvbuf = -1;

    /* BUG: if free_connection_n is 0, ngx_event_connect_peer returns
     * NGX_ERROR and the request fails — no attempt to free idle
     * connections first */
    rc = ngx_event_connect_peer(&pc);

    if (rc == NGX_ERROR) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "backend connect failed");
        return NGX_ERROR;
    }

    return NGX_OK;
}

Correct (closes module's own idle connections when free pool runs low):

static ngx_int_t
ngx_http_mymodule_connect_backend(ngx_http_request_t *r)
{
    ngx_peer_connection_t  pc;
    ngx_int_t              rc;

    /* proactively close module's idle connections when pool is under pressure;
     * threshold of 32 gives headroom for accept() and other modules */
    if (ngx_cycle->free_connection_n < 32) {
        ngx_http_mymodule_close_idle_connections(r->connection->log);

        if (ngx_cycle->free_connection_n == 0) {
            ngx_log_error(NGX_LOG_CRIT, r->connection->log, 0,
                          "no free connections after idle cleanup, "
                          "worker_connections may be too low");
            return NGX_DECLINED;
        }
    }

    ngx_memzero(&pc, sizeof(ngx_peer_connection_t));
    pc.sockaddr = r->upstream->peer.sockaddr;
    pc.socklen = r->upstream->peer.socklen;
    pc.name = r->upstream->peer.name;
    pc.get = ngx_event_get_peer;
    pc.log = r->connection->log;
    pc.rcvbuf = -1;

    rc = ngx_event_connect_peer(&pc);

    if (rc == NGX_ERROR) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "backend connect failed after idle cleanup");
        return NGX_ERROR;
    }

    return NGX_OK;
}

Note: Modules cannot call ngx_drain_connections() directly (it is static to ngx_connection.c). Instead, maintain your own list of idle connections and close them proactively when ngx_cycle->free_connection_n drops below a threshold. Also mark idle connections as reusable with ngx_reusable_connection(c, 1) (see conn-reusable-queue) so nginx's internal drain mechanism can reclaim them during ngx_get_connection().

Size Connection Pool to Avoid Runtime Reallocation

nginx pools grow by allocating new memory blocks when the current block is exhausted. Each growth triggers a malloc() call in the hot path, adding latency and heap fragmentation. The default connection pool size (256 bytes) is tuned for minimal connections but is insufficient for modules that attach context structures, buffers, or parsed headers to the connection pool. Sizing the pool upfront based on expected allocation volume eliminates growth allocations during request processing.

Incorrect (uses default small pool, causing repeated growth allocations):

static void
ngx_http_mymodule_init_connection(ngx_event_t *rev)
{
    ngx_connection_t      *c = rev->data;
    my_conn_ctx_t         *ctx;
    my_parsed_header_t    *headers;

    /* c->pool was created with default 256-byte block size;
     * the following allocations exceed it, forcing 2+ pool
     * growth allocations (malloc) in the request hot path */
    ctx = ngx_pcalloc(c->pool, sizeof(my_conn_ctx_t));       /* ~128 bytes */
    if (ctx == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    ctx->recv_buf = ngx_palloc(c->pool, 4096);               /* forces pool growth */
    if (ctx->recv_buf == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    headers = ngx_pcalloc(c->pool, sizeof(my_parsed_header_t) * 32);  /* forces another growth */
    if (headers == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    ctx->headers = headers;
    c->data = ctx;
}

Correct (sizes connection pool based on expected allocation volume):

static void
ngx_http_mymodule_accept_handler(ngx_event_t *rev)
{
    ngx_connection_t      *c = rev->data;
    ngx_pool_t            *pool;
    my_conn_ctx_t         *ctx;
    my_parsed_header_t    *headers;

    /* pre-calculate total allocation: ctx + recv buffer + headers;
     * create a pool sized to hold everything in a single block */
    pool = ngx_create_pool(
        sizeof(my_conn_ctx_t)                            /* ~128 bytes */
        + 4096                                           /* recv buffer */
        + sizeof(my_parsed_header_t) * 32                /* parsed headers */
        + 256,                                           /* pool overhead + headroom */
        c->log);

    if (pool == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    /* destroy the old undersized pool and replace it */
    ngx_destroy_pool(c->pool);
    c->pool = pool;

    ctx = ngx_pcalloc(c->pool, sizeof(my_conn_ctx_t));
    if (ctx == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    ctx->recv_buf = ngx_palloc(c->pool, 4096);
    if (ctx->recv_buf == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    headers = ngx_pcalloc(c->pool, sizeof(my_parsed_header_t) * 32);
    if (headers == NULL) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    ctx->headers = headers;
    c->data = ctx;
}

Note: The connection_pool_size directive in nginx.conf controls the initial pool size for accepted client connections. For modules that accept their own connections or create upstream connections with ngx_event_connect_peer, you must size the pool yourself. A good heuristic: sum the sizes of all structures you will allocate on the pool, add 20% overhead for pool metadata, and round up to the next power of two.

Mark Idle Connections as Reusable for Pool Recovery

When a custom module holds long-lived connections (e.g., persistent backend channels or health-check sockets), those connections count against the worker's worker_connections limit. Unless the module marks idle connections as reusable via ngx_reusable_connection(), nginx cannot reclaim them when new connections arrive -- the worker hits worker_connections and starts rejecting clients with no way to free capacity.

Incorrect (idle connections are never marked reusable, permanently consuming slots):

static void
ngx_http_mymodule_idle_handler(ngx_event_t *rev)
{
    ngx_connection_t  *c = rev->data;
    my_conn_ctx_t     *ctx = c->data;

    if (rev->timedout) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    /* connection sits idle waiting for next request to reuse it,
     * but it is NOT marked reusable — nginx cannot reclaim it
     * when worker_connections limit is approached */
    ctx->state = MY_CONN_IDLE;

    ngx_add_timer(rev, ctx->idle_timeout);

    if (ngx_handle_read_event(rev, 0) != NGX_OK) {
        ngx_http_mymodule_close_connection(c);
    }
}

Correct (marking idle connections reusable allows nginx to reclaim them under pressure):

static void
ngx_http_mymodule_idle_handler(ngx_event_t *rev)
{
    ngx_connection_t  *c = rev->data;
    my_conn_ctx_t     *ctx = c->data;

    if (rev->timedout) {
        ngx_reusable_connection(c, 0);
        ngx_http_mymodule_close_connection(c);
        return;
    }

    /* mark connection as reusable — nginx can close it to free
     * a slot when worker_connections limit is under pressure */
    ctx->state = MY_CONN_IDLE;
    ngx_reusable_connection(c, 1);

    c->read->handler = ngx_http_mymodule_reuse_handler;
    ngx_add_timer(rev, ctx->idle_timeout);

    if (ngx_handle_read_event(rev, 0) != NGX_OK) {
        ngx_reusable_connection(c, 0);
        ngx_http_mymodule_close_connection(c);
    }
}

static void
ngx_http_mymodule_reuse_handler(ngx_event_t *rev)
{
    ngx_connection_t  *c = rev->data;

    /* remove from reusable queue before active use */
    ngx_reusable_connection(c, 0);

    if (rev->timedout) {
        ngx_http_mymodule_close_connection(c);
        return;
    }

    /* connection has incoming data — reactivate for processing */
    ngx_http_mymodule_process(c);
}

Note: Always call ngx_reusable_connection(c, 0) before using the connection for active I/O or before closing it. The reusable queue is a doubly-linked list on ngx_cycle->reusable_connections_queue -- leaving a freed connection in the queue corrupts the list.

Enable SSL Session Caching in Upstream Connections

A full TLS handshake to an upstream server costs 1-2 additional RTTs on top of the TCP handshake. When a module establishes its own TLS connections to backends (bypassing the standard upstream module), each new connection performs a full handshake unless the module stores and restores SSL sessions. Session resumption (via session IDs or session tickets) reduces the handshake to a single RTT by reusing previously negotiated cryptographic parameters.

Incorrect (no session caching -- full TLS handshake on every connection):

static ngx_int_t
ngx_http_mymodule_ssl_connect(ngx_connection_t *c, my_upstream_conf_t *ucf)
{
    ngx_int_t  rc;

    if (ngx_ssl_create_connection(ucf->ssl, c, NGX_SSL_BUFFER) != NGX_OK) {
        return NGX_ERROR;
    }

    /* no session set — every connection performs a full TLS handshake
     * (ClientHello -> ServerHello -> Certificate -> KeyExchange -> Finished)
     * adding 1-2 RTTs of latency per upstream connection */

    rc = ngx_ssl_handshake(c);

    if (rc == NGX_AGAIN) {
        c->ssl->handler = ngx_http_mymodule_ssl_handshake_done;
        return NGX_AGAIN;
    }

    return rc;
}

static void
ngx_http_mymodule_ssl_handshake_done(ngx_connection_t *c)
{
    /* handshake complete — but session is not saved,
     * so next connection to same peer repeats full handshake */
    ngx_http_mymodule_send_request(c);
}

Correct (stores and restores SSL sessions for upstream session resumption):

static ngx_int_t
ngx_http_mymodule_ssl_connect(ngx_connection_t *c, my_upstream_peer_t *peer)
{
    ngx_int_t      rc;
    ngx_ssl_session_t  *sess;

    if (ngx_ssl_create_connection(peer->ucf->ssl, c,
                                  NGX_SSL_BUFFER) != NGX_OK)
    {
        return NGX_ERROR;
    }

    /* restore cached session — enables abbreviated handshake
     * (ClientHello with session ID -> ServerHello -> Finished)
     * saving 1-2 RTTs on resumed connections */
    sess = peer->ssl_session;
    if (sess != NULL) {
        if (ngx_ssl_set_session(c, sess) != NGX_OK) {
            ngx_log_error(NGX_LOG_WARN, c->log, 0,
                          "failed to set SSL session for %V",
                          peer->name);
            /* proceed without session — will do full handshake */
        }
    }

    rc = ngx_ssl_handshake(c);

    if (rc == NGX_AGAIN) {
        c->ssl->handler = ngx_http_mymodule_ssl_handshake_done;
        return NGX_AGAIN;
    }

    if (rc == NGX_OK) {
        ngx_http_mymodule_save_ssl_session(c, peer);
    }

    return rc;
}

/* handshake callback saves session then sends request */
static void
ngx_http_mymodule_ssl_handshake_done(ngx_connection_t *c)
{
    my_upstream_peer_t  *peer = c->data;
    /* save session: ngx_ssl_get_session → swap peer->ssl_session → free old */
    ngx_http_mymodule_save_ssl_session(c, peer);
    ngx_http_mymodule_send_request(c);
}

Note: ngx_ssl_get_session() returns a reference-counted SSL_SESSION object. Store one session per upstream peer address, not per connection. When a peer is removed or the module shuts down, free the cached session with ngx_ssl_free_session() to avoid memory leaks. For the built-in proxy module, the proxy_ssl_session_reuse directive handles this automatically -- this pattern is only needed for modules managing their own TLS connections.

Control TCP_NODELAY for Latency-Sensitive Responses

Nagle's algorithm batches small TCP writes into a single segment, adding up to 40ms of delay before sending data that does not fill an MSS-sized packet. For modules that generate small interactive responses (JSON API replies, WebSocket frames, server-sent events), failing to set TCP_NODELAY on the connection means every sub-MSS write stalls for the Nagle timer. nginx's core sets TCP_NODELAY on client connections only after tcp_nodelay on is configured, but modules managing their own connections must set it explicitly.

Incorrect (small writes suffer Nagle delay because TCP_NODELAY is never set):

static void
ngx_http_mymodule_send_event(ngx_connection_t *c, ngx_buf_t *event_buf)
{
    ngx_chain_t  out;

    /* SSE event is typically 50-200 bytes — well under MSS;
     * without TCP_NODELAY, Nagle holds it for ~40ms waiting
     * for more data to coalesce */
    out.buf = event_buf;
    out.next = NULL;

    event_buf->last_buf = 0;
    event_buf->flush = 1;

    c->send_chain(c, &out, 0);
}

Correct (sets TCP_NODELAY after connection establishment for immediate delivery):

static ngx_int_t
ngx_http_mymodule_init_connection(ngx_connection_t *c)
{
    int  tcp_nodelay = 1;

    /* disable Nagle for this connection — small writes
     * (SSE frames, JSON responses) are sent immediately */
    if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY,
                   &tcp_nodelay, sizeof(int)) == -1)
    {
        ngx_log_error(NGX_LOG_ALERT, c->log, ngx_socket_errno,
                      "setsockopt(TCP_NODELAY) failed");
        return NGX_ERROR;
    }

    c->tcp_nodelay = NGX_TCP_NODELAY_SET;

    return NGX_OK;
}

static void
ngx_http_mymodule_send_event(ngx_connection_t *c, ngx_buf_t *event_buf)
{
    ngx_chain_t  out;

    /* TCP_NODELAY is set — this 50-200 byte SSE event
     * is pushed to the wire immediately */
    out.buf = event_buf;
    out.next = NULL;

    event_buf->last_buf = 0;
    event_buf->flush = 1;

    c->send_chain(c, &out, 0);
}

Note: Set c->tcp_nodelay = NGX_TCP_NODELAY_SET after the setsockopt call so nginx's core does not attempt to set it again. For bulk transfer connections (file downloads, large proxied responses), leaving Nagle enabled is correct -- it reduces small-packet overhead and improves throughput. Only disable it for latency-sensitive interactive traffic.

Use Blocked Counter to Prevent Premature Request Destruction

When a handler initiates an asynchronous operation — AIO file read, cache lock wait, or thread task — the event loop may call ngx_http_finalize_request before the callback fires. If r->blocked is zero, nginx will free the request and its pool, and the callback will operate on freed memory. Incrementing r->blocked before dispatching the async operation tells nginx the request has outstanding work; decrementing it in the callback and re-running finalization lets nginx destroy the request only when all operations have completed.

Incorrect (dispatches AIO without incrementing r->blocked, risks use-after-free):

static ngx_int_t
ngx_http_mymodule_read_async(ngx_http_request_t *r, ngx_file_t *file)
{
    ngx_buf_t  *b;

    b = ngx_create_temp_buf(r->pool, 4096);
    if (b == NULL) {
        return NGX_ERROR;
    }

    /* BUG: if client disconnects during AIO, nginx finalizes the request
     * and frees r->pool — callback will dereference freed memory */
    file->aio = r->connection->aio;

    return ngx_file_aio_read(file, b->pos, 4096, 0, r->pool);
}

static void
ngx_http_mymodule_aio_handler(ngx_event_t *ev)
{
    ngx_http_request_t  *r = ev->data;

    /* r may already be freed here — use-after-free */
    ngx_http_finalize_request(r, ngx_http_mymodule_send_response(r));
}

Correct (increments r->blocked before AIO, decrements in callback before finalizing):

static ngx_int_t
ngx_http_mymodule_read_async(ngx_http_request_t *r, ngx_file_t *file)
{
    ngx_buf_t  *b;

    b = ngx_create_temp_buf(r->pool, 4096);
    if (b == NULL) {
        return NGX_ERROR;
    }

    file->aio = r->connection->aio;

    /* prevent nginx from destroying the request while AIO is in-flight */
    r->blocked++;
    r->aio = 1;

    return ngx_file_aio_read(file, b->pos, 4096, 0, r->pool);
}

static void
ngx_http_mymodule_aio_handler(ngx_event_t *ev)
{
    ngx_http_request_t  *r = ev->data;

    /* AIO complete — unblock the request so finalization can proceed */
    r->blocked--;
    r->aio = 0;

    if (r->blocked == 0) {
        /* safe to finalize now — no other async operations pending */
        ngx_http_finalize_request(r, ngx_http_mymodule_send_response(r));
    }
}

See also: The companion correctness skill's req-count-reference covers r->main->count++ for general async operations (subrequests, timers). Use r->blocked specifically for AIO and thread pool tasks — it prevents ngx_http_finalize_request from destroying the request while the kernel-level async operation is in-flight.

Cache ngx_errno Immediately to Prevent Overwrite

The global ngx_errno (mapped to errno on POSIX) is overwritten by any subsequent syscall or library call, including ngx_log_error itself which may invoke write(). If you reference ngx_errno after calling any function, you log whatever error that function produced — not the error you intended to diagnose. Capture it into a local ngx_err_t variable on the very first line after the failing call.

Incorrect (reads ngx_errno after intervening calls have overwritten it):

static ngx_int_t
ngx_http_mymodule_open_cache(ngx_http_request_t *r, ngx_str_t *path)
{
    ngx_fd_t  fd;

    fd = ngx_open_file(path->data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        /* BUG: ngx_close_file or internal log write overwrites ngx_errno */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, ngx_errno,
                      "failed to open cache file \"%V\"", path);
        return NGX_ERROR;
    }

    /* ... */
    ngx_close_file(fd);
    return NGX_OK;
}

Correct (saves ngx_errno into a local variable before any other call):

static ngx_int_t
ngx_http_mymodule_open_cache(ngx_http_request_t *r, ngx_str_t *path)
{
    ngx_fd_t   fd;
    ngx_err_t  err;

    fd = ngx_open_file(path->data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    if (fd == NGX_INVALID_FILE) {
        /* capture errno immediately — before any call can overwrite it */
        err = ngx_errno;

        ngx_log_error(NGX_LOG_ERR, r->connection->log, err,
                      "failed to open cache file \"%V\"", path);
        return NGX_ERROR;
    }

    /* ... */
    ngx_close_file(fd);
    return NGX_OK;
}

Note: This is primarily a diagnostic accuracy pattern, but it directly impacts reliability: misdiagnosed errors lead to wrong remediation, extending outage duration. Under incident conditions, accurate errno reporting is the difference between a 5-minute fix and a multi-hour investigation.

Check Connection Error Flag Before I/O Operations

Once a connection has experienced an error — client disconnect, reset, or write timeout — nginx sets c->error = 1 on the connection. Attempting further send() or recv() on an errored connection will always fail, wasting a syscall round-trip and generating redundant log entries for every subsequent operation. Checking c->error early and returning immediately avoids pointless kernel transitions and keeps error logs clean for the failures that actually matter.

Incorrect (attempts write without checking c->error, wastes syscalls on dead connections):

static ngx_int_t
ngx_http_mymodule_send_chunk(ngx_http_request_t *r, ngx_chain_t *out)
{
    ngx_connection_t  *c;
    ngx_chain_t       *cl;

    c = r->connection;

    /* BUG: if client already disconnected, every send() returns EPIPE
     * and floods the error log with identical "broken pipe" messages */
    cl = c->send_chain(c, out, 0);

    if (cl == NGX_CHAIN_ERROR) {
        ngx_log_error(NGX_LOG_ERR, c->log, 0,
                      "send_chain failed for \"%V\"", &r->uri);
        return NGX_ERROR;
    }

    return NGX_OK;
}

Correct (returns early when connection is already in error state):

static ngx_int_t
ngx_http_mymodule_send_chunk(ngx_http_request_t *r, ngx_chain_t *out)
{
    ngx_connection_t  *c;
    ngx_chain_t       *cl;

    c = r->connection;

    /* skip I/O entirely if the connection is already dead */
    if (c->error) {
        return NGX_ERROR;
    }

    cl = c->send_chain(c, out, 0);

    if (cl == NGX_CHAIN_ERROR) {
        c->error = 1;
        ngx_log_error(NGX_LOG_ERR, c->log, 0,
                      "send_chain failed for \"%V\"", &r->uri);
        return NGX_ERROR;
    }

    return NGX_OK;
}

Return Fallback Response When Upstream Fails

When an upstream is unreachable or times out, returning NGX_HTTP_BAD_GATEWAY directly gives the client an empty error page and no useful content. A resilient module should detect the upstream failure and generate a degraded fallback response — a cached snapshot, a static default body, or a minimal JSON payload — so downstream services and end users can continue operating with stale-but-functional data instead of a hard failure.

Incorrect (returns 502 directly, giving the client an empty error page):

static ngx_int_t
ngx_http_mymodule_upstream_finalize(ngx_http_request_t *r,
    ngx_http_upstream_t *u)
{
    if (u->peer.connection == NULL || u->headers_in.status_n == 0) {
        /* BUG: hard 502 — client gets no useful content, cascades to callers */
        return ngx_http_send_special(r, NGX_HTTP_LAST);
    }

    return NGX_OK;
}

Correct (generates a fallback response body when upstream is unavailable):

static ngx_int_t
ngx_http_mymodule_upstream_finalize(ngx_http_request_t *r,
    ngx_http_upstream_t *u)
{
    ngx_buf_t    *b;
    ngx_chain_t   out;

    if (u->peer.connection == NULL || u->headers_in.status_n == 0) {
        /* upstream failed — serve a degraded fallback instead of 502 */
        ngx_log_error(NGX_LOG_WARN, r->connection->log, 0,
                      "upstream unavailable, serving fallback for \"%V\"",
                      &r->uri);

        static u_char fallback_body[] =
            "{\"status\":\"degraded\",\"source\":\"fallback\"}";

        b = ngx_calloc_buf(r->pool);
        if (b == NULL) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }

        b->pos = fallback_body;
        b->last = fallback_body + sizeof(fallback_body) - 1;
        b->memory = 1;
        b->last_buf = (r == r->main) ? 1 : 0;
        b->last_in_chain = 1;

        out.buf = b;
        out.next = NULL;

        r->headers_out.status = NGX_HTTP_OK;
        r->headers_out.content_length_n = b->last - b->pos;

        ngx_str_set(&r->headers_out.content_type, "application/json");
        r->headers_out.content_type_len = r->headers_out.content_type.len;

        ngx_http_send_header(r);

        return ngx_http_output_filter(r, &out);
    }

    return NGX_OK;
}

Limit Repeated Error Logging to Prevent Log Storms

During sustained failure conditions — upstream outage, shared memory exhaustion, disk full — every request triggers the same ngx_log_error call. At high request rates this produces thousands of identical log lines per second, and the synchronous write() to the error log file becomes the dominant bottleneck, stalling event loop processing for healthy requests. Tracking an error count in module context and logging only periodically (e.g., every Nth occurrence or once per interval) preserves diagnostic visibility without letting log I/O cascade into a full worker stall.

Incorrect (logs every occurrence of a repeated error, causing log I/O storm):

static ngx_int_t
ngx_http_mymodule_shm_lookup(ngx_http_request_t *r,
    ngx_http_mymodule_loc_conf_t *lcf)
{
    ngx_http_mymodule_node_t  *node;

    node = ngx_http_mymodule_find_node(lcf->shm_zone, &r->uri);

    if (node == NULL) {
        /* BUG: under sustained cache-miss storm, this fires on every request
         * — at 10k rps that's 10k write() syscalls/sec to error.log */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "shared memory lookup failed for \"%V\"", &r->uri);
        return NGX_HTTP_SERVICE_UNAVAILABLE;
    }

    return NGX_OK;
}

Correct (tracks error count in shared memory and logs periodically):

static ngx_int_t
ngx_http_mymodule_shm_lookup(ngx_http_request_t *r,
    ngx_http_mymodule_loc_conf_t *lcf)
{
    ngx_http_mymodule_node_t  *node;
    ngx_http_mymodule_shm_t   *shm;
    ngx_atomic_int_t           prev;

    node = ngx_http_mymodule_find_node(lcf->shm_zone, &r->uri);

    if (node == NULL) {
        shm = lcf->shm_zone->data;

        /* atomic increment returns the previous value — use it directly */
        prev = ngx_atomic_fetch_add(&shm->lookup_errors, 1);

        /* log only every 1000th occurrence to avoid I/O storm */
        if ((prev + 1) % 1000 == 1) {
            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                          "shared memory lookup failed for \"%V\" "
                          "(%uA total failures)", &r->uri, prev + 1);
        }

        return NGX_HTTP_SERVICE_UNAVAILABLE;
    }

    return NGX_OK;
}

Handle Pool and Slab Allocation Exhaustion Gracefully

Shared memory slab allocations (ngx_slab_alloc_locked, ngx_slab_alloc) return NULL when the zone is full, and pool allocations can fail under extreme memory pressure. Dereferencing the result without a NULL check causes a segfault that kills the entire worker process, taking down every in-flight request on that worker. Always check the return value and degrade to a 503 response, which keeps the worker alive and lets other requests proceed normally.

Incorrect (dereferences slab alloc result without NULL check, crashes on exhaustion):

static ngx_int_t
ngx_http_mymodule_shm_store(ngx_http_request_t *r,
    ngx_http_mymodule_shm_t *shm, ngx_str_t *key, ngx_str_t *value)
{
    ngx_http_mymodule_node_t  *node;

    ngx_shmtx_lock(&shm->shpool->mutex);

    /* BUG: if slab is full this returns NULL — next line segfaults */
    node = ngx_slab_alloc_locked(shm->shpool,
                                 sizeof(ngx_http_mymodule_node_t));

    node->key = *key;
    ngx_memcpy(node->value, value->data, value->len);

    ngx_shmtx_unlock(&shm->shpool->mutex);

    return NGX_OK;
}

Correct (checks for NULL and returns 503 to keep the worker alive):

static ngx_int_t
ngx_http_mymodule_shm_store(ngx_http_request_t *r,
    ngx_http_mymodule_shm_t *shm, ngx_str_t *key, ngx_str_t *value)
{
    ngx_http_mymodule_node_t  *node;

    ngx_shmtx_lock(&shm->shpool->mutex);

    node = ngx_slab_alloc_locked(shm->shpool,
                                 sizeof(ngx_http_mymodule_node_t));
    if (node == NULL) {
        ngx_shmtx_unlock(&shm->shpool->mutex);

        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "shared memory zone \"%V\" exhausted, "
                      "rejecting request with 503", &shm->shm.name);

        /* 503 keeps worker alive — client can retry after zone pressure eases */
        return NGX_HTTP_SERVICE_UNAVAILABLE;
    }

    node->key = *key;
    ngx_memcpy(node->value, value->data, value->len);

    ngx_shmtx_unlock(&shm->shpool->mutex);

    return NGX_OK;
}

Perform Slab Allocation Outside Hot Path

ngx_slab_alloc acquires the shared zone mutex internally and may trigger slab page splitting or garbage collection, holding the lock for an unpredictable duration. Calling it during request processing turns the slab allocator into a serialization point across all workers. Pre-allocate slots during module init or maintain a free-list of reusable entries so the hot path never enters the slab allocator.

Incorrect (allocates from slab on every request):

static ngx_int_t
ngx_http_session_handler(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t   *shpool;
    my_session_t      *sess;
    my_shm_data_t     *data;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    data = (my_shm_data_t *) shpool->data;

    /* BAD: slab alloc on every new session request — under load this
       triggers page splitting while all other workers wait on the mutex */
    sess = ngx_slab_alloc_locked(shpool, sizeof(my_session_t));
    if (sess == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_HTTP_SERVICE_UNAVAILABLE;
    }

    ngx_memzero(sess, sizeof(my_session_t));
    sess->start = ngx_current_msec;

    ngx_queue_insert_head(&data->active_sessions, &sess->queue);

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Correct (pre-allocates a free-list during init, hot path pops from free-list):

static ngx_int_t
ngx_http_session_init_zone(ngx_shm_zone_t *zone, void *data)
{
    ngx_slab_pool_t   *shpool;
    my_shm_data_t     *shdata;
    my_session_t      *sess;
    ngx_uint_t         i;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    shdata = ngx_slab_alloc(shpool, sizeof(my_shm_data_t));
    if (shdata == NULL) {
        return NGX_ERROR;
    }

    ngx_queue_init(&shdata->active_sessions);
    ngx_queue_init(&shdata->free_sessions);

    /* pre-allocate session slots at init — no slab alloc at runtime */
    for (i = 0; i < 4096; i++) {
        sess = ngx_slab_alloc(shpool, sizeof(my_session_t));
        if (sess == NULL) {
            return NGX_ERROR;
        }
        ngx_queue_insert_tail(&shdata->free_sessions, &sess->queue);
    }

    shpool->data = shdata;
    zone->data = shdata;

    return NGX_OK;
}

static ngx_int_t
ngx_http_session_handler(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t   *shpool;
    my_session_t      *sess;
    my_shm_data_t     *data;
    ngx_queue_t       *q;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    data = (my_shm_data_t *) shpool->data;

    if (ngx_queue_empty(&data->free_sessions)) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_HTTP_SERVICE_UNAVAILABLE;
    }

    /* fast path: pop from pre-allocated free-list — no slab allocator */
    q = ngx_queue_last(&data->free_sessions);
    ngx_queue_remove(q);

    sess = ngx_queue_data(q, my_session_t, queue);
    ngx_memzero(sess, sizeof(my_session_t));
    sess->start = ngx_current_msec;

    ngx_queue_insert_head(&data->active_sessions, &sess->queue);

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Use Atomic Operations for Simple Counters Instead of Mutex

Atomic operations on ngx_atomic_t variables use CPU-level compare-and-swap instructions that complete in a single bus cycle without acquiring or releasing a mutex. For simple counters (request counts, byte totals, error tallies), replacing a lock/increment/unlock sequence with ngx_atomic_fetch_add eliminates all mutex overhead and contention entirely. This is the single most impactful optimization for high-frequency shared counters.

Incorrect (acquires mutex to increment a simple counter):

typedef struct {
    ngx_atomic_t    total_requests;
    ngx_atomic_t    total_bytes;
} my_shm_counters_t;

static ngx_int_t
ngx_http_metrics_count(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t     *shpool;
    my_shm_counters_t   *counters;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    /* BAD: full mutex round-trip for a single atomic-safe increment */
    ngx_shmtx_lock(&shpool->mutex);

    counters = (my_shm_counters_t *) shpool->data;
    counters->total_requests++;
    counters->total_bytes += r->headers_out.content_length_n;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Correct (uses ngx_atomic_fetch_add for lock-free counter updates):

typedef struct {
    ngx_atomic_t    total_requests;
    ngx_atomic_t    total_bytes;
} my_shm_counters_t;

static ngx_int_t
ngx_http_metrics_count(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t     *shpool;
    my_shm_counters_t   *counters;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;
    counters = (my_shm_counters_t *) shpool->data;

    /* lock-free: hardware CAS, no mutex contention */
    ngx_atomic_fetch_add(&counters->total_requests, 1);
    ngx_atomic_fetch_add(&counters->total_bytes,
                         r->headers_out.content_length_n);

    return NGX_OK;
}

Minimize Critical Section Duration in Shared Memory

Every nanosecond a worker holds the shared memory mutex is a nanosecond other workers spin or sleep waiting for it. Performing computation, string formatting, or data transformation inside the critical section serializes all workers through a single bottleneck. Prepare all data in worker-local memory first, then enter the critical section only for the final pointer swap or memcpy.

Incorrect (performs computation and formatting while holding the mutex):

static ngx_int_t
ngx_http_stats_update(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t   *shpool;
    my_shm_stats_t    *stats;
    ngx_time_t        *tp;
    u_char             buf[128];
    size_t             len;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    stats = (my_shm_stats_t *) shpool->data;

    /* BAD: time computation inside critical section */
    tp = ngx_timeofday();
    stats->last_access = tp->sec * 1000 + tp->msec;

    /* BAD: string formatting inside critical section */
    len = ngx_snprintf(buf, sizeof(buf), "%V:%ui",
                       &r->uri, r->headers_out.status)
          - buf;

    stats->last_uri.data = ngx_slab_alloc_locked(shpool, len);
    if (stats->last_uri.data == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    ngx_memcpy(stats->last_uri.data, buf, len);
    stats->last_uri.len = len;

    /* BAD: latency histogram computation under lock */
    stats->bucket[ngx_http_stats_latency_bucket(r)]++;
    stats->total_requests++;
    stats->total_latency += ngx_http_stats_request_ms(r);

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Correct (prepares all data outside lock, enters critical section only for memcpy and counter updates):

static ngx_int_t
ngx_http_stats_update(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t   *shpool;
    my_shm_stats_t    *stats;
    ngx_msec_t         now;
    ngx_uint_t         bucket;
    ngx_msec_int_t     latency;
    u_char             buf[128];
    size_t             len;

    /* prepare everything in worker-local stack */
    now = ngx_current_msec;
    bucket = ngx_http_stats_latency_bucket(r);
    latency = ngx_http_stats_request_ms(r);

    len = ngx_snprintf(buf, sizeof(buf), "%V:%ui",
                       &r->uri, r->headers_out.status)
          - buf;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    /* critical section: only pointer writes and increments */
    stats = (my_shm_stats_t *) shpool->data;
    stats->last_access = now;

    stats->last_uri.data = ngx_slab_alloc_locked(shpool, len);
    if (stats->last_uri.data == NULL) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_ERROR;
    }

    ngx_memcpy(stats->last_uri.data, buf, len);
    stats->last_uri.len = len;

    stats->bucket[bucket]++;
    stats->total_requests++;
    stats->total_latency += latency;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_OK;
}

Aggregate Per-Worker Counters to Reduce Shared Memory Access

When every request touches a shared memory counter, workers constantly compete for the same cache line and mutex even when using atomics. Instead, accumulate counts in a per-worker local variable and flush to shared memory periodically (every N requests or on a timer). This converts N atomic operations per second into one, eliminating cross-core cache invalidation traffic and shared memory bus contention entirely on the hot path.

Incorrect (increments shared atomic counter on every single request):

static ngx_int_t
ngx_http_metrics_handler(ngx_http_request_t *r)
{
    ngx_slab_pool_t     *shpool;
    my_shm_counters_t   *counters;

    shpool = (ngx_slab_pool_t *) my_shm_zone->shm.addr;
    counters = (my_shm_counters_t *) shpool->data;

    /* BAD: atomic on every request — at 100k rps across 8 workers,
       this causes ~800k cross-core cache invalidations per second */
    ngx_atomic_fetch_add(&counters->total_requests, 1);
    ngx_atomic_fetch_add(&counters->total_bytes,
                         r->headers_out.content_length_n);

    return NGX_DECLINED;
}

Correct (accumulates in per-worker local storage, flushes periodically):

#define MY_FLUSH_INTERVAL  128  /* flush every 128 requests */

typedef struct {
    ngx_uint_t    local_requests;
    ngx_uint_t    local_bytes;
} my_worker_ctx_t;

/* allocated per worker in init_process */
static my_worker_ctx_t  *my_worker_ctx;

static ngx_int_t
ngx_http_metrics_handler(ngx_http_request_t *r)
{
    ngx_slab_pool_t     *shpool;
    my_shm_counters_t   *counters;

    /* fast path: worker-local increment, no shared memory access */
    my_worker_ctx->local_requests++;
    my_worker_ctx->local_bytes += r->headers_out.content_length_n;

    if (my_worker_ctx->local_requests < MY_FLUSH_INTERVAL) {
        return NGX_DECLINED;
    }

    /* slow path: flush accumulated counters to shared memory */
    shpool = (ngx_slab_pool_t *) my_shm_zone->shm.addr;
    counters = (my_shm_counters_t *) shpool->data;

    ngx_atomic_fetch_add(&counters->total_requests,
                         my_worker_ctx->local_requests);
    ngx_atomic_fetch_add(&counters->total_bytes,
                         my_worker_ctx->local_bytes);

    my_worker_ctx->local_requests = 0;
    my_worker_ctx->local_bytes = 0;

    return NGX_DECLINED;
}

Use Read-Copy-Update Pattern for Read-Heavy Shared Data

When shared data is read on every request but updated rarely (configuration reloads, periodic aggregation), locking the mutex for every read serializes all workers through the same bottleneck. Instead, maintain two copies of the data and an atomic version pointer. Readers load the current pointer atomically with zero locking. Writers prepare the new version in the inactive copy, then atomically swap the pointer. This gives readers wait-free access at the cost of double memory for the shared structure.

Incorrect (locks mutex on every read of infrequently-updated shared config):

static ngx_int_t
ngx_http_geoblock_handler(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t    *shpool;
    my_geoblock_t      *geo;
    ngx_uint_t          blocked;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    /* BAD: mutex on every request for data that changes once per hour;
       at 50k rps across 8 workers this is ~400k lock acquisitions/sec */
    ngx_shmtx_lock(&shpool->mutex);

    geo = (my_geoblock_t *) shpool->data;
    blocked = ngx_http_geoblock_lookup(geo->rules, geo->nrules,
                                       r->connection->sockaddr);

    ngx_shmtx_unlock(&shpool->mutex);

    if (blocked) {
        return NGX_HTTP_FORBIDDEN;
    }

    return NGX_DECLINED;
}

Correct (atomic pointer read for lock-free hot path, mutex only on update):

typedef struct {
    my_geoblock_rules_t    versions[2];    /* double-buffered rule sets */
    ngx_atomic_t           active;         /* index of current version: 0 or 1 */
} my_geoblock_t;

static ngx_int_t
ngx_http_geoblock_handler(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t        *shpool;
    my_geoblock_t          *geo;
    my_geoblock_rules_t    *current;
    ngx_uint_t              idx, blocked;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;
    geo = (my_geoblock_t *) shpool->data;

    /* lock-free read: atomic load of active index */
    idx = (ngx_uint_t) *((volatile ngx_atomic_t *) &geo->active);
    current = &geo->versions[idx];

    blocked = ngx_http_geoblock_lookup(current->rules, current->nrules,
                                       r->connection->sockaddr);

    if (blocked) {
        return NGX_HTTP_FORBIDDEN;
    }

    return NGX_DECLINED;
}

static void
ngx_http_geoblock_update(ngx_shm_zone_t *zone, my_geoblock_rules_t *new_rules)
{
    ngx_slab_pool_t        *shpool;
    my_geoblock_t          *geo;
    ngx_uint_t              active, inactive;
    my_geoblock_rules_t    *target;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    ngx_shmtx_lock(&shpool->mutex);

    geo = (my_geoblock_t *) shpool->data;
    active = (ngx_uint_t) geo->active;
    inactive = 1 - active;

    /* write new data into the inactive version */
    target = &geo->versions[inactive];
    ngx_memcpy(target->rules, new_rules->rules,
               new_rules->nrules * sizeof(my_rule_t));
    target->nrules = new_rules->nrules;

    /* atomic pointer swap — readers immediately see new version */
    geo->active = (ngx_atomic_t) inactive;

    ngx_shmtx_unlock(&shpool->mutex);
}

Use ngx_shmtx_trylock with Fallback to Avoid Worker Stalls

ngx_shmtx_lock blocks the calling worker until the mutex is available, stalling all connections that worker is serving. In request hot paths where the shared data is not strictly required for correctness -- rate limit counters, cache updates, statistics -- use ngx_shmtx_trylock to attempt acquisition without blocking. If the lock is contended, fall back to serving stale data, skipping the update, or using a local estimate. This keeps worker latency deterministic under contention.

Incorrect (unconditional lock blocks worker during contention):

static ngx_int_t
ngx_http_ratelimit_check(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t    *shpool;
    my_ratelimit_t     *rl;
    ngx_msec_t          now;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    /* BAD: blocks worker — if another worker holds this lock during a
       slab GC or lengthy update, this worker stalls all its connections */
    ngx_shmtx_lock(&shpool->mutex);

    rl = (my_ratelimit_t *) shpool->data;
    now = ngx_current_msec;

    rl->tokens += (now - rl->last_refill) * rl->rate / 1000;
    if (rl->tokens > rl->burst) {
        rl->tokens = rl->burst;
    }
    rl->last_refill = now;

    if (rl->tokens < 1) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_HTTP_TOO_MANY_REQUESTS;
    }

    rl->tokens--;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_DECLINED;
}

Correct (trylock with graceful fallback when contended):

static ngx_int_t
ngx_http_ratelimit_check(ngx_http_request_t *r, ngx_shm_zone_t *zone)
{
    ngx_slab_pool_t    *shpool;
    my_ratelimit_t     *rl;
    ngx_msec_t          now;

    shpool = (ngx_slab_pool_t *) zone->shm.addr;

    if (!ngx_shmtx_trylock(&shpool->mutex)) {
        /* lock contended — allow request rather than stall worker;
           rate accuracy degrades slightly but latency stays bounded */
        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "ratelimit: lock contended, allowing request");
        return NGX_DECLINED;
    }

    rl = (my_ratelimit_t *) shpool->data;
    now = ngx_current_msec;

    rl->tokens += (now - rl->last_refill) * rl->rate / 1000;
    if (rl->tokens > rl->burst) {
        rl->tokens = rl->burst;
    }
    rl->last_refill = now;

    if (rl->tokens < 1) {
        ngx_shmtx_unlock(&shpool->mutex);
        return NGX_HTTP_TOO_MANY_REQUESTS;
    }

    rl->tokens--;

    ngx_shmtx_unlock(&shpool->mutex);

    return NGX_DECLINED;
}

Set Log Action String for Operation Context

When nginx logs an error, it automatically appends the string pointed to by log->action to the message (e.g., "while connecting to upstream"). Without setting this field, error messages lack operational context, forcing developers to grep through code to determine which phase produced the log entry. Setting log->action before each distinct operation costs only a pointer assignment at runtime but makes every subsequent error message self-documenting.

Incorrect (generic log messages without operation context):

static ngx_int_t
ngx_http_mymodule_process(ngx_http_request_t *r)
{
    ngx_int_t  rc;

    rc = ngx_http_mymodule_validate_headers(r);
    if (rc != NGX_OK) {
        /* produces: "header validation failed" — but which phase? */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "header validation failed");
        return rc;
    }

    rc = ngx_http_mymodule_transform_body(r);
    if (rc != NGX_OK) {
        /* produces: "body transformation error" — no automatic context */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "body transformation error");
        return rc;
    }

    rc = ngx_http_mymodule_send_upstream(r);
    if (rc != NGX_OK) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "upstream send failed");
        return rc;
    }

    return NGX_OK;
}

Correct (sets log->action before each operation for automatic context in errors):

static ngx_int_t
ngx_http_mymodule_process(ngx_http_request_t *r)
{
    ngx_int_t  rc;

    /* pointer assignment only — negligible cost */
    r->connection->log->action = "validating request headers in mymodule";

    rc = ngx_http_mymodule_validate_headers(r);
    if (rc != NGX_OK) {
        /* produces: "header validation failed while validating
         * request headers in mymodule" */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "header validation failed");
        return rc;
    }

    r->connection->log->action = "transforming response body in mymodule";

    rc = ngx_http_mymodule_transform_body(r);
    if (rc != NGX_OK) {
        /* produces: "body transformation error while transforming
         * response body in mymodule" */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "body transformation error");
        return rc;
    }

    r->connection->log->action = "sending request to upstream in mymodule";

    rc = ngx_http_mymodule_send_upstream(r);
    if (rc != NGX_OK) {
        /* any error inside send_upstream also gets this context
         * automatically, even from nginx core functions */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "upstream send failed");
        return rc;
    }

    return NGX_OK;
}

Attach Module Context to Connection Log for Tracing

When a module needs to include contextual information (request ID, upstream name, session token) in every log entry, formatting it manually in each ngx_log_error call duplicates work and bloats the code. Instead, set log->handler to a custom callback and log->data to your context structure once during request initialization. Nginx automatically invokes the handler to append context to every log line produced through that log object, giving you consistent tracing with zero per-call formatting overhead.

Incorrect (manually formats request ID into every log call):

static ngx_int_t
ngx_http_trace_handler(ngx_http_request_t *r)
{
    ngx_str_t   request_id;
    ngx_int_t   rc;

    request_id = get_request_id(r);

    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "[req_id=%V] starting upstream connection", &request_id);

    rc = connect_upstream(r);
    if (rc == NGX_ERROR) {
        /* BAD: must repeat the request_id formatting in every log call */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "[req_id=%V] upstream connection failed", &request_id);
        return NGX_HTTP_BAD_GATEWAY;
    }

    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "[req_id=%V] upstream connected successfully", &request_id);

    return NGX_OK;
}

Correct (sets log->handler and log->data once for automatic context injection):

typedef struct {
    ngx_str_t     request_id;
    ngx_log_t    *original_log;
} ngx_http_trace_log_ctx_t;

static u_char *
ngx_http_trace_log_handler(ngx_log_t *log, u_char *buf, size_t len)
{
    ngx_http_trace_log_ctx_t  *ctx;

    ctx = log->data;

    /* nginx calls this automatically for every log entry */
    return ngx_snprintf(buf, len, " req_id=%V", &ctx->request_id);
}

static ngx_int_t
ngx_http_trace_handler(ngx_http_request_t *r)
{
    ngx_http_trace_log_ctx_t  *ctx;
    ngx_int_t                  rc;

    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_trace_log_ctx_t));
    if (ctx == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    ctx->request_id = get_request_id(r);
    ctx->original_log = r->connection->log;

    /* attach once — all subsequent log calls include the request ID */
    r->connection->log->handler = ngx_http_trace_log_handler;
    r->connection->log->data = ctx;

    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "starting upstream connection");

    rc = connect_upstream(r);
    if (rc == NGX_ERROR) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "upstream connection failed");
        return NGX_HTTP_BAD_GATEWAY;
    }

    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "upstream connected successfully");

    return NGX_OK;
}

Deduplicate Repeated Error Messages with Throttling

When an upstream goes down or a resource becomes unavailable, the same error fires on every request and can produce thousands of identical log lines per second. This floods the disk, makes logs unreadable, and can saturate I/O bandwidth enough to degrade healthy request processing. Track the last time each recurring error was logged and suppress duplicates, emitting a periodic summary with the suppressed count instead.

Incorrect (logs every error occurrence, flooding disk during storms):

static ngx_int_t
ngx_http_proxy_connect(ngx_http_request_t *r, ngx_addr_t *addr)
{
    ngx_int_t  rc;

    rc = ngx_event_connect_peer(&r->upstream->peer);
    if (rc == NGX_ERROR) {
        /* BAD: under a sustained upstream outage this fires thousands
         * of times per second, saturating disk I/O */
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "upstream connection to %V failed", &addr->name);
        return NGX_HTTP_BAD_GATEWAY;
    }

    return NGX_OK;
}

Correct (throttles repeated errors and logs a periodic summary count):

typedef struct {
    ngx_msec_t      last_logged;
    ngx_atomic_t    suppressed;
} ngx_http_mymodule_error_state_t;

static ngx_http_mymodule_error_state_t  connect_error_state;

#define NGX_HTTP_MYMODULE_LOG_INTERVAL  5000  /* 5 seconds */

static ngx_int_t
ngx_http_proxy_connect(ngx_http_request_t *r, ngx_addr_t *addr)
{
    ngx_int_t    rc;
    ngx_msec_t   now;

    rc = ngx_event_connect_peer(&r->upstream->peer);
    if (rc == NGX_ERROR) {
        now = ngx_current_msec;

        if (now - connect_error_state.last_logged
            < NGX_HTTP_MYMODULE_LOG_INTERVAL)
        {
            /* suppress duplicate — just count it */
            ngx_atomic_fetch_add(&connect_error_state.suppressed, 1);

        } else {
            /* enough time has passed — emit summary and reset */
            if (connect_error_state.suppressed) {
                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                              "upstream connection to %V failed "
                              "(%uA occurrences suppressed in last %Ms)",
                              &addr->name,
                              connect_error_state.suppressed,
                              NGX_HTTP_MYMODULE_LOG_INTERVAL);
            } else {
                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                              "upstream connection to %V failed",
                              &addr->name);
            }

            connect_error_state.last_logged = now;
            connect_error_state.suppressed = 0;
        }

        return NGX_HTTP_BAD_GATEWAY;
    }

    return NGX_OK;
}

Guard Expensive Debug Argument Computation Behind Level Check

The ngx_log_debug macros already check log->log_level before formatting — when debug is disabled, they short-circuit. However, the macro arguments are evaluated before the macro call. If an argument involves a function call or computation (formatting a complex struct, computing a hash, walking a list), that work executes unconditionally — even when the result is discarded. Wrap expensive argument computation in a level check so the work is skipped entirely when debug is disabled.

Incorrect (expensive function call evaluated even when debug is disabled):

static ngx_int_t
ngx_http_mymodule_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_chain_t  *cl;

    for (cl = in; cl; cl = cl->next) {
        /* BAD: ngx_http_mymodule_dump_buf() runs on every buffer even
         * when debug is off — the macro short-circuits the format,
         * but the function call already happened (~5us per call) */
        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                        "mymodule filter: %s",
                        ngx_http_mymodule_dump_buf(r->pool, cl->buf));
    }

    return ngx_http_next_body_filter(r, in);
}

Correct (skips expensive computation when debug is disabled):

static ngx_int_t
ngx_http_mymodule_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_chain_t  *cl;
    ngx_log_t    *log;

    log = r->connection->log;

    for (cl = in; cl; cl = cl->next) {
        /* only compute the dump string when debug is actually enabled */
        if (log->log_level & NGX_LOG_DEBUG_HTTP) {
            ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0,
                            "mymodule filter: %s",
                            ngx_http_mymodule_dump_buf(r->pool, cl->buf));
        }
    }

    return ngx_http_next_body_filter(r, in);
}

When NOT to use this pattern:

• Simple scalar arguments (%p, %uz, %d) — the macro's built-in check is sufficient
• Outside hot paths — the overhead of a function call is negligible for infrequent debug logging

Collect Metrics via Shared Memory Counters

Logging every metric value as a text line (request latency, status codes, byte counts) generates enormous I/O volume under load and makes real-time aggregation impossible without external log parsing. Instead, use ngx_atomic_fetch_add to increment counters in a shared memory zone and expose the aggregated values through a lightweight status handler. This eliminates per-request disk writes entirely and provides instant metric reads at negligible cost.

Incorrect (logs every metric value as a text line, causing high I/O):

static ngx_int_t
ngx_http_metrics_log_handler(ngx_http_request_t *r)
{
    ngx_time_t   *tp;
    ngx_msec_t    latency;

    tp = ngx_timeofday();
    latency = (tp->sec - r->start_sec) * 1000 + (tp->msec - r->start_msec);

    /* BAD: writes to disk on every single request — disk I/O
     * becomes the bottleneck under load, not the upstream */
    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "METRIC status=%ui latency=%Mms bytes=%O",
                  r->headers_out.status, latency,
                  r->connection->sent);

    return NGX_DECLINED;
}

Correct (atomically increments shared memory counters, exposed via status handler):

typedef struct {
    ngx_atomic_t    requests;
    ngx_atomic_t    status_2xx;
    ngx_atomic_t    status_5xx;
    ngx_atomic_t    total_bytes;
    ngx_atomic_t    total_latency_ms;
} ngx_http_metrics_shm_t;

static ngx_int_t
ngx_http_metrics_log_handler(ngx_http_request_t *r)
{
    ngx_http_metrics_shm_t  *m;
    ngx_time_t              *tp;
    ngx_msec_t               latency;

    m = metrics_shm_zone->data;

    tp = ngx_timeofday();
    latency = (tp->sec - r->start_sec) * 1000 + (tp->msec - r->start_msec);

    /* lock-free counter updates — no I/O, no contention */
    ngx_atomic_fetch_add(&m->requests, 1);
    ngx_atomic_fetch_add(&m->total_bytes, r->connection->sent);
    ngx_atomic_fetch_add(&m->total_latency_ms, latency);

    if (r->headers_out.status >= 200 && r->headers_out.status < 300) {
        ngx_atomic_fetch_add(&m->status_2xx, 1);
    } else if (r->headers_out.status >= 500) {
        ngx_atomic_fetch_add(&m->status_5xx, 1);
    }

    return NGX_DECLINED;
}

/* expose via status handler: read atomic counters, format with ngx_sprintf */
/* static ngx_int_t ngx_http_metrics_status_handler(ngx_http_request_t *r) */

Use Exponential Backoff for Upstream Reconnection Attempts

When an upstream peer goes down, a naive immediate-retry loop causes all workers to hammer the recovering backend simultaneously -- the thundering herd effect can prevent the upstream from ever stabilizing. Using timer-based exponential backoff with a cap and jitter spreads reconnection attempts over time, giving the upstream breathing room to recover while keeping the module responsive to actual restoration.

Incorrect (immediate retry loop on upstream connection failure):

static void
ngx_http_myproxy_connect_to_backend(ngx_http_myproxy_peer_t *peer)
{
    ngx_int_t  rc;

retry:
    rc = ngx_event_connect_peer(&peer->pc);

    if (rc == NGX_ERROR || rc == NGX_DECLINED) {
        ngx_log_error(NGX_LOG_ERR, peer->log, 0,
                      "backend connect failed, retrying immediately");

        /* BUG: tight retry loop — all workers retry at once, overwhelming
         * the recovering backend and saturating the worker's event loop */
        goto retry;
    }

    if (rc == NGX_AGAIN) {
        ngx_add_timer(peer->pc.connection->write, peer->connect_timeout);
        return;
    }

    /* connected */
    ngx_http_myproxy_send_request(peer);
}

Correct (timer-based exponential backoff with jitter and cap):

#define MYPROXY_BACKOFF_BASE_MS    100
#define MYPROXY_BACKOFF_MAX_MS     30000
#define MYPROXY_BACKOFF_MAX_SHIFT  8       /* 2^8 * 100ms = 25600ms */

static void
ngx_http_myproxy_reconnect_handler(ngx_event_t *ev)
{
    ngx_http_myproxy_peer_t  *peer = ev->data;
    ngx_int_t                 rc;

    rc = ngx_event_connect_peer(&peer->pc);

    if (rc == NGX_ERROR || rc == NGX_DECLINED) {
        ngx_http_myproxy_schedule_reconnect(peer);
        return;
    }

    if (rc == NGX_AGAIN) {
        ngx_add_timer(peer->pc.connection->write, peer->connect_timeout);
        return;
    }

    /* connected — reset backoff state for next failure */
    peer->reconnect_attempts = 0;
    ngx_http_myproxy_send_request(peer);
}

static void
ngx_http_myproxy_schedule_reconnect(ngx_http_myproxy_peer_t *peer)
{
    ngx_msec_t  delay, jitter;
    ngx_uint_t  shift;

    /* exponential backoff: base * 2^attempts, capped at max */
    shift = ngx_min(peer->reconnect_attempts, MYPROXY_BACKOFF_MAX_SHIFT);
    delay = MYPROXY_BACKOFF_BASE_MS << shift;

    if (delay > MYPROXY_BACKOFF_MAX_MS) {
        delay = MYPROXY_BACKOFF_MAX_MS;
    }

    /* add jitter: +/- 25% to desynchronize workers */
    jitter = (ngx_random() % (delay / 2)) - (delay / 4);
    delay += jitter;

    peer->reconnect_attempts++;

    ngx_log_error(NGX_LOG_WARN, peer->log, 0,
                  "backend down, reconnect in %Mms (attempt %ui)",
                  delay, peer->reconnect_attempts);

    peer->reconnect_ev.handler = ngx_http_myproxy_reconnect_handler;
    peer->reconnect_ev.data = peer;
    peer->reconnect_ev.log = peer->log;

    ngx_add_timer(&peer->reconnect_ev, delay);
}

Set Client Body Timeout to Bound Slow-Client Resource Usage

A module that reads the client request body with ngx_http_read_client_request_body without enforcing a read timeout allows malicious or extremely slow clients to hold a connection open indefinitely -- sending one byte at a time to keep the socket alive. This is a variant of the slowloris attack that exhausts worker_connections. Setting client_body_timeout on the request's upstream config or adding an explicit timer in a custom body handler bounds the maximum time a client can occupy a worker connection during body transfer.

Incorrect (reads body with no timeout -- slow client holds connection forever):

static ngx_int_t
ngx_http_myupload_handler(ngx_http_request_t *r)
{
    ngx_int_t  rc;

    /* BUG: no client_body_timeout configured, and no explicit timer —
     * a client sending 1 byte/sec on a 10MB upload holds this worker
     * connection for ~10 million seconds */
    rc = ngx_http_read_client_request_body(r,
                                           ngx_http_myupload_body_handler);

    if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
        return rc;
    }

    return NGX_DONE;
}

static void
ngx_http_myupload_body_handler(ngx_http_request_t *r)
{
    /* process body — but no timeout was ever set during the read */
    ngx_http_myupload_process(r);
}

Correct (sets client body timeout and adds timer to bound slow reads):

static void *
ngx_http_myupload_create_loc_conf(ngx_conf_t *cf)
{
    ngx_http_myupload_loc_conf_t  *conf;

    conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_myupload_loc_conf_t));
    if (conf == NULL) {
        return NULL;
    }

    conf->client_body_timeout = NGX_CONF_UNSET_MSEC;

    return conf;
}

static char *
ngx_http_myupload_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
    ngx_http_myupload_loc_conf_t  *prev = parent;
    ngx_http_myupload_loc_conf_t  *conf = child;

    /* 60s between successive read operations — drop idle clients */
    ngx_conf_merge_msec_value(conf->client_body_timeout,
                              prev->client_body_timeout, 60000);

    return NGX_CONF_OK;
}

static ngx_int_t
ngx_http_myupload_handler(ngx_http_request_t *r)
{
    ngx_int_t                      rc;
    ngx_http_myupload_loc_conf_t  *mlcf;

    mlcf = ngx_http_get_module_loc_conf(r, ngx_http_myupload_module);

    /* set the client body timeout so ngx_http_read_client_request_body
     * arms a timer between each recv() call — slow clients get dropped */
    r->read_event_handler = ngx_http_block_reading;

    ngx_http_core_loc_conf_t *clcf;
    clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
    clcf->client_body_timeout = mlcf->client_body_timeout;

    rc = ngx_http_read_client_request_body(r,
                                           ngx_http_myupload_body_handler);

    if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
        return rc;
    }

    return NGX_DONE;
}

static void
ngx_http_myupload_body_handler(ngx_http_request_t *r)
{
    /* body fully received within timeout bounds — safe to process */
    if (r->request_body == NULL || r->request_body->bufs == NULL) {
        ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST);
        return;
    }

    ngx_http_myupload_process(r);
}

Configure next_upstream Mask for Retriable Failures

The next_upstream bitmask on ngx_http_upstream_conf_t controls which failures trigger automatic retry on the next peer. The default (NGX_HTTP_UPSTREAM_FT_ERROR | NGX_HTTP_UPSTREAM_FT_TIMEOUT) retries on connection errors and timeouts, but a module that blindly adds NGX_HTTP_UPSTREAM_FT_HTTP_500 or uses an overly broad mask will retry POST/PUT requests on 500 errors -- causing duplicate writes. Explicitly configure the mask to match your upstream's idempotency guarantees.

Incorrect (retries all failures including non-idempotent operations):

static void *
ngx_http_myproxy_create_loc_conf(ngx_conf_t *cf)
{
    ngx_http_myproxy_loc_conf_t  *conf;

    conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_myproxy_loc_conf_t));
    if (conf == NULL) {
        return NULL;
    }

    /* BUG: retries on every possible failure — a 500 on a POST that
     * already committed will be re-sent to the next upstream peer */
    conf->upstream.next_upstream = NGX_HTTP_UPSTREAM_FT_ERROR
                                 | NGX_HTTP_UPSTREAM_FT_TIMEOUT
                                 | NGX_HTTP_UPSTREAM_FT_HTTP_500
                                 | NGX_HTTP_UPSTREAM_FT_HTTP_502
                                 | NGX_HTTP_UPSTREAM_FT_HTTP_503
                                 | NGX_HTTP_UPSTREAM_FT_HTTP_504;

    conf->upstream.next_upstream_tries = 0;  /* unlimited retries */

    return conf;
}

Correct (limits retries to safe, transient failures with bounded attempts):

static void *
ngx_http_myproxy_create_loc_conf(ngx_conf_t *cf)
{
    ngx_http_myproxy_loc_conf_t  *conf;

    conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_myproxy_loc_conf_t));
    if (conf == NULL) {
        return NULL;
    }

    conf->upstream.next_upstream = NGX_CONF_BITMASK_SET;
    conf->upstream.next_upstream_tries = NGX_CONF_UNSET_UINT;
    conf->upstream.next_upstream_timeout = NGX_CONF_UNSET_MSEC;

    return conf;
}

static char *
ngx_http_myproxy_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
    ngx_http_myproxy_loc_conf_t  *prev = parent;
    ngx_http_myproxy_loc_conf_t  *conf = child;

    /* retry only on connection-level failures and 502/503 (peer down);
     * never retry on 500 (application error) or 504 (upstream timeout
     * that may have already committed a side effect) */
    ngx_conf_merge_bitmask_value(conf->upstream.next_upstream,
                                 prev->upstream.next_upstream,
                                 NGX_HTTP_UPSTREAM_FT_ERROR
                                 | NGX_HTTP_UPSTREAM_FT_TIMEOUT
                                 | NGX_HTTP_UPSTREAM_FT_HTTP_502
                                 | NGX_HTTP_UPSTREAM_FT_HTTP_503);

    /* bound total retry attempts and cumulative retry time */
    ngx_conf_merge_uint_value(conf->upstream.next_upstream_tries,
                              prev->upstream.next_upstream_tries, 2);

    ngx_conf_merge_msec_value(conf->upstream.next_upstream_timeout,
                              prev->upstream.next_upstream_timeout, 10000);

    return NGX_CONF_OK;
}

Set Separate Timeouts for Connect, Send, and Read Phases

Nginx's upstream defaults all three phase timeouts (connect, send, read) to 60 seconds. A single slow upstream that takes 55 seconds to accept a TCP connection still has the full 60 seconds for headers and another 60 for the body -- holding a worker for nearly three minutes on one request. Setting explicit per-phase timeouts lets you fail-fast on connection establishment while allowing more time for legitimate data transfer, keeping worker utilization predictable.

Incorrect (relies on 60s defaults for all upstream phases):

static ngx_int_t
ngx_http_myproxy_handler(ngx_http_request_t *r)
{
    ngx_http_upstream_t  *u;

    if (ngx_http_upstream_create(r) != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    u = r->upstream;

    /* BUG: no timeout configuration — all phases default to 60000ms,
     * a single stalled upstream can hold a worker for up to 180s */
    u->conf = &mlcf->upstream;

    u->create_request = ngx_http_myproxy_create_request;
    u->process_header = ngx_http_myproxy_process_header;
    u->finalize_request = ngx_http_myproxy_finalize_request;

    ngx_http_upstream_init(r);
    return NGX_DONE;
}

Correct (configures per-phase timeouts tuned to upstream SLA):

static void *
ngx_http_myproxy_create_loc_conf(ngx_conf_t *cf)
{
    ngx_http_myproxy_loc_conf_t  *conf;

    conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_myproxy_loc_conf_t));
    if (conf == NULL) {
        return NULL;
    }

    conf->upstream.connect_timeout = NGX_CONF_UNSET_MSEC;
    conf->upstream.send_timeout = NGX_CONF_UNSET_MSEC;
    conf->upstream.read_timeout = NGX_CONF_UNSET_MSEC;

    return conf;
}

static char *
ngx_http_myproxy_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
    ngx_http_myproxy_loc_conf_t  *prev = parent;
    ngx_http_myproxy_loc_conf_t  *conf = child;

    /* fail-fast on connect: 3s is enough for local/regional backends */
    ngx_conf_merge_msec_value(conf->upstream.connect_timeout,
                              prev->upstream.connect_timeout, 3000);

    /* send timeout: request bodies are small, 10s is generous */
    ngx_conf_merge_msec_value(conf->upstream.send_timeout,
                              prev->upstream.send_timeout, 10000);

    /* read timeout: response generation may take longer, allow 30s */
    ngx_conf_merge_msec_value(conf->upstream.read_timeout,
                              prev->upstream.read_timeout, 30000);

    return NGX_CONF_OK;
}

static ngx_int_t
ngx_http_myproxy_handler(ngx_http_request_t *r)
{
    ngx_http_upstream_t           *u;
    ngx_http_myproxy_loc_conf_t   *mlcf;

    mlcf = ngx_http_get_module_loc_conf(r, ngx_http_myproxy_module);

    if (ngx_http_upstream_create(r) != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    u = r->upstream;

    /* per-phase timeouts — connect fails fast, read allows backend processing */
    u->conf = &mlcf->upstream;

    u->create_request = ngx_http_myproxy_create_request;
    u->process_header = ngx_http_myproxy_process_header;
    u->finalize_request = ngx_http_myproxy_finalize_request;

    ngx_http_upstream_init(r);
    return NGX_DONE;
}

Understand Accept Mutex Impact on Connection Distribution

When accept_mutex is off (the default since nginx 1.11.3), every worker process is notified of a new connection on a shared listen socket, causing all of them to wake and race to accept() -- only one succeeds, and the rest waste CPU cycles returning to sleep. Module code that registers its own listen sockets or connection handlers must be aware of this behavior: if the module assumes exclusive ownership of incoming connections without coordinating through the accept mutex, it will either duplicate work or silently drop connections under high concurrency.

Incorrect (all workers wake and race to accept on shared socket):

static ngx_int_t
ngx_http_mymodule_init_process(ngx_cycle_t *cycle)
{
    ngx_listening_t  *ls;
    ngx_connection_t *c;

    ls = ngx_http_mymodule_get_listening(cycle);

    c = ngx_get_connection(ls->fd, cycle->log);
    if (c == NULL) {
        return NGX_ERROR;
    }

    c->log = cycle->log;
    c->read->handler = ngx_http_mymodule_accept_handler;

    /* BUG: every worker adds the listen fd to its own epoll set
     * without checking accept_mutex — all workers wake on every
     * incoming connection (thundering herd), wasting CPU */
    if (ngx_add_event(c->read, NGX_READ_EVENT, 0) == NGX_ERROR) {
        ngx_close_connection(c);
        return NGX_ERROR;
    }

    return NGX_OK;
}

Correct (respects accept mutex to serialize accept() across workers):

static ngx_int_t
ngx_http_mymodule_init_process(ngx_cycle_t *cycle)
{
    ngx_listening_t  *ls;
    ngx_connection_t *c;

    ls = ngx_http_mymodule_get_listening(cycle);

    c = ngx_get_connection(ls->fd, cycle->log);
    if (c == NULL) {
        return NGX_ERROR;
    }

    c->log = cycle->log;
    c->read->handler = ngx_http_mymodule_accept_handler;
    c->listening = ls;

    /* register the listen socket through nginx's event framework;
     * NGX_USE_ACCEPT_MUTEX_EVENT tells the event system to defer
     * adding this fd to epoll until the worker acquires the accept
     * mutex — only one worker polls the socket at a time */
    if (ngx_use_accept_mutex) {
        return NGX_OK;  /* ls will be added during ngx_trylock_accept_mutex */
    }

    /* accept_mutex is off — add directly but use EPOLLEXCLUSIVE
     * (NGX_EXCLUSIVE_EVENT) so the kernel wakes only one worker */
    if (ngx_add_event(c->read, NGX_READ_EVENT,
                      ngx_event_flags & NGX_USE_EPOLL_EVENT
                      ? NGX_EXCLUSIVE_EVENT : 0)
        == NGX_ERROR)
    {
        ngx_close_connection(c);
        return NGX_ERROR;
    }

    return NGX_OK;
}

Note: On Linux kernels 4.5+ with epoll, EPOLLEXCLUSIVE provides kernel-level thundering-herd avoidance without the accept mutex overhead. The correct approach checks for epoll availability via ngx_event_flags and falls back gracefully.

Use Pre-Allocated Free List for Module Data Structures

Allocating module-specific data structures on every request introduces pool fragmentation and allocation overhead in the hot path. When the structure size and maximum concurrency are known (e.g., one context per connection), pre-allocating a slab in init_process and recycling entries via a free list eliminates per-request ngx_palloc calls entirely -- turning a ~30ns allocation into a ~2ns pointer swap on every request.

Incorrect (allocates module context from request pool on every request):

typedef struct {
    ngx_buf_t    scratch;
    u_char       digest[32];
    ngx_uint_t   state;
} ngx_http_mymodule_ctx_t;

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_http_mymodule_ctx_t  *ctx;

    /* BUG: allocates 72+ bytes from the request pool on every single
     * request — under 50k req/s this is 3.6M allocations per minute,
     * fragmenting pools and pressuring the allocator in the hot path */
    ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_mymodule_ctx_t));
    if (ctx == NULL) {
        return NGX_ERROR;
    }

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    /* ... process request ... */

    return NGX_OK;
}

Correct (pre-allocates contexts at worker init and recycles via free list):

typedef struct {
    ngx_buf_t    scratch;
    u_char       digest[32];
    ngx_uint_t   state;
} ngx_http_mymodule_ctx_t;

/* per-worker free list — no lock needed, single-threaded */
static ngx_http_mymodule_ctx_t  *ctx_free_list;

static ngx_int_t
ngx_http_mymodule_init_process(ngx_cycle_t *cycle)
{
    ngx_http_mymodule_ctx_t  *pool_block, *entry;
    ngx_uint_t                i, n;

    n = cycle->connection_n;
    pool_block = ngx_alloc(n * sizeof(ngx_http_mymodule_ctx_t), cycle->log);
    if (pool_block == NULL) { return NGX_ERROR; }

    /* intrusive singly-linked free list */
    ctx_free_list = NULL;
    for (i = 0; i < n; i++) {
        entry = &pool_block[i];
        *(ngx_http_mymodule_ctx_t **) entry = ctx_free_list;
        ctx_free_list = entry;
    }
    return NGX_OK;
}

static ngx_int_t
ngx_http_mymodule_handler(ngx_http_request_t *r)
{
    ngx_http_mymodule_ctx_t  *ctx;
    ngx_pool_cleanup_t       *cln;

    if (ctx_free_list == NULL) {
        /* fallback: all pre-allocated contexts in use */
        ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_mymodule_ctx_t));
        if (ctx == NULL) {
            return NGX_ERROR;
        }
    } else {
        /* ~2ns pointer swap instead of pool allocation */
        ctx = ctx_free_list;
        ctx_free_list = *(ngx_http_mymodule_ctx_t **) ctx;
        ngx_memzero(ctx, sizeof(ngx_http_mymodule_ctx_t));

        /* register cleanup to return ctx to free list on request end */
        cln = ngx_pool_cleanup_add(r->pool, 0);
        if (cln == NULL) { return NGX_ERROR; }
        cln->handler = ngx_http_mymodule_ctx_release;
        cln->data = ctx;
    }

    ngx_http_set_ctx(r, ctx, ngx_http_mymodule_module);

    return NGX_OK;
}

Note: The free list is per-worker and accessed only from the event loop thread, so no locking is required. Register a ngx_pool_cleanup_t handler to return the context to the free list when the request pool is destroyed.