How Nginx Works Under the Hood

You've configured Nginx dozens of times. You know what proxy_pass does, you've tuned worker_processes, you've written location blocks that almost made sense. But here's the thing that trips people up: Nginx isn't just "fast Apache." It has a fundamentally different architecture — one that lets a single machine with 4 cores serve hundreds of thousands of concurrent connections without breaking a sweat.

Let's look at what's actually happening inside the binary.

The two-process tiers

When you run Nginx, you don't get one process. You get at least two tiers.

The master process runs as root (or with elevated privileges). It's basically a supervisor. It reads the config, binds to ports (the privileged operation), spawns workers, and then mostly waits for signals. SIGHUP means reload config. SIGUSR1 means reopen log files. SIGTERM means shut down. That's the master's entire job after startup.

The worker processes do all the actual work. By default, worker_processes auto gives you one worker per CPU core. Each worker is completely independent — no shared heap, no message passing, no locks between them (with one exception: shared memory zones for things like rate limiting and cache metadata). A bug that corrupts one worker's memory doesn't take down the others.

# nginx.conf
worker_processes auto;  # one per CPU core
 
events {
  worker_connections 1024;  # max connections per worker
}

There are also helper processes, and they are separate processes, not threads in a worker. The cache manager enforces max_size and min_free on the cache directory, evicting least-recently-used entries in bounded iterations (100 files, 200ms, then a 50ms pause). The cache loader runs once, a minute after startup, and — per nginx's docs — "loads information about previously cached data stored on file system into a cache zone." Metadata, not content. It populates the shared-memory keys zone so workers can tell a cache hit from a miss; the response bodies stay on disk until something asks for them.

The event loop: why there's no thread-per-connection

This is the core of everything. A traditional server (older Apache with prefork, for instance) gives each connection its own thread or process. That model works fine at low concurrency, but threads are expensive.

It is worth being careful about how expensive, because the usual version of this argument is wrong. glibc's default thread stack is 8 MiB — you can read it back with pthread_attr_getstacksize, which returns 8388608 on a stock Linux box. Multiply that by 10,000 connections and you get "80GB of RAM", which is the number everyone quotes and which is not true. Stacks are mapped lazily; you reserve address space, you don't commit pages you never touch. Spawning 1,000 threads with 8 MiB stacks on the machine I wrote this on:

before:            VmSize=16 MiB  VmRSS=8 MiB
with 1000 threads: VmSize=10067 MiB  VmRSS=25 MiB
delta: virtual +10050 MiB, resident +17 MiB

Ten gigabytes of virtual address space; seventeen megabytes of actual memory. So the real costs of thread-per-connection are the ones that don't show up as RSS: scheduler run-queue pressure, context-switch cost at every I/O boundary, and — on 32-bit, historically — genuinely running out of address space. The RAM argument is a myth that happens to point at a correct conclusion.

Nginx takes a different approach: each worker runs a single-threaded event loop. One thread, thousands of connections.

Here's the simplified inner loop in pseudocode:

// What each Nginx worker does, forever
while (true) {
    // Ask the kernel: which of my registered fds are ready?
    int n = epoll_wait(epoll_fd, events, MAX_EVENTS, timeout_ms);
 
    for (int i = 0; i < n; i++) {
        if (events[i].fd == listen_socket) {
            // New connection arriving
            int conn_fd = accept(listen_socket, ...);
            set_nonblocking(conn_fd);
            epoll_ctl(epoll_fd, EPOLL_CTL_ADD, conn_fd, EPOLLIN);
        } else {
            // Existing connection has data or is writable
            handle_connection(events[i].fd);
        }
    }
}

The critical detail is epoll_wait. This is a Linux system call that says "block until at least one of the file descriptors I've registered becomes ready." The kernel maintains the watch list in kernel space and wakes the worker only when there's actual work. No polling, no spin-waiting.

An idle keepalive connection consumes essentially nothing: it's just a file descriptor number in the kernel's epoll table. The worker thread is sleeping in epoll_wait. The connection exists without costing CPU.

On BSD/macOS, Nginx uses kqueue instead of epoll. On older Solaris, /dev/poll. The abstraction layer in Nginx's source (ngx_event.h) hides this behind a common interface — the platform-specific event module gets selected at compile time.

The 11-phase HTTP pipeline

Once a worker has read enough bytes to parse a request, it doesn't just jump straight to sending a response. Nginx processes every HTTP request through an 11-phase pipeline. Modules hook into specific phases to do their work.

When you write a module or use OpenResty/Lua hooks, you're inserting handlers into specific phases. The satisfy directive in the ACCESS phase is a good example of why this design matters — it lets multiple access modules (IP whitelist AND auth_basic, OR either) compose cleanly without any of them knowing about each other.

Only one handler wins the CONTENT phase. And when people say "return beats proxy_pass in the same location block", the reason usually given — that return is registered earlier in the content phase — is wrong. return isn't a content handler at all. It's a directive of ngx_http_rewrite_module, whose handler is pushed onto phases[NGX_HTTP_SERVER_REWRITE_PHASE] and phases[NGX_HTTP_REWRITE_PHASE]. It wins because it finalises the request two phases before the content phase is ever reached. Same observable behaviour, completely different mechanism — and the difference is what tells you that return also short-circuits your access and auth_request handlers, which a content handler would not.

Memory pools: malloc is slow

Every Nginx request gets its own memory pool (ngx_pool_t). Instead of calling malloc and free for each individual allocation during a request, Nginx pre-allocates a slab of memory when the request starts and does bump-pointer allocation from it:

// Simplified pool allocation
void *ngx_palloc(ngx_pool_t *pool, size_t size) {
    if (size <= pool->max) {
        // small allocation: bump the pointer
        u_char *m = pool->current->d.last;
        if (m + size <= pool->current->d.end) {
            pool->current->d.last = m + size;
            return m;
        }
        // current block full, allocate new block
        return ngx_palloc_block(pool, size);
    }
    // large allocation: go to system malloc, track pointer for cleanup
    return ngx_palloc_large(pool, size);
}
 
// When request finishes:
ngx_destroy_pool(pool);  // frees everything in one shot

The payoff: no fragmentation, no per-object free overhead, and no risk of leaking individual allocations across requests. When the request completes or errors, the whole pool goes at once.

Not in O(1), though, and it's worth reading ngx_destroy_pool rather than assuming. It walks three linked lists: every registered cleanup handler, then every large allocation (each gets its own ngx_free), then every pool block. So it's linear in blocks plus large allocations plus cleanups — which is still far cheaper than one free per object, because the small allocations that make up the bulk of a request never appear in any of those lists. They just vanish with the block they were bumped out of.

For shared state across workers — rate limiting counters, upstream health state, shared cache metadata — Nginx uses a separate slab allocator in a POSIX shared memory zone. This is what limit_req_zone creates:

limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;

The 10m is 10 megabytes of shared memory. All workers read and write this zone under a mutex. Because it's shared memory (not a socket or pipe), the coordination overhead is minimal.

Upstream buffering: why your backend doesn't talk to the client

When Nginx proxies to an upstream (your Node.js app, your Django server), it doesn't just pipe bytes between the client and backend. By default (proxy_buffering on) it decouples the two ends — and it's worth quoting the docs, because "buffers the entire response before sending anything" is a common and wrong summary:

When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. […] When buffering is disabled, the response is passed to a client synchronously, immediately as it is received.

So nginx reads from the upstream flat out and starts writing to the client concurrently — proxy_busy_buffers_size exists precisely to bound "buffers that can be busy sending a response to the client while the response is not yet fully read." The point isn't to withhold the response, it's that nginx will never make the upstream wait for a slow client. Backend workers are expensive; nginx will spill to a temp file rather than let one dial-up user occupy a Django thread.

location /api/ {
    proxy_pass http://backend;
 
    # buffer up to 4 × 8k = 32k in memory per response
    proxy_buffers 4 8k;
    proxy_buffer_size 4k;  # first buffer (headers)
 
    # if response > 32k, spill to a temp file
    proxy_max_temp_file_size 1024m;
}

For streaming responses (SSE, chunked transfer), you want to disable this:

proxy_buffering off;

Now Nginx pipes bytes through immediately. Your backend connection stays open until the client disconnects, but for push-based use cases that's exactly what you want.

Zero-downtime config reloads

This is one of Nginx's most practical features and the process model is what makes it work.

When you run nginx -s reload (or kill -SIGHUP $(cat nginx.pid)):

1. The master re-reads the config, checks syntax, then tries to apply it — opening log files and new listen sockets. If any of that fails it rolls back and keeps running the old configuration; the old workers never notice.
2. The master spawns new workers using the new config.
3. The master tells the old workers to shut down gracefully. Note the mechanism: ngx_signal_worker_processes writes an NGX_CMD_QUIT message over the socketpair channel to each worker and only falls back to kill() if the channel write fails. The docs describe it as "sends messages to old worker processes requesting them to shut down gracefully" — sending a raw SIGQUIT to a worker works too, it just isn't what a reload does.
4. Old workers close their listen sockets, finish any in-flight requests, then exit.

For a few seconds, old and new workers coexist. Clients in the middle of long-running requests (big uploads, slow proxied responses) complete normally. New connections go to the new workers.

No dropped connections. No restart required. This is why you can change proxy_pass targets, update TLS certificates, or tune buffer sizes on a live production server.

Practical implications

worker_processes auto is almost always right. One worker per core, which is what nginx recommends and what the event-loop design assumes. The cases where you deviate are ones you can name: fewer workers when nginx is sharing a box with an application server that needs the cores, and worker_cpu_affinity when you want to pin workers rather than change their count. If your bottleneck is TLS handshake CPU, more workers is the direction that helps, not fewer — there is no version of this where leaving cores idle makes crypto faster.

worker_connections is per worker, not total. worker_processes 4; worker_connections 1024 means 4,096 total connections max — and each connection uses two file descriptors when proxying (one client-side, one upstream). Make sure worker_rlimit_nofile is set high enough:

worker_rlimit_nofile 65535;
 
events {
    worker_connections 16384;
}

Proxy buffer tuning matters for throughput. If your upstream returns large responses (APIs returning big JSON blobs), small proxy_buffers means Nginx spills to a temp file. Know what you're changing from: the default is proxy_buffers 8 4k|8k — eight buffers of one memory page, so 32k on a 4k-page platform and 64k on an 8k one — not "8k". Raising it to proxy_buffers 8 16k (128k) or proxy_buffers 4 32k is a real change; "raising it from 8k" is a misreading of the default. Watch for an upstream response is buffered to a temporary file in your error log — that's the signal that you're paying for disk I/O you could have avoided.

keepalive upstream connections are a genuine win. By default, Nginx closes the upstream TCP connection after each request. Add this to get connection pooling to your backends:

upstream backend {
    server 127.0.0.1:3000;
    keepalive 32;  # pool of up to 32 persistent connections per worker
}
 
location / {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";  # required for keepalive to work
}

On high-throughput services, this eliminates the TCP handshake + slow-start penalty for every single request to your backend. At 10k req/s, that's 10k TCP handshakes per second you just avoided.

The underlying insight in all of this is the same: Nginx was designed around the constraint that the OS kernel is better at multiplexing I/O than userspace threads are. The event loop isn't an optimization bolted on — it's the entire architectural premise, carried consistently from how workers are sized to how memory is allocated to how config reloads work.