Node.js Graceful Shutdown — Handle SIGTERM, Drain Connections, and Exit Without Losing Data

Your Node.js service is running fine in production. Kubernetes rolls out a new version. It sends SIGTERM to the old pod. You have 30 seconds before SIGKILL arrives. What does your app do?

If you haven't explicitly handled it: nothing. Node.js exits immediately on SIGTERM by default. Every in-flight HTTP request gets dropped. Every database transaction mid-flight gets severed. Every queued job your worker was processing disappears. Users get connection reset errors they never asked for.

This is the graceful shutdown problem. It's been solved many times — but the implementations floating around production codebases still get it half-right.

What Goes Wrong Without It

Here's what most Node.js services actually do:

// server.ts — the naive version (don't do this)
import express from 'express'
import { PrismaClient } from '@prisma/client'
 
const app = express()
const prisma = new PrismaClient()
 
app.get('/orders/:id', async (req, res) => {
  // This request could be mid-flight when SIGTERM hits
  const order = await prisma.order.findUnique({
    where: { id: req.params.id },
  })
  // If we exit here, the client gets a connection reset
  res.json(order)
})
 
const server = app.listen(3000, () => {
  console.log('Server running on port 3000')
})
 
// No signal handlers. SIGTERM kills the process immediately.
// Kubernetes waits 30s then sends SIGKILL anyway, but your
// in-flight requests are already dead.

Work out your own blast radius with Little's Law rather than guessing: concurrency is arrival rate times service time, so a service at 500 RPS with a 50 ms mean response time has about 25 requests in flight at any instant. Every abrupt restart drops all of them. Twenty rolling-deploy restarts a day and that's 500 failed requests — and none of them appear in your error-rate dashboard, because the connection never returns a response to count.

The Signal Landscape

Before fixing it, understand what you're handling:

• SIGTERM — the polite shutdown. Sent by Kubernetes, systemd, PM2, and Docker. The standard "please stop soon" signal.
• SIGINT — Ctrl+C. Used in local development. Treat it the same as SIGTERM.
• SIGHUP — traditionally "terminal closed". Some process managers use it to signal config reload. Usually fine to ignore or restart.
• SIGKILL — cannot be caught. Node won't even let you pretend: process.on('SIGKILL', fn) throws uv_signal_start on registration. The OS kills the process; you get exit status 137. Kubernetes sends this after terminationGracePeriodSeconds expires (default: 30 seconds).

You get SIGTERM, you do your cleanup, you exit before SIGKILL arrives. That's the entire contract.

The Shutdown Sequence

The correct order for a typical Node.js API with a database and a queue:

SIGTERM received
    → flip isShuttingDown = true (health check returns 503)
    → wait ~5s for load balancer to stop routing new traffic
    → server.close() — stop accepting new TCP connections
    → drain in-flight HTTP requests (wait for active request count = 0)
    → stop queue workers (finish current job, don't pick up new ones)
    → disconnect from database
    → disconnect from Redis/cache
    → process.exit(0)
    
    (timeout: force process.exit(1) after 25s)

The 5-second delay before server.close() is not arbitrary. Your load balancer (or Kubernetes Ingress + kube-proxy) takes a few seconds to propagate the endpoint removal. If you close immediately, you'll drop requests that were already in-flight from the load balancer before it knew you were going down.

Building the Shutdown Manager

The cleanest way to handle this is a central shutdown registry. Each resource registers a cleanup function. On shutdown, they run in order with a hard timeout.

// lib/shutdown-manager.ts
type CleanupFn = () => Promise<void>
 
interface Registration {
  name: string
  fn: CleanupFn
  timeoutMs: number
}
 
class ShutdownManager {
  private registrations: Registration[] = []
  private isShuttingDown = false
 
  register(name: string, fn: CleanupFn, timeoutMs = 10_000): void {
    this.registrations.push({ name, fn, timeoutMs })
  }
 
  get shuttingDown(): boolean {
    return this.isShuttingDown
  }
 
  async shutdown(exitCode = 0): Promise<never> {
    if (this.isShuttingDown) {
      // A second SIGTERM arrived. Never resolves — the first call owns the exit.
      return new Promise<never>(() => {})
    }
    this.isShuttingDown = true
    console.log('[shutdown] Starting graceful shutdown...')
 
    for (const { name, fn, timeoutMs } of this.registrations) {
      let timer: NodeJS.Timeout | undefined
      try {
        await Promise.race([
          fn(),
          new Promise<never>((_, reject) => {
            timer = setTimeout(() => reject(new Error(`Timeout after ${timeoutMs}ms`)), timeoutMs)
          }),
        ])
        console.log(`[shutdown] ${name}: done`)
      } catch (err) {
        console.error(`[shutdown] ${name}: failed —`, err)
        // Don't abort — keep cleaning up other resources
      } finally {
        clearTimeout(timer) // otherwise every step leaves a live timer behind
      }
    }
 
    console.log('[shutdown] All cleanup complete. Exiting.')
    process.exit(exitCode)
  }
}
 
export const shutdownManager = new ShutdownManager()

// server.ts — the full implementation
import http from 'http'
import express from 'express'
import { PrismaClient } from '@prisma/client'
import { shutdownManager } from './lib/shutdown-manager'
 
const app = express()
const prisma = new PrismaClient()
const server = http.createServer(app)
 
// --- Health check that respects shutdown state ---
app.get('/health/ready', (req, res) => {
  if (shutdownManager.shuttingDown) {
    res.status(503).json({ status: 'shutting_down' })
  } else {
    res.status(200).json({ status: 'ready' })
  }
})
 
// --- Track active request count for HTTP draining ---
// Decrement EXACTLY ONCE. 'close' always fires on the response, after 'finish'
// on a normal request and on its own if the client hangs up, so listening to
// both double-counts and drives this negative. See the section below.
let activeRequests = 0
 
app.use((req, res, next) => {
  activeRequests++
  res.on('close', () => { activeRequests-- })
  next()
})
 
// --- Register cleanup handlers in shutdown order ---
 
// 1. HTTP server: stop listening, drain in flight, then reap the sockets that
//    went idle *during* the drain — server.close() only reaped the ones that
//    were already idle when it was called.
shutdownManager.register('http-server', () => {
  return new Promise<void>((resolve, reject) => {
    server.close((err) => (err ? reject(err) : resolve()))
 
    if (activeRequests > 0) {
      console.log(`[shutdown] draining ${activeRequests} in-flight requests`)
    }
    const poll = setInterval(() => {
      if (activeRequests === 0) {
        clearInterval(poll)
        server.closeIdleConnections()
      }
    }, 100)
    poll.unref()
  })
}, 15_000)
 
// 2. Database — after the last request that might use it has finished
shutdownManager.register('prisma', async () => {
  await prisma.$disconnect()
}, 5_000)
 
// --- Signal handlers ---
const HARD_TIMEOUT_MS = 25_000
 
const handleShutdown = (signal: string) => {
  console.log(`[shutdown] Received ${signal}`)
  // Last resort: if a cleanup step wedges past its own timeout, still exit
  // under our own control rather than waiting for SIGKILL.
  setTimeout(() => {
    console.error('[shutdown] Hard timeout reached. Forcing exit.')
    process.exit(1)
  }, HARD_TIMEOUT_MS).unref()
 
  void shutdownManager.shutdown(0)
}
 
process.on('SIGTERM', () => handleShutdown('SIGTERM'))
process.on('SIGINT', () => handleShutdown('SIGINT'))
 
// --- Catch unhandled errors in production ---
process.on('uncaughtException', (err) => {
  console.error('[crash] Uncaught exception:', err)
  void shutdownManager.shutdown(1)
})
 
process.on('unhandledRejection', (reason) => {
  console.error('[crash] Unhandled rejection:', reason)
  void shutdownManager.shutdown(1)
})
 
// --- Start ---
server.listen(3000, () => {
  console.log('Server listening on port 3000')
})

The Keep-Alive Problem — and Why Most Write-Ups Get It Wrong

Every graceful-shutdown post you'll find says the same thing: server.close() stops accepting new connections but leaves idle keep-alive sockets open, so the callback never fires. That was true once. It has not been true since Node 19. From the server.close() docs: "Stops the server from accepting new connections and closes all connections connected to this server which are not sending a request or waiting for a response", with a changelog entry against v19.0.0 reading "The method closes idle connections before returning."

Measured, on Node 22.22.3 — open a keep-alive connection, complete one request, leave it idle, then close:

response received, socket left IDLE + keep-alive
server.close() callback fired after 0 ms

Zero milliseconds. But there is a real problem here, and it's the one nobody writes about, because it only shows up when you actually drain. server.close() reaps the connections that are idle at the moment you call it. A connection carrying an in-flight request is not idle, so it survives — and when that request finishes, the socket goes back to keep-alive idle and nothing reaps it. Same script, but with a 1500 ms request already in flight when close() is called:

  205 ms  calling server.close()
 1507 ms  res.end
 1511 ms  client got: HTTP/1.1 200 OK / Connection: keep-alive
 7521 ms  close cb fired

Six seconds of dead air, and note the Connection: keep-alive header on a response written after close() was called. What eventually reaps the socket is keepAliveTimeout (5000 ms) plus keepAliveTimeoutBuffer (1000 ms, added in v22.19.0) — 1511 + 6000 ≈ 7511.

The fix is one line, in the right place: after the in-flight count reaches zero, call server.closeIdleConnections(). Same scenario:

  213 ms  server.close()
 1516 ms  client got response
 1559 ms  drained, calling closeIdleConnections()
 1560 ms  close cb fired

That's what the registration above does, and it's what the docs recommend — "calling this after server.close is recommended as to avoid race conditions where new connections are created between a call to this and a call to server.close."

Test It With A Real Signal

Everything above is worth exactly nothing until you've sent the process an actual SIGTERM and timed it. This is the whole test:

node server.js > out.txt 2>&1 &
PID=$!
sleep 0.7
PORT=$(grep -oE '[0-9]+' out.txt | head -1)
# open a keep-alive connection, make one request, leave it idle
(printf 'GET / HTTP/1.1\r\nHost: x\r\nConnection: keep-alive\r\n\r\n'; sleep 30) \
  | timeout 30 nc 127.0.0.1 $PORT > /dev/null &
sleep 1.5
S=$(date +%s.%N); kill -TERM $PID; wait $PID; E=$(date +%s.%N)
echo "exited after $(echo "$E - $S" | bc) s"
cat out.txt

I wrote this section because I ran that harness against the naive version of the code above — the one that listens to both finish and close, and registers the drain as a separate step after server.close(). One completed request, one idle keep-alive socket, no traffic at all:

exited after 20.019464343 s
[shutdown] Received SIGTERM
[shutdown] Starting graceful shutdown...
[shutdown] http-server: done (+2ms)
[shutdown] Waiting for -1 in-flight requests...
[shutdown] in-flight-requests: failed after 20003ms — Timeout after 20000ms
[shutdown] All cleanup complete. Exiting.

Waiting for -1 in-flight requests. The counter went negative because res emits finish and then close on a normal request, so a single request decremented twice. activeRequests === 0 is never true again, the poll spins until its 20-second timeout, and the process burns two thirds of a 30-second grace period doing nothing. Every request after the first pushes it further negative.

With one decrement and closeIdleConnections() after the drain:

exited after .005425459 s
[shutdown] Received SIGTERM
[shutdown] Starting graceful shutdown...
[shutdown] http-server: done (+1ms)
[shutdown] All cleanup complete. Exiting.

Five milliseconds. Use close alone, never finish: on a normal request close fires after finish, and if the client hangs up mid-response finish never fires but close still does. It's the only event that fires exactly once on both paths.

Adding Queue Workers

If you're running BullMQ, Bee-Queue, or any other job worker alongside your HTTP server, they need their own cleanup:

// With BullMQ
import { Worker } from 'bullmq'
import { connection } from './lib/redis'
 
const emailWorker = new Worker(
  'emails',
  async (job) => {
    // Process job...
  },
  { connection }
)
 
// Register after HTTP server in shutdown sequence
shutdownManager.register('bullmq-worker', async () => {
  await emailWorker.close()
}, 20_000)
 
// Redis connection last, after the worker that uses it
shutdownManager.register('redis', async () => {
  await connection.quit()
}, 5_000)

BullMQ's own graceful-shutdown guide describes close() as marking the worker as closing "so it will not pick up new jobs, and at the same time it will wait for all the current jobs to be processed (or failed)" — and, importantly, "this call will not timeout by itself". That last part is why the timeout wrapper matters here more than anywhere else: an unbounded wait inside a bounded grace period is a SIGKILL waiting to happen. Keep this step's timeout under the grace period you've actually configured; a 30-second timeout inside a 30-second terminationGracePeriodSeconds is not a timeout, it's a coin flip.

Order matters for the obvious reason: a worker that's still finishing a job needs Redis to record the result. Close consumers before the connections they consume through — reverse of the order you opened them. BullMQ's stalled-job mechanism means an ungraceful kill isn't data loss, but it does mean another worker re-runs the job, so your handlers need to be idempotent either way.

Kubernetes Integration

Get the nesting right, because this is the mistake I keep finding in copy-pasted manifests: terminationGracePeriodSeconds is a pod spec field, but lifecycle is a container field. Put lifecycle next to containers and the YAML still parses — it just doesn't do anything.

# deployment.yaml — podSpec
spec:
  terminationGracePeriodSeconds: 40  # preStop (5s) + cleanup (25s) + buffer
  containers:
    - name: api
      livenessProbe:
        httpGet:
          path: /health/live       # must keep returning 200 during shutdown
          port: 3000
        initialDelaySeconds: 5
        periodSeconds: 10
      readinessProbe:
        httpGet:
          path: /health/ready     # flips to 503 as soon as SIGTERM lands
          port: 3000
        periodSeconds: 5
        failureThreshold: 1
      lifecycle:                  # container-level, not pod-level
        preStop:
          exec:
            # Buys kube-proxy time to update iptables on every node
            # before the container gets its TERM.
            command: ["/bin/sh", "-c", "sleep 5"]

Two details that change how you reason about this.

The readiness probe is not what takes you out of rotation. Per the Kubernetes docs, "at the same time as the kubelet is starting graceful shutdown of the Pod, the control plane evaluates whether to remove that shutting-down Pod from EndpointSlice objects", and terminating endpoints "always have their ready status as false, so load balancers will not use it for regular traffic". Endpoint removal is concurrent with your SIGTERM, not a consequence of your probe failing. The 503 still earns its keep — for ingress controllers and external LBs that health-check pods directly and don't watch EndpointSlices — but if you were relying on failureThreshold: 1 to be the mechanism, it isn't, and up to 5 seconds of periodSeconds would elapse before it noticed anyway. That's what the preStop sleep is actually covering.

Your liveness probe must keep passing while you drain. If /health/live starts failing during a long shutdown, the kubelet's answer is to restart the container out from under your cleanup. Only readiness flips.

Variations

Serverless / Lambda

Don't port this pattern over. The execution environment is frozen between invocations rather than continuously running, so there is no "drain in-flight requests" phase to implement, and whether your handler receives a SIGTERM at all depends on the runtime's shutdown semantics rather than on anything in your code. Check your provider's documented shutdown hooks before writing a signal handler you can't test.

Worker-only processes (no HTTP server)

Skip the HTTP draining steps. Your shutdown sequence is just: stop consuming from queue → finish current job → disconnect connections → exit.

// Background worker, no HTTP
process.on('SIGTERM', async () => {
  await worker.close()        // Finish current job
  await prisma.$disconnect()
  await redisClient.quit()
  process.exit(0)
})

Multiple HTTP servers (e.g., internal + external ports)

Write the close-and-reap dance once, then register it per server. Registrations run sequentially, so if you'd rather drain both at the same time, wrap them in one step:

import type { Server } from 'http'
 
const closeServer = (srv: Server, active: () => number) =>
  new Promise<void>((resolve, reject) => {
    srv.close((err) => (err ? reject(err) : resolve()))
    const poll = setInterval(() => {
      if (active() === 0) { clearInterval(poll); srv.closeIdleConnections() }
    }, 100)
    poll.unref()
  })
 
// Sequential: admin drains only after public is done
shutdownManager.register('public-server', () => closeServer(publicServer, () => publicActive))
shutdownManager.register('admin-server', () => closeServer(adminServer, () => adminActive))
 
// Or concurrently, in one registration
shutdownManager.register('all-servers', () =>
  Promise.all([
    closeServer(publicServer, () => publicActive),
    closeServer(adminServer, () => adminActive),
  ]).then(() => undefined)
)

When NOT to Use This Pattern

Single-request CLI tools — if your script makes one API call and exits, there's nothing to drain. process.exit() is fine.

Stateless edge functions — Cloudflare Workers and similar runtimes don't expose signal handlers. They have request-scoped lifecycles and cleanup through waitUntil() instead.

Development with nodemon — nodemon already waits for your process to exit cleanly before restarting. The pattern still applies, but you won't feel the pain without it in dev.

When you don't have long-running requests — if every request completes in under 50ms and you have no workers or transactions, a 1-second timeout is enough. Graceful shutdown here is about correctness, but the blast radius of getting it wrong is small.

Why That Hard Timeout Is .unref()'d

The per-registration timeouts protect you from one wedged cleanup step. The setTimeout in handleShutdown protects you from everything else — a step that swallows its own rejection, a $disconnect() that hangs below the driver, a listener you forgot to close. Kubernetes will SIGKILL you eventually, but then you lose the exit code and any final log line, and "why did the pod exit 137" is a much worse investigation than "why did the pod log 'hard timeout reached'".

The .unref() is what makes it safe to arm unconditionally. An active Node timer counts as a reference on the event loop, so a plain 25-second setTimeout would itself keep the process alive for 25 seconds after cleanup finished — you'd have built a shutdown handler whose only remaining job is waiting for its own watchdog. Unreferenced, the timer still fires if something else is keeping the loop alive, and gets ignored if nothing is.

Same reason the drain poller inside the http-server registration is unreffed. A setInterval you forgot to clear on the timeout path is a process that never exits, and the failure mode is identical to the one you were trying to prevent.

The Checklist

Before shipping a Node.js service to production, verify:

• process.on('SIGTERM') and process.on('SIGINT') are both handled
• /health/ready returns 503 during shutdown — and /health/live keeps returning 200
• In-flight requests are counted with res.on('close') only, never finish as well
• server.closeIdleConnections() is called once the in-flight count hits zero
• server.close() is called before disconnecting databases
• Queue workers have their own cleanup registered, after the HTTP server, with a timeout smaller than the grace period
• Resources close in dependency order: consumers before connections
• A hard timeout forces exit if cleanup hangs, and it's .unref()'d
• terminationGracePeriodSeconds exceeds preStop + cleanup timeout, with room to spare
• lifecycle.preStop is nested under the container, not the pod spec
• uncaughtException and unhandledRejection both trigger the shutdown path
• You have sent the running process a real kill -TERM and timed how long it took to exit

Graceful shutdown is boring infrastructure. You write it once, test it occasionally, and it never comes up in retros. Until you forget it, and then it comes up in every incident retro for six months.