Node.js Streams: Process Large Data Without Running Out of Memory

You have a 2 GB CSV of user events you need to process nightly. Straightforward enough — read the file, parse the rows, write the results to a database. You write the obvious version:

const fs = require('node:fs');
const data = fs.readFileSync('./events.csv', 'utf8');
const rows = data.split('\n');
processRows(rows);

It works fine on your laptop with a 50 MB test file. Then it hits prod.

RangeError [ERR_STRING_TOO_LONG]: Cannot create a string longer than 0x1fffffe8 characters

Not the out-of-memory crash you were braced for. readFileSync(path, 'utf8') has to materialise the whole file as a single JavaScript string, and V8 caps string length at 0x1fffffe8 — 536,870,888 characters, about 512 MB. Your 2 GB file doesn't get far enough to exhaust the heap; it's rejected at 512 MB and it will be rejected identically on a machine with a terabyte of RAM.

Either way the fix isn't "give Node more RAM" — it's streams.

Why Loading Everything At Once Is The Wrong Move

The core issue is treating data as a thing you have rather than a thing that flows through. When you call fs.readFile() on a large file, Node reads every byte into a Buffer before your code touches any of it. For small files this is fine. For anything large it's a footgun.

Streams flip the mental model: instead of reading then processing, you read while processing. Data comes in as chunks — 64 KB at a time from fs.createReadStream on Node 22 — you process each chunk, and memory stays flat.

Flat is the claim, so here it is measured. Same 334 MB CSV of 9 million rows, same machine, Node 22.22.3, peak RSS sampled every 20 ms:

readFileSync + split('\n')          peak RSS  804.5 MB
createReadStream + Transform + sink peak RSS   82.9 MB   (heapUsed 12.5 MB)

And the "regardless of file size" part, streaming two files through the same pipeline:

 95 MB file  → peak RSS 77.0 MB
334 MB file  → peak RSS 84.5 MB

A 3.5x bigger input cost 7 MB more memory. The eager version, by contrast, cost roughly 2.4x the file size — which is what makes it a time bomb rather than a slow path.

The Four Stream Types

Node's stream module gives you four primitives:

Most real work uses Readable → Transform → Writable pipelines. The data flows left to right, chunk by chunk.

The Pattern: pipeline()

The modern way to connect streams is stream/promises's pipeline(). It wires everything together, handles errors from any stage, and cleans up resources on failure.

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
 
async function compressFile(input: string, output: string): Promise<void> {
  await pipeline(
    createReadStream(input),
    createGzip(),
    createWriteStream(output)
  );
}
 
await compressFile('events.csv', 'events.csv.gz');

That's it. Three lines of actual logic. pipeline() does the rest:

• Automatically handles backpressure between stages
• If createGzip() throws, createReadStream() is cleaned up
• Returns a Promise that resolves when all data is flushed, rejects on any error

How Data Actually Flows

Here's what the runtime is doing:

Each 64 KB chunk moves through the chain independently. When the Writable's buffer fills up (because a downstream DB write is slow), backpressure automatically pauses the Readable. When the buffer drains, flow resumes. Memory stays bounded.

Writing a Transform Stream

The real power is in custom Transform streams. You can inject arbitrary processing logic between any two stages.

Here's a Transform that parses CSV line by line:

import { Transform, type TransformCallback } from 'node:stream';
 
class CSVParser extends Transform {
  private buffer = '';
  private headers: string[] = [];
  private isFirstLine = true;
 
  constructor() {
    super({ objectMode: true }); // Output objects, not Buffers
  }
 
  _transform(chunk: Buffer, _encoding: string, callback: TransformCallback): void {
    this.buffer += chunk.toString();
    const lines = this.buffer.split('\n');
 
    // Keep the last incomplete line in the buffer
    this.buffer = lines.pop() ?? '';
 
    for (const line of lines) {
      if (!line.trim()) continue;
 
      if (this.isFirstLine) {
        this.headers = line.split(',').map(h => h.trim());
        this.isFirstLine = false;
        continue;
      }
 
      const values = line.split(',');
      const row = Object.fromEntries(
        this.headers.map((header, i) => [header, values[i]?.trim() ?? ''])
      );
      this.push(row);
    }
 
    callback();
  }
 
  _flush(callback: TransformCallback): void {
    // Process any remaining buffered data
    if (this.buffer.trim()) {
      const values = this.buffer.split(',');
      const row = Object.fromEntries(
        this.headers.map((header, i) => [header, values[i]?.trim() ?? ''])
      );
      this.push(row);
    }
    callback();
  }
}

A few things worth noting here:

• objectMode: true lets you push objects instead of Buffers. Without this, this.push({ ... }) will throw.
• _flush() is your cleanup hook — called once when the upstream ends. Always handle the partial buffer at the end.
• callback() signals to Node that you're ready for the next chunk. Call it exactly once per _transform invocation.

Now wire it into a pipeline:

import { pipeline } from 'node:stream/promises';
import { createReadStream } from 'node:fs';
import { Writable } from 'node:stream';
 
// A writable that batches rows and inserts to DB
class DatabaseWriter extends Writable {
  private batch: Record<string, string>[] = [];
  private readonly BATCH_SIZE = 500;
 
  constructor() {
    super({ objectMode: true });
  }
 
  async _write(
    row: Record<string, string>,
    _encoding: string,
    callback: (err?: Error | null) => void
  ): Promise<void> {
    this.batch.push(row);
 
    if (this.batch.length >= this.BATCH_SIZE) {
      try {
        await insertBatch(this.batch);
        this.batch = [];
        callback();
      } catch (err) {
        callback(err instanceof Error ? err : new Error(String(err)));
      }
    } else {
      callback();
    }
  }
 
  async _final(callback: (err?: Error | null) => void): Promise<void> {
    // Flush remaining rows below batch size
    if (this.batch.length > 0) {
      try {
        await insertBatch(this.batch);
        callback();
      } catch (err) {
        callback(err instanceof Error ? err : new Error(String(err)));
      }
    } else {
      callback();
    }
  }
}
 
async function processEventLog(filePath: string): Promise<void> {
  await pipeline(
    createReadStream(filePath),
    new CSVParser(),
    new DatabaseWriter()
  );
}

This processes a 2 GB file in constant memory. The DatabaseWriter batches rows to avoid hitting the DB on every single row, and _final() flushes whatever's left at the end.

Backpressure: Why It Matters And What Goes Wrong

Backpressure is the mechanism that keeps a fast producer from overwhelming a slow consumer. Every Writable stream has a highWaterMark, and when the internal buffer exceeds that mark, write() returns false.

The default is the detail people carry around wrong, because it changed. On Node 22:

new Readable().readableHighWaterMark              65536   (64 KB)
new Writable().writableHighWaterMark              65536   (64 KB)
fs.createReadStream(f).readableHighWaterMark      65536   (64 KB)
new Writable({ objectMode: true })                   16   (16 objects)

16 KB was the binary default for years and is still what most write-ups quote; Node 22 raised it to 64 KB. Object mode is unchanged at 16 objects — note that's objects, so an object-mode buffer's memory footprint depends entirely on how big your objects are. Sixteen parsed CSV rows is nothing; sixteen decoded images is not.

If you're using pipeline(), this is handled automatically. But if you ever write manual stream code, you can accidentally blow past it:

// ❌ This ignores backpressure entirely
readable.on('data', (chunk) => {
  writable.write(chunk); // Ignores the return value
});
 
// ✅ Respect the backpressure signal
readable.on('data', (chunk) => {
  const canContinue = writable.write(chunk);
  if (!canContinue) {
    readable.pause();
    writable.once('drain', () => readable.resume());
  }
});

The difference in memory usage isn't small — it's the same 82.9 MB versus 804.5 MB gap measured earlier, except now you've written it by hand instead of inheriting it from readFileSync. Node's own "Backpressuring in Streams" guide runs the equivalent experiment on a much larger file and reports roughly 87.81 MB peak when you respect write()'s return value. Ignore it and the ceiling is however much data your producer can read before the consumer catches up, which on a big enough input is "all of it".

Streaming HTTP Responses

Streams aren't just for files. HTTP responses in Node are Readable streams, which means you can pipe them directly.

import { pipeline } from 'node:stream/promises';
import { createWriteStream } from 'node:fs';
import https from 'node:https';
 
function downloadFile(url: string, dest: string): Promise<void> {
  return new Promise((resolve, reject) => {
    https.get(url, async (response) => {
      try {
        await pipeline(response, createWriteStream(dest));
        resolve();
      } catch (err) {
        reject(err);
      }
    }).on('error', reject);
  });
}

No temp buffers, no waiting for the full response. The bytes flow from the network socket straight to disk.

TypeScript Types Worth Knowing

The stream module types can be annoying if you don't know which ones to reach for:

// Transform is a class you extend, so it's a value import, not `import type`.
// TransformCallback is a type — it has to be imported too, it isn't global.
import { Transform, type TransformCallback } from 'node:stream';
 
interface ParsedRow {
  userId: string;
  event: string;
  timestamp: string;
}
 
// There's no official generic typing for object-mode transforms,
// so just type the _transform signature directly:
class TypedTransform extends Transform {
  _transform(chunk: Buffer, _encoding: string, callback: TransformCallback): void {
    const row: ParsedRow = JSON.parse(chunk.toString());
    this.push(row);
    callback();
  }
}

Two mistakes are easy to make here and both are compile errors rather than runtime surprises, which is the good news. Writing import type { Transform } and then extends Transform gives you TS1361: 'Transform' cannot be used as a value because it was imported using 'import type'. And using TransformCallback without importing it gives TS2304: Cannot find name 'TransformCallback' — it looks ambient because it shows up in so many snippets, but it's a named export of node:stream.

If you want end-to-end generics rather than hand-typed signatures, that's the Web Streams API — import type { ReadableStream } from 'node:stream/web' gives you ReadableStream<ParsedRow>. For server-side file processing the classic stream module types are fine.

When NOT To Use Streams

Streams add complexity. Don't reach for them when:

The data fits comfortably in memory. If you're processing a 200 KB config file or a small JSON response, fs.readFile() is simpler and just as fast. The overhead of setting up a pipeline isn't worth it.

You need random access. Streams are inherently sequential. If you need to jump to byte offset 500,000 and read backward, you're fighting the model. Use fs.open() with explicit position reads instead.

The processing is CPU-bound, not I/O-bound. Streams help when the bottleneck is reading/writing data. If you're doing heavy computation on each chunk (image processing, crypto), streams won't save you — that's a job for worker threads.

You're in a request handler processing small payloads. Reading req.body with streams in an Express handler that receives 10 KB JSON objects adds latency and complexity with no benefit. Use express.json() middleware.

The Practical Takeaway

The pattern to internalize is this: anything with "large" in the description — large file, large response, large export — defaults to a stream pipeline. The setup cost is small (pipeline() takes three lines), and the memory behavior is predictable.

The typical structure is always:

createReadStream(source) → [zero or more Transforms] → createWriteStream(dest)

Get comfortable writing Transform classes. Once you've written one CSV parser and one batch database writer, the pattern clicks and you'll find yourself reaching for it naturally.