Node.js Streams and Buffers Explained

A user uploads a 2 GB video. Your Node.js server reads the whole file into memory, then writes it to disk. With one user it works. With ten at the same time, the process runs out of memory and restarts.

Streams and Buffers exist to prevent this. This guide explains what each one is, how they fit together, and how to use them without the common mistakes.

What Is a Buffer?

A Buffer is a fixed-size block of memory that holds raw binary data. Files, network packets, images and compressed data are all bytes, and a Buffer is how Node.js holds those bytes before they become text or objects.

const buf = Buffer.from('hello', 'utf8')

console.log(buf)            // <Buffer 68 65 6c 6c 6f>
console.log(buf.length)     // 5
console.log(buf.toString()) // 'hello'

Two things to remember:

• A Buffer has a fixed length. You cannot grow it. You allocate a new one and copy.
• A Buffer is not a string. Turning bytes into text needs an encoding such as utf8, and getting the encoding wrong produces garbled output.

You rarely create Buffers by hand. You meet them as the chunks that streams hand you.

What Is a Stream?

A stream is a way to process data in small pieces as it arrives, instead of loading all of it first. Each piece is called a chunk, and by default a chunk is a Buffer.

Think of water in a pipe. You do not wait for a lake to fill before you start using it. You use the water as it flows.

This is why a stream can handle a 2 GB file with a few tens of kilobytes of memory. Only the current chunk is held at any moment.

Reading a Large File: Buffer vs Stream

Here is the version that holds the whole file in memory:

import { readFile, writeFile } from 'node:fs/promises'

const data = await readFile('input.mp4') // 2 GB in memory
await writeFile('copy.mp4', data)

And here is the same copy as a stream:

import { createReadStream, createWriteStream } from 'node:fs'
import { pipeline } from 'node:stream/promises'

await pipeline(
  createReadStream('input.mp4'),
  createWriteStream('copy.mp4'),
)

The first version needs memory equal to the file size. The second needs memory equal to one chunk, which is 64 KB by default for file streams. The second also starts writing as soon as the first chunk is read, so the first byte lands on disk sooner.

For small files, readFile is fine and easier to read. Reach for streams when the data is large, when its size is unknown, or when it arrives over time.

The Four Stream Types

A Transform is a Duplex whose output depends on its input. Most useful stream work is a Readable, a chain of Transforms, then a Writable.

Backpressure: The Part That Causes Bugs

A fast reader can feed a slow writer. If you read from a disk at 500 MB/s and write to a slow network at 10 MB/s, the unwritten data piles up in memory. This mismatch is the reason streams have backpressure.

A Writable stream has an internal buffer with a limit called highWaterMark. When write() returns false, the buffer is full and the producer should stop until the stream emits drain.

function writeAll(writable, chunks) {
  let i = 0

  function next() {
    while (i < chunks.length) {
      const ok = writable.write(chunks[i++])
      if (!ok) {
        writable.once('drain', next) // resume when there is room
        return
      }
    }
    writable.end()
  }

  next()
}

Ignoring the return value of write() is the most common stream bug. The code works in testing, then uses gigabytes of memory under real load.

You almost never need to write this by hand. pipe and pipeline handle it for you.

Use pipeline, Not pipe

readable.pipe(writable) handles backpressure, but it does not handle errors. If the source fails, the destination is left open and the file descriptor leaks.

pipeline handles both. It forwards errors, destroys every stream in the chain when one fails, and with node:stream/promises it returns a promise you can await.

import { createReadStream, createWriteStream } from 'node:fs'
import { pipeline } from 'node:stream/promises'
import { createGzip } from 'node:zlib'

try {
  await pipeline(
    createReadStream('logs.txt'),
    createGzip(),
    createWriteStream('logs.txt.gz'),
  )
  console.log('Compressed')
} catch (error) {
  console.error('Pipeline failed', error)
}

If logs.txt does not exist, you get one rejected promise and nothing is left open.

Writing a Transform

A Transform is a good fit when you need to change data in flight. This one uppercases text:

import { Transform } from 'node:stream'
import { pipeline } from 'node:stream/promises'

const upperCase = new Transform({
  transform(chunk, _encoding, callback) {
    callback(null, chunk.toString().toUpperCase())
  },
})

await pipeline(process.stdin, upperCase, process.stdout)

One caveat: a chunk boundary can fall in the middle of a multi-byte character. For text that matters, such as splitting by line, use readline or set an encoding on the stream, which uses a decoder that handles split characters correctly. Calling chunk.toString() on raw chunks is fine for a demo and risky for real input.

Async Iteration

Readable streams are async iterables, so you can consume them with for await:

import { createReadStream } from 'node:fs'
import { createInterface } from 'node:readline'

const lines = createInterface({
  input: createReadStream('big.csv'),
  crlfDelay: Infinity,
})

let count = 0
for await (const line of lines) {
  if (line.startsWith('ERROR')) count++
}

console.log(`${count} error lines`)

This reads a file of any size one line at a time. Memory stays flat, and the loop applies backpressure on its own because the next chunk is not requested until the loop body finishes.

Streams in an HTTP Server

The request is a Readable and the response is a Writable, so serving a file looks like this:

import { createServer } from 'node:http'
import { createReadStream } from 'node:fs'
import { pipeline } from 'node:stream'

createServer((req, res) => {
  res.setHeader('Content-Type', 'video/mp4')

  pipeline(createReadStream('video.mp4'), res, (error) => {
    if (error) res.destroy(error)
  })
}).listen(3000)

The client starts receiving data immediately, and a client that disconnects part way stops the file read instead of leaving it running.

Common Mistakes

• Using readFile on unbounded input. Uploads, exports and logs have no size you control.
• Ignoring the return value of write(). It is the backpressure signal.
• Using pipe without error handling. Prefer pipeline.
• Concatenating chunks into one string or Buffer. Doing this brings back the memory problem you used streams to avoid.
• Assuming a chunk is a whole record. A chunk is an arbitrary slice of bytes. A line, a JSON object or a character can be split across two chunks.
• Forgetting that a Buffer is raw bytes. Convert with the right encoding, and only when you need text.

When to Use What

Summary

A Buffer holds bytes. A stream moves those bytes in chunks so memory stays flat no matter how much data passes through. Backpressure keeps a fast producer from drowning a slow consumer, and pipeline handles both backpressure and cleanup for you.

If a Node.js service handles uploads, exports, media or large files and its memory use climbs with load, streams are usually the fix.