Streams in JavaScript let you process data as it arrives instead of loading everything into memory at once. Whether you're downloading a 500MB file, processing a CSV with millions of rows, or streaming AI token responses in real time, streams are your answer. Today we'll explore how Web Streams, fetch streaming, and Node streams work, and when to reach for them.
Why Streams Matter
Before streams became mainstream, you had two bad options: load the entire dataset into memory (RAM exhaustion) or write custom polling loops (complexity and bugs). Streams give you a third way—a standard, composable interface to handle data in chunks. This matters because real production systems deal with files larger than available RAM, network latency, and unpredictable data arrival rates.
I learned this the hard way. Early in my career, I built an e-commerce system that crashed whenever customers uploaded large product catalogs as CSV files. The code tried to parse the entire file at once. After switching to streams, the same endpoint handled 100MB+ uploads without breaking a sweat.
Web Streams — ReadableStream, WritableStream and TransformStream
The Web Streams API gives you a standardized way to handle streaming data in the browser and modern Node.js. Three core types form the foundation:
- ReadableStream — data flows out; you pull or it pushes to you
- WritableStream — data flows in; you write chunks to a destination
- TransformStream — combines both; transforms data as it passes through
Here's a practical example—a ReadableStream that generates numbers on demand:
const numberStream = new ReadableStream({
start(controller) {
console.log('Stream started');
},
pull(controller) {
// Called when consumer is ready for more data
const randomNumber = Math.floor(Math.random() * 100);
if (randomNumber > 95) {
controller.close(); // Stop stream
} else {
controller.enqueue(randomNumber); // Send one value
}
},
});
// Consume with a reader
const reader = numberStream.getReader();
async function consume() {
while (true) {
const { done, value } = await reader.read();
if (done) break;
console.log('Got number:', value);
}
}
consume();
The pull() method is called when the consumer is ready. You decide the pace—this prevents overwhelming downstream systems.
Now let's build a TransformStream that doubles every number:
const doubleTransform = new TransformStream({
transform(chunk, controller) {
controller.enqueue(chunk * 2);
},
});
// Pipe: numberStream → doubleTransform → outputStream
numberStream
.pipeThrough(doubleTransform)
.pipeTo(new WritableStream({
write(chunk) {
console.log('Final value:', chunk);
},
}));
This pattern—pipeThrough for transformation and pipeTo for destination—is the Web Streams way. No callbacks, no event listeners, just declarative composition.
fetch Streaming — Read Response Bodies Chunk by Chunk
In real applications, HTTP responses arrive gradually. The classic await response.json() waits for the entire body. With streaming, you process chunks as they arrive.
Imagine building a progress bar for a large file download:
async function downloadWithProgress(url: string) {
const response = await fetch(url);
const contentLength = parseInt(
response.headers.get('content-length') || '0',
10
);
let downloaded = 0;
const reader = response.body!.getReader();
const chunks: Uint8Array[] = [];
while (true) {
const { done, value } = await reader.read();
if (done) break;
chunks.push(value);
downloaded += value.length;
const percent = ((downloaded / contentLength) * 100).toFixed(2);
console.log(`Downloaded: ${percent}%`);
}
// Combine all chunks
const total = chunks.reduce((acc, c) => acc + c.length, 0);
const buffer = new Uint8Array(total);
let offset = 0;
for (const chunk of chunks) {
buffer.set(chunk, offset);
offset += chunk.length;
}
return buffer;
}
For text data like JSON Lines (one JSON object per line), streaming makes even more sense:
async function streamJsonLines(url: string) {
const response = await fetch(url);
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
// Process complete lines
for (let i = 0; i < lines.length - 1; i++) {
if (lines[i]) {
const obj = JSON.parse(lines[i]);
console.log('Parsed object:', obj);
}
}
// Keep incomplete line in buffer
buffer = lines[lines.length - 1];
}
// Handle final line
if (buffer) {
console.log('Final object:', JSON.parse(buffer));
}
}
This pattern is essential for server-sent events (SSE) and streaming APIs. The data arrives in unpredictable chunks, and you decode and parse on the fly.
Node Streams — Pipe Files and Network Data with Backpressure
Node.js streams predate the Web Streams API but follow similar principles. Node streams are event-based, which feels less modern but is deeply integrated into core APIs like file I/O and HTTP.
Here's a simple file copy using Node streams:
import { createReadStream, createWriteStream } from 'fs';
const source = createReadStream('large-file.txt');
const dest = createWriteStream('copy.txt');
source.pipe(dest);
dest.on('finish', () => {
console.log('File copied successfully');
});
dest.on('error', (err) => {
console.error('Write error:', err);
});
The pipe method automatically handles backpressure. If the destination is slow to write, Node pauses the source until the destination catches up. This prevents memory spikes.
For more control, read and write manually:
import { createReadStream, createWriteStream } from 'fs';
const source = createReadStream('large-file.txt', { highWaterMark: 64 * 1024 });
const dest = createWriteStream('copy.txt');
source.on('data', (chunk) => {
console.log(`Read chunk of ${chunk.length} bytes`);
// Manual backpressure handling
const canContinue = dest.write(chunk);
if (!canContinue) {
console.log('Destination buffer full, pausing read');
source.pause();
}
});
dest.on('drain', () => {
console.log('Destination drained, resuming read');
source.resume();
});
source.on('end', () => {
dest.end();
});
The highWaterMark option controls chunk size (default 16KB for text). Higher values trade memory for throughput; lower values reduce latency.
For CSV processing in production, you'd use a Transform stream:
import { Transform } from 'stream';
import { parse } from 'csv-parse';
import { stringify } from 'csv-stringify';
const input = createReadStream('input.csv');
const output = createWriteStream('output.csv');
const csvParser = parse({ columns: true });
const csvStringifier = stringify({ header: true });
const normalizeTransform = new Transform({
objectMode: true,
transform(record, encoding, callback) {
// Modify each row
record.email = record.email?.toLowerCase();
record.name = record.name?.trim();
callback(null, record);
},
});
input
.pipe(csvParser)
.pipe(normalizeTransform)
.pipe(csvStringifier)
.pipe(output);
output.on('finish', () => {
console.log('CSV processed and written');
});
Notice objectMode: true—Transform streams usually work with Buffers, but CSV rows are objects. This mode tells Node to treat each chunk as a discrete value.
Use Cases — Large Downloads, CSV Processing and AI Token Streaming
Streams shine in three real scenarios I've encountered repeatedly.
Large Downloads
A ride-hailing app needed to let drivers download offline maps (1-2GB per region). Downloading to memory would crash the app. Streams let us write directly to disk while showing progress:
async function downloadMap(mapUrl: string, filename: string) {
const response = await fetch(mapUrl);
const reader = response.body!.getReader();
const total = parseInt(response.headers.get('content-length') || '0', 10);
let downloaded = 0;
const chunks: Uint8Array[] = [];
while (true) {
const { done, value } = await reader.read();
if (done) break;
chunks.push(value);
downloaded += value.length;
updateProgressBar(downloaded, total);
}
// On native mobile, you'd write this buffer to app storage
const blob = new Blob(chunks, { type: 'application/octet-stream' });
saveToStorage(filename, blob);
}
CSV Processing
E-commerce platforms often import bulk product feeds (10,000+ items). Streaming prevents memory bloat and lets you validate or enrich rows on the fly. I built this for a trading system importing market data:
const validateTransform = new Transform({
objectMode: true,
transform(record, encoding, callback) {
if (!record.symbol || !record.price) {
callback(new Error(`Invalid record: ${JSON.stringify(record)}`));
return;
}
record.price = parseFloat(record.price);
callback(null, record);
},
});
input
.pipe(csvParser)
.pipe(validateTransform)
.pipe(csvStringifier)
.pipe(output)
.on('error', (err) => {
console.error('Stream error:', err.message);
});
AI Token Streaming
Modern AI APIs (OpenAI, Anthropic) stream tokens as they're generated. This gives users instant feedback instead of waiting for a full response. Here's how to consume an OpenAI stream:
async function streamCompletion(prompt: string) {
const response = await fetch('https://api.openai.com/v1/chat/completions', {
method: 'POST',
headers: {
'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'gpt-4',
messages: [{ role: 'user', content: prompt }],
stream: true,
}),
});
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
for (let i = 0; i < lines.length - 1; i++) {
const line = lines[i].trim();
if (line.startsWith('data: ')) {
const json = line.slice(6);
if (json === '[DONE]') break;
try {
const event = JSON.parse(json);
const token = event.choices?.[0]?.delta?.content;
if (token) {
process.stdout.write(token); // Real-time output
}
} catch (e) {
// Ignore parse errors
}
}
}
buffer = lines[lines.length - 1];
}
}
The user sees text appearing character by character instead of waiting 10 seconds for a complete response. This is transformative for UX.
Common Mistakes
- Ignoring backpressure — In Node streams, check the return value of
write(). If it returns false, pause the source. Ignoring this spikes memory and crashes production systems. - Buffering entire chunks in memory — Streams lose their advantage if you collect all data before processing. Process incrementally instead.
- Not handling encoding — When converting streams of Uint8Array to strings, use TextDecoder with the
stream: trueflag to handle multi-byte characters split across chunks. - Mixing APIs without care — Web Streams (fetch, browser) and Node streams (fs, net) are different. Don't assume
.pipe()works everywhere. UseReadable.fromWeb()andWritable.toWeb()for conversion when needed. - Silent error swallowing — Always attach error listeners. A stream might fail silently, leaving you wondering why processing stopped.
Key Takeaways
- Streams process data as it arrives, preventing memory exhaustion on large datasets.
- Web Streams (
ReadableStream,WritableStream,TransformStream) usepipeThroughandpipeTofor composition. - fetch streaming with
response.body.getReader()is ideal for large downloads and text-based APIs like SSE and OpenAI. - Node streams use
.pipe()and handle backpressure automatically; manual control requires listening to 'data' and 'drain' events. - Real-world wins: large file handling, CSV imports, and AI token streaming all demand streams for acceptable performance and UX.
- Always handle backpressure and errors explicitly; silent failures are the enemy of reliability.
Comments
Loading comments…