Piping Between Processes

Streaming Data Between Processes

A child's stdin, stdout and stderr are ordinary Node streams, so Buffers and Streams applies in full: pipeline for error propagation, backpressure for free, constant memory however large the data.

Feeding two million rows through a child processJavaScript

import { spawn } from 'node:child_process'; import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises'; const ROWS = 2_000_000;
function* rows() {
  for (let i = 0; i < ROWS; i++) yield `${i},${i % 7 ? 'emea' : 'apac'},${i * 3}\n`;
}
const child = spawn(process.execPath, ['filter.js'], { stdio: ['pipe', 'pipe', 'inherit'] });
const t0 = performance.now(); let out = ''; child.stdout.on('data', (c) => (out += c));
await pipeline(Readable.from(rows(), { encoding: 'utf8' }), child.stdin);
await new Promise((r) => child.on('close', r));
const secs = ((performance.now() - t0) / 1000).toFixed(2), rss = process.memoryUsage().rss;
console.log(`child pid ${child.pid} reported: ${out.trim()}`);
console.log(`${ROWS} rows in ${secs} s, parent rss ${(rss / 1048576).toFixed(0)} MB`);

Output

child pid 79292 reported: 285715 apac rows out of 2000000
2000000 rows in 1.57 s, parent rss 69 MB

filter.js is five lines: it iterates readline.createInterface({ input: process.stdin }) and writes one summary line. About 34 MB of text crossed the boundary in 1.57 s while the parent held 69 MB of RSS, because neither side ever buffered more than a pipe's worth.

Setting stderr to 'inherit' saves hours: the child's errors reach your terminal instead of a stream nobody reads, which is how a crashed child becomes a silent hang.

MENU