Node.js streams are a powerful feature for handling large data sets efficiently. They allow you to process data in chunks, reducing memory usage and improving performance. Let’s dive into some advanced techniques for using streams in Node.js.

First, let’s look at how to create a readable stream:

const fs = require('fs');

const readStream = fs.createReadStream('largefile.txt', { encoding: 'utf8' });

readStream.on('data', (chunk) => {
  console.log(chunk);
});

readStream.on('end', () => {
  console.log('Finished reading the file');
});

This code creates a readable stream from a large file and logs each chunk of data as it’s read. It’s a simple example, but it illustrates the basic concept of streaming data.

Now, let’s explore how we can use streams to transform data on-the-fly:

const { Transform } = require('stream');

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

readStream.pipe(upperCaseTransform).pipe(process.stdout);

In this example, we’ve created a transform stream that converts incoming data to uppercase. We then pipe our readable stream through this transform and finally to stdout. This demonstrates how we can chain streams together to create powerful data processing pipelines.

One of the great things about streams is that they’re not just for files. We can use them with network requests too. Here’s an example of using streams with an HTTP server:

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
  const fileStream = fs.createReadStream('largefile.txt');
  fileStream.pipe(res);
});

server.listen(3000, () => console.log('Server running on port 3000'));

This server will stream the contents of ‘largefile.txt’ to any client that connects. Because we’re using streams, we can handle multiple concurrent connections without loading the entire file into memory for each one.

Streams aren’t just for reading data, though. We can also use them for writing. Here’s an example of a writable stream:

const fs = require('fs');

const writeStream = fs.createWriteStream('output.txt');

for (let i = 0; i < 1000000; i++) {
  writeStream.write(`This is line ${i}\n`);
}

writeStream.end('This is the end\n');

writeStream.on('finish', () => {
  console.log('Finished writing to file');
});

This code writes a million lines to a file using a writable stream. Because we’re using a stream, we don’t need to hold all of this data in memory at once.

Now, let’s look at a more complex example that combines several stream concepts. We’ll create a program that reads a large CSV file, transforms the data, and writes it to a new file:

const fs = require('js');
const csv = require('csv-parse');
const { Transform } = require('stream');

const readStream = fs.createReadStream('large_data.csv');
const writeStream = fs.createWriteStream('transformed_data.csv');

const parser = csv();

const transformer = new Transform({
  objectMode: true,
  transform(record, encoding, callback) {
    // Assume the CSV has columns: id, name, age
    const transformedRecord = {
      id: record[0],
      name: record[1].toUpperCase(),
      age: parseInt(record[2]) + 1
    };
    this.push(Object.values(transformedRecord).join(',') + '\n');
    callback();
  }
});

readStream
  .pipe(parser)
  .pipe(transformer)
  .pipe(writeStream)
  .on('finish', () => console.log('Transformation complete'));

This example reads a CSV file, parses it, transforms each record (converting names to uppercase and incrementing ages), and writes the result to a new CSV file. All of this is done using streams, so we can process files much larger than our available memory.

Streams can also be used with compression. Here’s an example of compressing a file on-the-fly:

const fs = require('fs');
const zlib = require('zlib');

const readStream = fs.createReadStream('largefile.txt');
const writeStream = fs.createWriteStream('largefile.txt.gz');
const gzip = zlib.createGzip();

readStream.pipe(gzip).pipe(writeStream);

This code reads a file, compresses it using gzip, and writes the compressed data to a new file. Again, all of this is done in a streaming fashion, so we can handle files of any size.

One of the most powerful aspects of streams is their ability to be paused and resumed. This can be incredibly useful when dealing with backpressure - situations where data is coming in faster than it can be processed. Here’s an example:

const fs = require('fs');

const readStream = fs.createReadStream('verylargefile.txt');
const writeStream = fs.createWriteStream('output.txt');

readStream.on('data', (chunk) => {
  if(!writeStream.write(chunk)) {
    readStream.pause();
  }
});

writeStream.on('drain', () => {
  readStream.resume();
});

readStream.on('end', () => {
  writeStream.end();
});

In this example, we pause the read stream if the write stream can’t keep up, and resume it when the write stream is ready for more data. This ensures that we don’t overwhelm our system’s memory by reading data faster than we can write it.

Streams can also be used with external processes. Here’s an example that uses a stream to capture the output of a system command:

const { spawn } = require('child_process');
const fs = require('fs');

const ls = spawn('ls', ['-lh', '/usr']);
const writeStream = fs.createWriteStream('ls_output.txt');

ls.stdout.pipe(writeStream);

ls.stderr.on('data', (data) => {
  console.error(`stderr: ${data}`);
});

ls.on('close', (code) => {
  console.log(`child process exited with code ${code}`);
});

This code runs the ‘ls -lh /usr’ command and streams its output to a file. It’s a great way to handle the output of long-running processes or commands that produce a lot of output.

Streams aren’t just for files and processes, though. They can also be used with network sockets. Here’s an example of a TCP server that uses streams:

const net = require('net');
const fs = require('fs');

const server = net.createServer((socket) => {
  console.log('Client connected');
  
  const writeStream = fs.createWriteStream('client_data.txt');
  
  socket.pipe(writeStream);
  
  socket.on('end', () => {
    console.log('Client disconnected');
    writeStream.end();
  });
});

server.listen(8080, () => console.log('Server listening on port 8080'));

This server listens for TCP connections and streams any data received from clients to a file. It’s a simple example, but it shows how streams can be used in network programming.

One of the great things about Node.js streams is that they’re composable. You can chain them together to create complex data processing pipelines. Here’s an example that reads a file, compresses it, encrypts it, and sends it over a network socket:

const fs = require('fs');
const zlib = require('zlib');
const crypto = require('crypto');
const net = require('net');

const readStream = fs.createReadStream('largefile.txt');
const gzip = zlib.createGzip();
const encrypt = crypto.createCipher('aes-256-cbc', 'secret-key');

const socket = net.connect(8080, 'example.com');

readStream
  .pipe(gzip)
  .pipe(encrypt)
  .pipe(socket);

This code demonstrates how powerful streams can be when combined. We’re reading a file, compressing it, encrypting it, and sending it over the network, all without ever holding the entire file in memory at once.

Streams aren’t just for Node.js core modules either. Many npm packages support streaming interfaces. For example, the ‘request’ package (a popular HTTP client) supports streams:

const request = require('request');
const fs = require('fs');

request('https://example.com/largefile.zip')
  .pipe(fs.createWriteStream('largefile.zip'));

This code downloads a large file from the internet and writes it directly to disk, all using streams. It’s an efficient way to handle large downloads without consuming excessive memory.

In conclusion, streams are a powerful feature of Node.js that allow you to handle large amounts of data efficiently. They’re versatile, composable, and can significantly improve the performance of your applications. Whether you’re working with files, network connections, or complex data processing pipelines, streams are a tool you’ll want in your Node.js toolkit. Happy streaming!