fs.createRead/WriteStream() versus fs.promises.open()

Question

Which of these is more suitable for handling file read/write operations in file manager type of node server application ?

Is one faster then the other? Speed is very important because the app is suppose to be able to handle many user requests at the same time

Answer 1

What makes streams unique, is that instead of a program reading a file into memory all at once like in the traditional way, streams read chunks of data piece by piece, processing its content without keeping it all in memory.

This makes streams really powerful when working with large amounts of data, for example, a file size can be larger than your free memory space, making it impossible to read the whole file into the memory in order to process it. That’s where streams come to the rescue!

Using streams to process smaller chunks of data, makes it possible to read larger files.

Streams basically provide two major advantages compared to other data handling methods:

1. Memory efficiency: you don’t need to load large amounts of data in memory before you are able to process it
2. Time efficiency: it takes significantly less time to start processing data as soon as you have it, rather than having to wait with processing until the entire payload has been transmitted

Answer 2

Which of these is more suitable for handling file read/write operations in file manager type of node server application?

Both are usable for a node server applications. However, both request and response in HTTP implementation in node are stream-based, which means the stream-based approach is more flexible in terms of dealing with large I/O operations.

Is one faster then the other? Speed is very important because the app is supposed to be able to handle many user requests at the same time.

There's strong evidence that stream is better at memory usage and time. I'll borrow some examples from Node.js Design Patterns - Second Edition: Master best practices to build modular and scalable server-side web applications Chapter 5 - Coding with Streams

Buffer-approach:

const fs = require('fs');
const zlib = require('zlib');

const file = process.argv[2];

fs.readFile(file, (err, buffer) => {
  zlib.gzip(buffer, (err, buffer) => {
    fs.writeFile(file + '.gz', buffer, err => {
      console.log('File successfully compressed');
    });
  });
});

The result will be normal, but when trying to a file with over 1GB of size. We'll experience this error.

RangeError: File size is greater than possible Buffer: 0x3FFFFFFF bytes

With the same file for stream-approach:

const fs = require('fs');
const zlib = require('zlib');
const file = process.argv[2];

fs.createReadStream(file)
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream(file + '.gz'))
  .on('finish', () => console.log('File successfully compressed'));

Imagining if a node process has to handle the concurrent of 100 requests, each request trying to upload the file size up to 100MB. This node.js process simply will hold all the buffer of file binary in memory and your server simply suffer memory leak.

For time efficiency, stream helps the data processed by chunk, so it definitely increases the speed

Disclaimer: most of the examples and image are from Node.js Design Patterns - Second Edition - Chapter 5, I don't own any of this material, and this is for education purpose only.