Slow Buffer.concat

Question

When I read a 16MB file in pieces of 64Kb, and do Buffer.concat on each piece, the latter proves to be incredibly slow, takes a whole 4s to go through the lot.

Is there a better way to concatenate a buffer in Node.js?

Node.js version used: 7.10.0, under Windows 10 (both are 64-bit).

---

This question is asked while researching the following issue: https://github.com/brianc/node-postgres/issues/1286, which affects a large audience.

The PostgreSQL driver reads large bytea columns in chunks of 64Kb, and then concatenates them. We found out that calling Buffer.concat is the culprit behind a huge loss of performance in such examples.

Answer 1

Rather than concatenating every time (which creates a new buffer each time), just keep an array of all of your buffers and concat at the end.

Buffer.concat() can take a whole list of buffers. Then it's done in one operation. https://nodejs.org/api/buffer.html#buffer_class_method_buffer_concat_list_totallength

Answer 2

If you read from a file and know the size of that file, then you can pre-allocate the final buffer. Then each time you get a chunk of data, you can simply write it to that large 16Mb buffer.

// use the "unsafe" version to avoid clearing 16Mb for nothing
let buf = Buffer.allocUnsafe(file_size)

let pos = 0
file.on('data', (chunk) => {
    buf.fill(chunk, pos, pos + chunk.length)
    pos += chunk.length
})

if(pos != file_size) throw new Error('Ooops! something went wrong.')

The main difference with @Brad's code sample is that you're going to use 16Mb + size of one chunk (roughly) instead of 32Mb + size of one chunk.

Also, each chunk has a header, various pointers, etc. so you are not unlikely to use 33Mb or even 34Mb... that's a lot more RAM. The amount of RAM copied is otherwise the same. That being said, it could be that Node starts reading the next chunk while you copy so it could make it transparent. When done in one large chunk in the 'end' event, you're going to have to wait for the contact() to complete while doing nothing else in parallel.

---

In case you are receiving an HTTP POST and are reading it. Remember that you get a Content-Length parameter so you also have the length in that case and can pre-allocate the entire buffer before reading the data.