Cannot create a string longer than 0x3fffffe7 characters

Question

I'm using Request library (gzip enabled) and it crash with this error.

It seems this error occur at this line,

response.body = response.body.toString(self.encoding)

Scope containing that line,

response.body = Buffer.concat(buffers, bufferLength)
if (self.encoding !== null) {
 response.body = response.body.toString(self.encoding)
}

Full Stacktrace,

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.utf8Slice (<anonymous>)
    at Buffer.toString (buffer.js:797:17)
    at Request.<anonymous> (/home/proj/node_modules/request/request.js:1128:39)
    at Request.emit (events.js:315:20)
    at IncomingMessage.<anonymous> (/home/proj/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:421:28)
    at IncomingMessage.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_STRING_TOO_LONG'
}

What's the best way to convert the buffer to String when it's longer than 1073741799 characters?

Answer 1

It seems I ran into a crawling trap while crawling the websites. The page that request fetched is almost over 2GB in size.

NodeJS's Buffer.toString has a limit of 1 GB and this content is way over that.

The authors of request library has forgotten to put a catch block inside the toString operation.

The fix is to add the catch block like below in request.js file's line 1130,

  if (self.encoding !== null) {
    try {
      response.body = response.body.toString(self.encoding)
    } catch(e) {
      self.emit('error', e)
    }
  }