array join string Invalid string length (5 million data)

Question

My code:

const fs = require('fs');
const os = require('os');
const path = require('path');
const desktopDir = path.join(os.homedir(), "Desktop");

var F_HASH = "........";
var F_AMOUNT = 5000000; //number of data

var game_res = [];

for (var i = 1; i < F_AMOUNT+1; i++) {        
    /*.................code
    .............................    
    ..................................*/
    game_res.push(/*..............code...............*/);  //(/r/n)       
}     
     
var h_row = i-1;
h_row = h_row.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
    
game_res = game_res.join(''); 
    
fs.writeFileSync(desktopDir + "/" + "Latest " + h_row + " hash.txt", game_res);

Text file output looks like this:

xxxxxxxxxxhashxxxxxxxxxx 0.00
xxxxxxxxxxhashxxxxxxxxxx 0.00
xxxxxxxxxxhashxxxxxxxxxx 0.00
xxxxxxxxxxhashxxxxxxxxxx 0.00

i set F_AMOUNT to 1,000,000 it works fine, no error

but when i set F_AMOUNT to 5,000,000 error show up like this:

****DIR*****\****file****.js:34
game_res = game_res.join('');
                    ^

RangeError: Invalid string length
    at Array.join (<anonymous>)
    at Object.<anonymous> (****DIR*****\****file****.js:34:21)
←[90m    at Module._compile (internal/modules/cjs/loader.js:1068:30)←[39m
←[90m    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)←[39m
←[90m    at Module.load (internal/modules/cjs/loader.js:933:32)←[39m
←[90m    at Function.Module._load (internal/modules/cjs/loader.js:774:14)←[39m
←[90m    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)←[39m
←[90m    at internal/main/run_main_module.js:17:47←[39m

i set "node_options max-old-space-size" to 10240
but nothing happen, still give me error

real data:

Answer 1

Node.js has a maximum (single) string length of about 1GB on 64-bit system, or about half that on 32-bit systems. When you run into this limit, the error shown is exactly the one that you're getting.

To reproduce the issue, here's a small script:

const str = '*'.repeat(108);
const arr = [];
for (let i = 0; i < 5000000; i++) {
  arr.push(str);
}
arr.join('');

If you replace 108 with 107, it doesn't crash. It's not entirely clear why you're running into the same issue with only 75 characters per item, but I'm still pretty sure that this is your problem.

To fix it, either write the contents of game_res to the file in batches, or open the output file before the loop and write each item to the file directly from the loop.