Manipulating elements of an array in parallel using node.js?

Question

I have a an array of json objects like this -

var resultsArr = [
{
    "upvotes": "45",
    "postID": "4555",
    "numberOfComments": "45",
    "shares":"22"
},
{
    "upvotes": "21",
    "postID": "4665",
    "numberOfComments": "20",
    "shares":"24"
},
{
    "upvotes": "4",
    "postID": "77366",
    "numberOfComments": "0",
    "shares":"4"
},
{
    "upvotes": "49",
    "postID": "6565",
    "numberOfComments": "22",
    "shares":"54",

}];

I need to compute a value score based on upvotes,numberOfComments,sharesand then push it back into the JSON dictionary so that every object in array looks like this -

var resultsArr= [{
....
},
{
    "upvotes": "49",
    "postID": "6565",
    "numberOfComments": "22",
    "shares":"54",
    "score":"20"
}]

I can access the json objects in this array using a for loop but from my understanding goes, it accesses each element in sequence.

Given that I will have roughly 100-200 items in the array, how can I speed up the score computation process to access each element parallelly and thus reduce the time it takes to compute the score for every element in the array?

P.S I'm coding this, with the assumption, that the elements in the array might grow to have 300-400 elements in the future.

Answer 1

The original answer under the ---- below was written in 2015 and was true at the time. Since then, Node.js has gained worker threads. But they can only literally share memory with SharedArrayBuffer, and you can't store objects in a SharedArrayBuffer (except in some serialized form), SharedArrayBuffer can only be used with typed arrays, which use numeric elements.

You can transfer an array from one thread to another (the sending thread loses access to it, the receiving thread gains access), so you could spin up N threads and transfer a portion of the array to each of them. They would work on it in parallel and the post results back to the main thread.

Just a sketch:

const { Worker } = require("worker_threads"); // If still using Node.js's CJS modules

function processChunkInWorker(script, chunk) {
    return new Promise((resolve, reject) => {
        const w = new Worker(script, {
            workerData: chunk
        });
        w.on("message", result => {
           resolve(result);
        });
    });
}
async function setScores(data, workerCount = 4) {
    const chunkSize = Math.round(data.length / workerCount);
    await Promise.all(
        Array.from({length: workerCount}, async (_, index) => {
            let chunkStart = index * chunkSize;
            const chunkEnd = index < workerCount - 1 ? (chunkStart + chunkSize) : data.length;
            const scores = await processChunkInWorker("./calcscore.js", data.slice(chunkStart, chunkEnd));
            for (const score of scores) {
                data[chunkStart++].score = score;
            }
        })
    );
}

(async () => {
    try {
        const data = /*...load the data...*/;
        await setScores(data);
        console.log(data);
    } catch (e) {
        console.error(e.message, e.stack);
    }
})();

where calcscore.js is something like:

const { Worker, isMainThread, parentPort, workerData } = require("worker_threads"); // If still using Node.js's CJS modules
if (!isMainThread) {
    const scores = new Float64Array(workerData.map(({upvotes, numberOfComments, shares}) => +upvotes + +numberOfComments + +shares));
    parentPort.postMessage(scores, scores.buffer);
}

---

Original answer:

how can I speed up the score computation process to access each element parallelly and thus reduce the time it takes to compute the score for every element in the array?

You can't (reasonably). NodeJS only runs a single thread. To get more than one thread of execution, you'd have to spawn a child process, which would be a lot slower than just doing it on the one thread.

300-400 elements is nothing (even 3-4 million would be nothing; 3M took about 289ms on my machine, 4M took 384ms). No need to make things complicated. Just:

resultsArr.forEach(function(entry) {
    // update `entry` here
});