Node async loop - how to make this code run in sequential order?

Question

I know there are several posts about this, but according to those I've found, this should work correctly.

I want to make an http request in a loop and I do not want the loop to iterate until the request callback has been fired. I'm using the async library like so:

const async = require("async");
const request = require("request");

let data = [
    "Larry",
    "Curly",
    "Moe"
];

async.forEachOf(data, (result, idx, callback) => {
    console.log("Loop iterated", idx);
    let fullUri = "https://jsonplaceholder.typicode.com/posts";
    request({
        url: fullUri
    }, 
    (err, res, body) => {
        console.log("Request callback fired...");
        if (err || res.statusCode !== 200) return callback(err);
        console.log(result);
        callback();
    });
});

What I see is:

Loop iterated 0
Loop iterated 1
Loop iterated 2
Request callback fired...
Curly
Request callback fired...
Larry
Request callback fired...
Moe

What I need to see is:

Loop iterated 0
Request callback fired...
Curly
Loop iterated 1
Request callback fired...
Larry
Loop iterated 2
Request callback fired...
Moe

Also, if there's a built-in way to do the same thing (async/await? Promise?) and the async library could be removed, that'd be even better.

I've seen some examples of recursion out there that are clever, but when I put this to use in a much more complex situation (e.g. multiple request calls per-loop, etc.) I feel like that approach is hard to follow, and isn't as readable.

Answer 1

You can ditch async altogether and go for async/await quite easily.

Promisify your request and use async/await

Just turn request into a Promise so you can await on it.

Better yet just use request-promise-native that already wraps request using native Promises.

Serial example

From then on it's a slam dunk with async/await:

const rp = require('request-promise-native')

const users = [1, 2, 3, 4]
const results = []

for (const idUser of users) {
  const result = await rp('http://foo.com/users/' + idUser)

  results.push(result)
}

Parallel example

Now, the problem with the above solution is that it's slow - the requests run serially. That's not ideal most of the time.

If you don't need the result of the previous request for the next request, just go ahead and do a Promise.all to fire parallel requests.

const users = [1, 2, 3, 4]

const pendingPromises = []
for (const idUser of users) {
  // Here we won't `await` on *each and every* request.
  // We'll just prepare it and push it into an Array
  pendingPromises.push(rp('http://foo.com/users/' + idUser))
}

// Then we `await` on a a `Promise.all` of those requests
// which will fire all the prepared promises *simultaneously*, 
// and resolve when all have been completed
const results = await Promise.all(pendingPromises)

Error handling

Error handling in async/await is provided by plain-old try..catch blocks, which I've omitted for brevity.