How to properly use Javascript Async eachLimit

Question

I'm having problems with using async.eachLimit. It works properly for the first 10 elements, but it doesn't continue past that; it simply ends. So, if there are 100 elements, it only does the first 10. This is clearly an issue of me misunderstanding callbacks. What is the proper way of using eachLimit with an external function that does not contain a callback? Or is it required for such a function to have one?

async.eachLimit(items, 10, function(item, callback) {
        outsideFunction(item.attrOne, item.attrTwo};
        //callback();  ---> leads to all running in parallel.
    },
    function(err) {
        console.log(err);
    }
 );

Answer 1

Your issue here is you're using an async library for a function that isn't asynchronous (or isn't acting like it's asynchronous). What async.eachLimit does is go over each item in the array, only executing limit amount at a time, and waits for callback() to be called saying the current iteration is finished and can add another one to be executed.

In your code example, the callback (when uncommented) gets called immediately after it tries calling outsideFunction because the function call is non-blocking. It doesn't wait because async says "I've been told it's done, I'll go onto the next one" so all 100 will try and be executed at the same time. If outsideFunction is an asynchronous function, it needs a callback (or have it use promises) to say it has finished executing, and inside that callback you call the callback for async.eachLimit and then it will only do 10 at a time in the way you want. Here's an example:

async.eachLimit(items, 10, function(item, callback)
{
    outsideFunction(item.attrOne, item.attrTwo, function(someResult)
    {
        // Your outside function calls back saying it's finished ...
        callback(); // ... so we tell async we're done
    });
},
function(err)
{
    console.log(err);
});

If outsideFunction isn't your function, and the function is actually asynchronous, then it's either using promises or you need to find a library that writes asynchronous functions properly. If the function isn't asynchronous, then async.eachLimit won't work.

If it's your function, you should make it send back a callback to say it's done (or use promises).