Mongoose eachAsync with async function = memory leak?

Question

I am not sure if it's a bug in mongoose or if I am doing something wrong. Once I start using async functions when iterating on a cursor with eachAsync I experience memory leaks (quickly goes up to 4gb and then crashes). After trying some things I noticed that this wouldn't happen if I don't use an async function as callback.

No Memory leak:

const playerCursor: QueryCursor<IPlayerProfileModel> = PlayerProfile.find({}, projection).lean().cursor();
    await playerCursor.eachAsync(
      (profile: IPlayerProfileModel) => {
        return;
      },
      { parallel: 50 }
    );

Memory leak:

const playerCursor: QueryCursor<IPlayerProfileModel> = PlayerProfile.find({}, projection).lean().cursor();
    await playerCursor.eachAsync(
      async (profile: IPlayerProfileModel) => {
        return;
      },
      { parallel: 50 }
    );

Obviously above code doesn't make any sense but I need to perform an asynchronous operation within the function.

Question:

What is causing the memory leak / how can I avoid it?

Answer 1

It has to do with how async functions work.

Quoting the documentation:

When the async function returns a value, the Promise will be resolved with the returned value.

Meaning, values returned by async functions will be automatically wrapped into a Promise.

In your first code sample, your code returns undefined whereas in the second code sample your code returns Promise.resolve(undefined).

What is causing the memory leak?

I didn't take a look at mongoose code but the documentation states:

If fn returns a promise, will wait for the promise to resolve before iterating on to the next one.

Since your first example does not return a Promise, I am thinking your callback is executed on each result all at once rather than sequentially.

How can I avoid it?

I'd recommend using async/wait as you used it on your second code sample.

After taking a look at the code (looking for an answer myself), if provided with a callback that doesn't return a promise eachAsync will run the callback as many times as fast as possible as it can.

This line is where your callback is executed. The next line checks whether it's a Promise and if it's not then execute right away callback which effectively calls your eachAsync callback on the next result. If your callback has any sort of async operation but returns right away then you end up with thousands and thousands of async operations running all at once.

On top of that, you set the option parallel to 100 so it executes eachAsync callback one hundred times in parallel.

This isn't a bug on mongoose because there are cases where this behavior is wanted and it does provide with a sequential processing using Promise. The documentation should mention the caveat of using a callback which doesn't return a Promise.

To go a little further, express uses next on middleware callbacks for the purpose of sequencing them.