How to handle error raise - Asyncio in Python3

Question

I've got a function f() that makes an API call and I want to call it multiple times asynchronously. I use the asyncio lib like that:

async def main():
    loop = asyncio.get_event_loop()
    futures = [loop.run_in_executor(None, f) for i in range(10)]
    await asyncio.gather(*futures)
    return futures

result = asyncio.get_event_loop().run_until_complete(main())

The problem is that sometimes f() raises an Exception and I'm not sure how to handle it. The doc says that Futures can contain an Exception, but that's not the case here, error is raised and program crashes.

How do I achieve that ? I think I could write a wrapper for f() and try: catch: the Exception, but that seems ugly if the feature is provided by the lib.

Thanks in advance for help,

Answer 1

The problem is that sometimes f() raises an Exception and I'm not sure how to handle it.

That will depend on what you want to do when an exception occurs. Remember that asyncio.gather() is convenience API that propagates exceptions by default to avoid blindly continuing in case of error. If you want to proceed in case of exception, you have other options:

• Pass return_exceptions=True to gather - this will cause gather to return the exception objects along with other results. Convenient and easy to use, but mixes exceptions with regular results, which is a bit messy.

• Use asyncio.wait() instead of asyncio.gather(). It returns sets of futures, each of which you can test whether it completed by raising or by producing a result.

• Wrap f() in your own function that catches the exception as you see fit. You considered and rejected this, but in some cases it's exactly the right approach.