Python multiprocessing continuous processing with await

Question

I am using an event based system using the new Python 3.5 coroutines and await. I register events and these events are called by the system.

@event
aysnc def handleevent(args):
    # handle the event

I need to initialize some classes to handle the work(time consuming). Then call instance methods, also time consuming (they actually use selenium to browse certain sites).

Ideally I would want something like the following code

# supposedly since this is multiprocessing this is a different driver per process
driver = None
def init():
    # do the heavy initialization here
    global driver
    driver = webdriver.Chrome()

def longworkmethod():
    ## need to return some data
    return driver.dolongwork()

class Drivers:
""" A class to handle async and multiprocessing"""
    def __init__(self, numberOfDrivers):
        self.pool = multiprocessing.Pool(processes=numberOfDrivers, initializer=init)       

    async def dowork(self, args):
        return self.pool.apply_async(longworkmethod, args=args)


### my main python class
drivers = Drivers(5)

@event
aysnc def handleevent(args):
    await drivers.dowork(args)

@event
aysnc def quit(args):
    ## do cleanup on drivers
    sys.exit(0)

This code doesn't work, but I have tried many different ways and none seem to be able to do what I want.

It doesn't have to be this exact form, but how do I go about mixing the await and coroutines with a program that needs multiprocessing?

Answer 1

While there nothing technically speaking that would limit you from mixing asyncio and multiprocessing, I would suggest avoiding doing so. It's going to add a lot of complexity as you'll end up needing an event loop per thread and passing information back and forth will be tricky. Just use one or the other.

asyncio supplies functions for running tasks in another thread - such as AbstractEventLoop.run_in_executor. Take a look at these answers

• https://stackoverflow.com/a/33025287/66349 (calling selenium within a coroutine)
• https://stackoverflow.com/a/28492261/66349

Alternatively you could just use multiprocessing as selenium has a blocking (non asyncio) interface, however it sounds like some of your code is using already using asyncio so maybe stick with the above.