Async alternative to pycurl in python 3.5+

Question

I have a discord bot that I suspect is having periodic issues due to occasional slow pycurl calls. After some research I found out pycurl is not asynchronous, and likely the cause for my troubles.

I have this function:

def communicate_wallet(wallet_command):
    buffer = BytesIO()
    c = pycurl.Curl()
    c.setopt(c.URL, '[::1]')
    c.setopt(c.PORT, 7076)
    c.setopt(c.POSTFIELDS, json.dumps(wallet_command))
    c.setopt(c.WRITEFUNCTION, buffer.write)
    c.perform()
    c.close()

    body = buffer.getvalue()
    parsed_json = json.loads(body.decode('iso-8859-1'))
    return parsed_json

This is equivalent to a curl command like:

curl -g -d '{ "action": "action_def" }' '[::1]:7076'

I'm wondering if there's an async alternative to do this, so i can call communicate_wallet with await. I couldn't seem to find any asynchronous-compatible alternatives to pycurl.

Thanks

Answer 1

I'm wondering if there's an async alternative to do this, so i can call communicate_wallet with await.

The simplest option is to use run_in_executor for the blocking code:

loop = asyncio.get_event_loop()
data = await loop.run_in_executor(None, communicate_wallet, wallet_command)

This will submit the blocking function to a thread pool and awaken your coroutine when complete, allowing asyncio to go about its business in the meantime.

A better way is to replace pycurl with an http client that natively supports asyncio, such as aiohttp. This will take more work initially, but might pay off in the long run because it will allow the http code to communicate with the tasks run by asyncio without thread synchronization.

Answer 2

The Tornado package seems to have what you want in the form of tornado.curl_httpclient.CurlAsyncHTTPClient.