aiohttp https request with an http proxy fail

Question

I am building a test project to learn asyncio. I am building a progam that fetches multiple pages over a proxy server.

It works fine for http pages, but it fails with https pages. When i use the regular request library i get https pages to work aswell, but not with asyncio.

I isolated the code that breaks:

import aiohttp
import asyncio

async def fetch(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url, proxy="http://192.168.0.2:9001") as response:
            print(await response.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(fetch("https://google.com")) #http websites do work

The error:

Traceback (most recent call last):
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\asyncio\proactor_events.py", line 395, in _loop_writing
    self._write_fut = self._loop._proactor.send(self._sock, data)
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 525, in send
    self._register_with_iocp(conn)
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 714, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] The parameter is incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/data/Python/playground/sayncio_session.py", line 10, in <module>
    loop.run_until_complete(fetch("https://google.com")) #http websites do work
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:/data/Python/playground/sayncio_session.py", line 6, in fetch
    async with session.get(url, proxy="http://192.168.0.2:9001") as response:
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\client.py", line 480, in _request
    conn = await self._connector.connect(
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 855, in _create_connection
    _, proto = await self._create_proxy_connection(
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 1093, in _create_proxy_connection
    transport, proto = await self._wrap_create_connection(
  File "C:\Users\GG\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host google.com:443 ssl:default [The parameter is incorrect]

Process finished with exit code 1

I am using aiohttp=3.6.2 and python=3.8.2

How can i fix this error?

Answer 1

I have also experienced this issue with aiohttp on Windows when trying to use the ProactorEventLoop. Since Python 3.8, it is now the default event loop on Windows (instead of the SelectorEventLoop), so until the the library gets a fix, the workaround is to switch back to SelectorEventLoop explicitly:

import aiohttp
import asyncio

async def fetch(url):
    ...

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

loop = asyncio.get_event_loop()
loop.run_until_complete(fetch(...))

Related: issue #2245 in aiohttp's tracker.