Why Python http request creates TIME_WAIT connections?

Question

I have this simple code, which connects with an external server. I call this function 100s of time a minute. And after a while I'm getting system lacked sufficient buffer exception. When I viewed the connections using TCPView it shows hundreds of connections to external server in TIME_WAIT status.

1. Why this is happening?

2. Is python request module not suitable if I have to send 100s of requests, then what should I do?

     def sendGetRequest(self, url, payload):
   
           success = True
           url = self.generateUrl(url)
           result = requests.get(url, params=urllib.parse.urlencode(payload))
           code = result.status_code
           text = result.text
   
           if code < 200 or code >= 300:
               success = False
   
           result.close()
           return success, code, text
   

Answer 1

You are closing a lot of connections you opened with requests at the client side, where the server expected them to be re-used instead.

Because HTTP is a TCP protocol, a bidirectional protocol, closing a socket on the client side means the socket can't yet fully close until the other end (the server end) acknowledges that the connection has been closed properly. Until the acknowledgement has been exchanged with the server (or until a timeout, set to 2x the maximum segment lifetime is reached), the socket remains in the TIME_WAIT state. In HTTP closing normally happens on the server side, after a response has been completed; it is the server that'll wait for your client to acknowledge closure.

You see a lot of these on your side, because each new connection must use a new local port number. A server doesn't see nearly the same issues because it uses a fixed port number for the incoming requests, and that single port number can accept more connections even though there may be any number of outstanding TIME_WAIT connection states. A lot of local outgoing ports in TIME_WAIT on the other hand means you'll eventually run out of local ports to connect from.

This is not unique to Python or to requests.

What you instead should do is minimize the number of connections and minimize closing. Modern HTTP servers expect you to be reusing connections for multiple requests. You want to use a requests.Session() object, so it can manage connections for you, and then do not close the connections yourself.

You can also drastically simplify your function by using standard requests functionality; params already handles url encoding, for example, and comparisons already give you a boolean value you could assign directly to success:

session = requests.Session()

def sendGetRequest(self, url, payload):
    result = session.get(self.generateUrl(url), params=payload)
    success = 200 <= result.status_code < 300
    return success, result.status_code, result.text

Note that a 3xx status code is already handled automatically, so you could just use response.ok:

def sendGetRequest(self, url, payload):
    result = session.get(self.generateUrl(url), params=payload)
    return result.ok, result.status_code, result.text

Next, you may want to consider using asyncio coroutines (and aiohttp, still using sessions) to make all those check requests. That way your code doesn't have to sit idle for each request-response roundtrip to complete, but could be doing something else in that intervening period. I've build applications that handle 1000s of concurrent HTTP requests at a time without breaking a sweat, all the while doing lots of meaningful operations while slow network I/O operations are completing.