Abort a GET request in Python when the server is not responding

Question

I've implemented an HTTP long-poller in Python. In simple terms, it's a program which periodically connects to a server and retrieve some informations with a GET request. The long-polling technique differs from the "normal" polling because if the server receive a request and it doesn't have new data, instead of sending an empty response it waits for new data to become available, leaving the request open.

Basically, when I make a request to this server, it might send me a reply immediately, or it might hold my GET open for minutes.

Everything works fine, the problem shows up when I want my program to close. I've tried both urllib3 and requests modules, they just don't let my program close if there is an hanging request.

So, my question is if there is a way to abort a GET request when the server is not responding (using the mentioned modules). In these cases, setting a timeout would solve my problem, but it is clearly impossible in this specific situation.

One solution might be to put the request in a thread and kill it when the program closes, but killing a thread isn't a good practice. Maybe there is a more indicated module to implement long-polling?

Answer 1

I suggest you to use urllib2 in a thread (I don't see any other way), then close the connection while waiting for response.

The other ways all solve the problem but not in a sense you'd like.

One is that you always return data from the server, but when you have no info to return, return some message indicating that the client should check again later.

One you mentioned is to kill the thread, but, well, it's not really nice solution.

I propose closing the connection to server and waiting for it to break:

from urllib2 import urlopen
from thread import start_new_thread as thread
from time import sleep

class Req:
    def __init__ (self, url, callback=lambda: None):
        self.polled_conn = None
        self.url   = url
        self.data  = None
        self.error = None
        self.callback = callback
        self.cancelled = 0
        self.polling = 0

    def poll (self):
        thread(self._poll,())

    def get (self):
        self.data  = None
        self.error = None
        cb = self.callback
        self.callback = lambda: None
        thread(self._poll,())
        while self.data==None and self.error==None: sleep(0.001)
        self.callback = cb
        if self.error: raise self.error
        return self.data

    def _poll (self):
        if self.polling: return
        self.polling = 1
        self.data  = None
        self.error = None
        self.cancelled = 0
        try:
            self.polled_conn = u = urlopen(self.url)
        except Exception, e: self.error = e; self.polling = 0; return self.callback()
        try:
            self.data = u.read()
        except AttributeError, e:
            if "recv" in str(e): self.cancelled = 1; self.polling = 0; return # Receiving aborted!
            else: self.error = e
        except IOError, e: self.cancelled = 1; self.polling = 0; return # Receiving aborted
        except Exception, e: self.error = e
        self.polling = 0
        self.callback()

    def cancel (self):
        if self.polled_conn:
            self.polled_conn.close()

    def iscancelled (self): return self.cancelled

Some usage is presented through get() method, but you have few more possibilities. With get() you have standard blocking return of data:

r = Req("http://example.com")
data = r.get()
print data

To achieve what you want, you can specify a callback here and use it to do something with data, and on program exit you can cancel any pending transfer:

def oncallback ():
    if r.data!=None:
        print len(r.data), r.data[:100]
    else: raise r.error
    sleep(1)
    r.poll() # Poll again for new info

r = None
def start ():
    global r
    r = Req("http://example.com", oncallback)
    r.poll()

start()
raw_input() # Wait for enter to close
r.cancel()
print "Cancelling"
if r.polling:
    while not r.iscancelled(): sleep(0.001)
print "Ready to close!"
# You can set r.polling to 1, to prevent the callback to poll again in midle of shutdown
#r.polling = 1 # Dirty trick
# Using threading module would make things just a little easier

Or you can still do it without callbacks (asynchronous-like):

running = 1
r = Req("http://example.com")
r.poll()
while running:
    if r.error!=None:
        print r.error
        # Decide what to do according to this error
        #if not isinstance(r.error, URLError): r.poll() # or something
        continue
    if r.data==None: sleep(0.001); continue # Do something anyway while waiting for data
    # Data arrived so:
    print data[:100] # Do some stuff
    r.poll() # Poll again for new
    # Somewhere here check whether user wants to close a program and then
    #running = 0 or break

r.cancel()
if r.polling:
    while not r.iscancelled(): sleep(0.001)

This works perfectly for me. Either for hanging connection or a transfer in progress. There may be still some caveats or errors, better to say bugs, to fix.