Python Requests package: lost connection while streaming

Question

I'd like to use the Requests package to connect to the streaming API of a web service. Suppose I use the following code to send a request, receive the response and iterate through the lines of response as they arrive:

import requests

r = requests.get('http://httpbin.org/stream/20', stream=True)
for line in r.iter_lines():
    if line:
        print line

While waiting to receive new data, we are basically waiting for r.iter_lines() to generate a new piece of data. But what if I lose internet connection while waiting? How can I find out so I can attempt to reconnect?

Answer 1

You can disconnect from your network to have a try. Requests raise such error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /stream/20 (Caused by : [Errno -3] Temporary failure in name resolution)

The error message shows Requests already retries for network error. You can refer to this answer for setting the max_retries. If you wants more customization (e.g. waits between retries), do it in a loop:

import socket
import requests
import time

MAX_RETRIES = 2
WAIT_SECONDS = 5

for i in range(MAX_RETRIES):
    try:
        r = requests.get('http://releases.ubuntu.com/14.04.1/ubuntu-14.04.1-desktop-amd64.iso',
                         stream=True, timeout=10)
        idx = 1
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
                print 'Chunk %d received' % idx
                idx += 1
        break
    except requests.exceptions.ConnectionError:
        print 'build http connection failed'
    except socket.timeout:
        print 'download failed'
    time.sleep(WAIT_SECONDS)
else:
    print 'all tries failed'

EDIT: I tested with a large file. I used iter_content instead, because it's a binary file. iter_lines is based on iter_content (source codes), so I believe the behaviour is same. Procedure: run the codes with network connected. After receiving some chunks, disconnect. Wait 2-3 seconds, reconnect, the downloading continued. So requests package DOES retry for connection lost in the iteration.

Note: If no network when build the connection (requests.get()), ConnectionError is raised; if network lost in the iter_lines / iter_content, socket.timeout is raised.