Applying retry on grequests in Python

Question

I have a list of URLs for which I want to do a HTTP Get request using Python's grequests module.

Some of the URLs don't return an OK status, and in that case, I want to retry that URL.

I can do this using a Queue which stores all such URLs that have not been tried yet, or the ones that didn't return 200 in previous tries, and keep sending requests in batches. I am looking for a cleaner/more "Pythonic" implementation for this though.

I have used the retrying module for retries before and it is pretty neat. I'm wondering if there is some similar, concise implementation for retrying requests sent by grequests.

Answer 1

You can pass in your own requests session to grequests and set retrying on that session ie.

    s = requests.Session()
    retries = Retry(total=5, backoff_factor=0.2, status_forcelist=[500, 502, 503, 504], raise_on_redirect=True,
                    raise_on_status=True)
    s.mount('http://', HTTPAdapter(max_retries=retries))
    s.mount('https://', HTTPAdapter(max_retries=retries))
    reqs = (grequests.get(url, session=s) for url in urls)
    for resp in grequests.imap(reqs, stream=False):
        ...

You can put stream=True if you want streaming.