urllib3: How to get response when MaxRetryError is thrown?

Question

I'm using the Retry module from Python urllib3 + requests for a case where a 3rd party API gives sporadic errors. One issue I have is that if the retries keep failing, I get a exceptions.MaxRetryError and never get to see what the response was. What if there was valuable debugging data coming from the server?

Is there a way to still get the response in cases that would throw MaxRetryError?

Here's my code below

from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def req_with_retry(retries=3, backoff_factor=0.5, status_forcelist=(400, 404, 500, 502, 504,), method_whitelist=frozenset(['POST', 'HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE']), session=None,):
    ''' 
    this returns a session that functions like the requests module but with retries built it for certain status codes
    '''
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

r = req_with_retry().get(url="https://www.google.com")

Answer 1

First, I would like to mention that from requests.packages.urllib3.util.retry import Retry is no longer needed since urllib3 has been unvendored from requests. Consider using the from urllib3.util.retry import Retry form instead as it does not involve hacks (see https://github.com/psf/requests/blob/v2.22.0/requests/packages.py).

You can't get both the MaxRetryError and the response from urllib3. But you can get the response from urllib3, and use raise_for_status at the requests level. However, raise_for_status() won't be affected by your status_forcelist, so you may want to reimplement it yourself.

Here's an adaption of your code that 1/ retries on statuses 2/ prints the final response 3/ raises an exception.

import logging

import urllib3
import requests
import requests.adapters

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)


def req_with_retry(
    retries=3,
    backoff_factor=0.5,
    status_forcelist=(400, 404, 500, 502, 504),
    method_whitelist=frozenset(
        ["POST", "HEAD", "TRACE", "GET", "PUT", "OPTIONS", "DELETE"]
    ),
    session=None,
):
    """Returns a session that functions like the requests module but with retries
    built it for certain status codes
    """
    session = session or requests.Session()
    retry = urllib3.Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        method_whitelist=method_whitelist,
        raise_on_status=False,
    )
    adapter = requests.adapters.HTTPAdapter(max_retries=retry)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session


r = req_with_retry().get(url="https://httpbin.org/status/404")
print(r.status_code, r.text)
r.raise_for_status()