Requests http lib: include X-Request-ID

Question

I use the requests HTTP client library in Python.

Sometimes an HTTP request fails, and I get an HTTP response with status 500.

This can be in CI or production, and I see something like this:

AssertionError: 200 != 500 : <Response [500]>

That does not help much.

It would be very great if I could see the X-Request-ID in the above message. In my environment, this exists for every HTTP response.

This means the HTTP response object of the requests library should add it.

I want the repr() to look like <Response [500] XejfkmxcPfhM3dqhY2HJgQAAAAM>

Since this is not my code, but the code of the requests lib, I am unsure how I could implement this.

How to include the X-Request-ID in the repr() of requests response objects?

Answer 1

I would prefer to create my own "response flow" in case I can do it, instead of using any sort of monkey patching. I've checked sources of the requests library and we are good to go there! We can implement the required feature using requests Event Hooks system.

There is a working example where we can do anything with the response. The only change is that we have to use our own Session object instance for using the feature. But! If we don't want to override any source code, we can do a one linemonkey-patch, override the default Session class for the default api calls and it will work everywhere like a charm.

My solution is here

import requests


class ResponseVerbose(requests.Response):
    extra_header_repr = 'X-Request-Guid'

    def __repr__(self):
        return '<Response [{}] {}: {}>'.format(
            self.status_code,
            self.extra_header_repr,
            self.headers.get(self.extra_header_repr, 'None')
        )


class Session(requests.Session):
    def __init__(self):
        super().__init__()

        self.hooks['response'] = self.build_response

    @staticmethod
    def build_response(resp, *args, **kwargs):
        """
        Let's rebuild the source response into required verbose response object using all fields from the original

        FYI: requests.adapters.HTTPAdapter.build_response
        """
        response = ResponseVerbose()
        response.status_code = resp.status_code
        response.headers = resp.headers
        response.encoding = resp.encoding
        response.raw = resp.raw
        response.reason = response.raw.reason
        response.url = resp.url
        response.cookies = resp.cookies.copy()
        response.request = resp.request
        response.connection = resp.connection

        return response


def main():
    url = 'https://stackoverflow.com/'

    sess = Session()
    print('response using our own session object: {}'.format(sess.get(url)))

    import requests.api
    requests.api.sessions.Session = Session
    print('response using monkey patched global Session class: {}'.format(requests.get(url)))


if __name__ == '__main__':
    main()

outputs

# python test123.py
response using our own session object: <Response [200] X-Request-Guid: 0c446bb5-7c96-495d-a831-061f5e3c2afe>
response using monkey patched global Session class: <Response [200] X-Request-Guid: 1db5aea7-8bc9-496a-addc-1231e8543a89>

One more shorter example which uses Response.__getstate__() function

More info https://github.com/psf/requests/blob/master/requests/models.py#L654

As I see from the source code, you shouldn't do it for very large content responses since it fetches the whole resp.content to be able to convert the response state into a state dict. So use it only if you know that there are not gigabytes in responses :)

The function looks much easier.

import requests


class ResponseVerbose(requests.Response):
    extra_header_repr = 'X-Request-Guid'

    def __repr__(self):
        return '<Response [{}] {}: {}>'.format(
            self.status_code,
            self.extra_header_repr,
            self.headers.get(self.extra_header_repr, 'None')
        )


class Session(requests.Session):
    def __init__(self):
        super().__init__()

        self.hooks['response'] = self.build_response

    @staticmethod
    def build_response(resp, *args, **kwargs):
        """
        Let's rebuild the source response into required verbose response object using all fields from the original

        FYI: requests.adapters.HTTPAdapter.build_response
        """

        response = ResponseVerbose()
        for k, v in resp.__getstate__().items():
            setattr(response, k, v)

        return response


def main():
    url = 'https://stackoverflow.com/'

    sess = Session()
    print('response using our own session object: {}'.format(sess.get(url)))

    import requests.api
    requests.api.sessions.Session = Session
    print('response using monkey patched global Session class: {}'.format(requests.get(url)))


if __name__ == '__main__':
    main()

The solution prints StackOverflow's extra response header X-Request-Guid just for example. This extra header I did easily configurable just to show how it could be done in the right way.

Answer 2

A way would be to overwrite the requests.models.Response.__repr__ method at runtime (also called [Wikipedia]: Monkey patch) as @heemayl commented. Note that this is one variant (the simplest, I think) of this way.

code00.py:

#!/usr/bin/env python3

import sys
import requests


def __response_repr(self):
    repr_headers = (
        "X-Request-ID",
        "Content-Encoding",  # @TODO - cfati: For demo purposes only!!! DELETE (COMMENT) THIS LINE.
    )
    repr_parts = ["<Response [{0:d}]".format(self.status_code)]
    for repr_header in repr_headers:
        if repr_header in self.headers:
            repr_parts.append(" {0:}".format(self.headers[repr_header]))
    repr_parts.append(">")
    return "".join(repr_parts)


def main(*argv):
    if argv:
        print("Monkey patch {0:}...\n".format(requests.models.Response))
        requests.models.Response.__str__ = requests.models.Response.__repr__  # Keep the original __str__ behavior
        requests.models.Response.__repr__ = __response_repr

    url = "https://www.google.com"
    print("Connecting to: {0:s} ...\n".format(url))

    r = requests.get(url)
    print("str(Response): {0:s}".format(str(r)))
    print("repr(Response): {0:s}".format(repr(r)))


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    main(*sys.argv[1:])
    print("\nDone.")

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q059193447]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" code00.py
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32

Connecting to: https://www.google.com ...

str(Response): <Response [200]>
repr(Response): <Response [200]>

Done.

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q059193447]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" code00.py dummy_arg_to_trigger_monkey_patch
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32

Monkey patch <class 'requests.models.Response'>...

Connecting to: https://www.google.com ...

str(Response): <Response [200]>
repr(Response): <Response [200] gzip>

Done.