Python urllib.request.Request parameter 'data' object type

Question

I try to use urllib.request.Request (Python 3.6.7) to make an API call to a internal web services to get some json results. I need to send some data and headers to the server, so I use the urllib.request.Request class to do this. For the input of data, I try to find out what is the format it will accept. From the Python docs, it says:

The supported object types include bytes, file-like objects, and iterables.

So I use a dictionary data type for this parameter data. Here is my code:

import urllib

my_url = "https://httpbin.org/post"
my_headers = { "Content-Type" : "application/x-www-form-urlencoded" }
my_data = {
        "client_id" : "ppp",
        "client_secret" : "000",
        "grant_type" : "client_credentials" }

req = urllib.request.Request(url=my_url, data=my_data, headers=my_headers)
response = urllib.request.urlopen(req)
html = response.read()
print(html)

I then get error like this:

Traceback (most recent call last):
  File "./callapi.py", line 23, in <module>
    response = urllib.request.urlopen(req)
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib64/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1064, in _send_output
    + b'\r\n'
TypeError: can't concat str to bytes

I then follow the example in this docs page, and change my code to:

import urllib

my_url = "https://httpbin.org/post"
my_headers = { "Content-Type" : "application/x-www-form-urlencoded" }
my_data = {
        "client_id" : "ppp",
        "client_secret" : "000",
        "grant_type" : "client_credentials" }

my_uedata = urllib.parse.urlencode(my_data)
my_edata = my_uedata.encode('ascii')

req = urllib.request.Request(url=my_url, data=my_edata,headers=my_headers)
response = urllib.request.urlopen(req)
html = response.read()
print(html)

it then works.

My question is, isn't it in the docs it says this class accept data type iterables ? why does my parameter in dict is wrong ? My final result that is working use str.encode() method which returns an byte object, and it seems this class must take a byte object and not an iterables object.

I am trying to use Python Standard Library docs as the main source of reference to code in Python, however I am having a hard time to use it, hope anybody can shed some light in helping me to understand more on how the library docs works, or if there is any other tutorial I need to go through before I can use it in a better way. Thanks.

Answer 1

I agree with you, the doc is not explicit. What is implicit is that if the data parameter is an iterable, it must be an iterable of bytes. When I have tried to pass a string as data I got an explicit error message:

TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

So for that reason, the iterable cannot be a dictionnary. As an exemple of valid iterable that in not a byte object (ok, just an example, no reason to use that in real code...):

def dict_iter(d):
    for i in d.items():
        yield(str(i).encode())

You can use that generator for the data parameter:

req = urllib.request.Request(url=my_url, data=dict_iter(my_data), headers=my_headers)