Making a POST request using urllib with multiple headers gives 400 Bad Request error

Question

I have used requests library and I know how to work with it, but I need to work with standard library only, so I would appreciate if you don't encourage me to use requests instead.

I made a flask server that handles POST requests and then from a different script I call urllib to make POST calls to the flask server. I need to send a raw json in body just like we do in Postman.

Flask Server

from flask import Flask, request

app = Flask(__name__)
@app.route('/random', methods=['POST'])
def random():
    if request.method == 'POST':
        if request.headers.get('Authorization') and request.headers.get('Content-Type') == 'application/json':
            print(request.get_json())
            return "Success"
        else:
            print(request.get_json())
            return "Bad request"

app.run(host='0.0.0.0', port=5000, debug=True)

Urllib Client (saved as test.py) -

import urllib.request
import urllib.parse
d = {"spam": 1, "eggs": 2, "bacon": 0}
data = urllib.parse.urlencode(d)
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
    print(f.read().decode('utf-8'))

With only Authorization header I get Bad Request as output and the json is None on the flask server side as expected.

With ONLY Content-Type header OR both the headers I get this error -

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    with urllib.request.urlopen(req) as f:
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: BAD REQUEST

The whole thing is simple enough, but I am not able to understand why is this happening and the error message doesn't help much either.

Answer 1

The server is failing in request.get_json(). It's only happening when the client sends both headers because that's when it reaches this line.

To fix it, change the client to send the data as JSON:

import json            # <-- Import json
import urllib.request
import urllib.parse

d = {"spam": 1, "eggs": 2, "bacon": 0}
data = json.dumps(d)   # <-- Dump the dictionary as JSON
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
    print(f.read().decode('utf-8'))

I hope this helps