Python requests: URL with percent character

Question

I have been searching all over the place for this, but I couldn't solve my issue.

I am using a local API to fetch some data, in that API, the wildcard is the percent character %.

The URL is like so : urlReq = 'http://myApiURL?ID=something&parameter=%w42'

And then I'm passing this to the get function:

req = requests.get(urlReq,auth=HTTPBasicAuth(user, pass))

And get the following error: InvalidURL: Invalid percent-escape sequence: 'w4'

I have tried escaping the % character using %%, but in vain. I also tried the following:

urlReq = 'http://myApiURL?ID=something&parameter=%sw42' % '%' but didn't work as well.

Does anyone know how to solve this?

PS I'm using Python 2.7.8 :: Anaconda 1.9.1 (64-bit)

Answer 1

You should have a look at urllib.quote - that should do the trick. Have a look at the docs for reference.

To expand on this answer: The problem is, that % (+ a hexadecimal number) is the escape sequence for special characters in URLs. If you want the server to interpret your % literaly, you need to escape it as well, which is done by replacing it with %25. The aforementioned qoute function does stuff like that for you.

Answer 2

Let requests construct the query string for you by passing the parameters in the params argument to requests.get() (see documentation):

api_url = 'http://myApiURL'
params = {'ID': 'something', 'parameter': '%w42'}
r = requests.get(api_url, params=params, auth=(user, pass))

requests should then percent encode the parameters in the query string for you. Having said that, at least with requests version 2.11.1 on my machine, I find that the % is encoded when passing it in the url, so perhaps you could check which version you are using.

Also for basic authentication you can simply pass the user name and password in a tuple as shown above.