Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to scrape JSON file from url and print contents

Here is the website I want to scrape: https://s2.coinmarketcap.com/generated/search/quick_search.json

I have no idea how to scrape the JSON file from the URL and print out the contents. I tried reading around, but the most I got was this code, which doesn't work when I run it:

import requests
import json

url = "https://s2.coinmarketcap.com/generated/search/quick_search.json"
r = requests.get(url)
cont = json.loads(r.content)
print(cont)
like image 336
j.doe Avatar asked Sep 06 '25 03:09

j.doe


2 Answers

Almost correct. Should be r.json(). Requests has a built in json parser.

import requests
import json

url = "https://s2.coinmarketcap.com/generated/search/quick_search.json"
r = requests.get(url)
cont = r.json()
print(cont)

Also if you want to print the specific contents just iterate over it like you would any other json obj

for k in cont:
    if k['name'] == 'VapersCoin':
        do_something()
like image 76
PhilippeT Avatar answered Sep 09 '25 01:09

PhilippeT


Looks like you ran into string hell. Sometimes with Python3 you have to do encode() or decode() to convert byte strings to unicode strings.

In your sample code, if you change line 6 to

cont = json.loads(r.content.decode())

it will run. @PhilippeT 's solution is also fine, as the json() function seems to automatically return a unicode string.

like image 22
bfris Avatar answered Sep 09 '25 01:09

bfris