Scraping elements generated by javascript queries using python

Question

I am trying to access the text in an element whose content is generated by javascript. For example getting the number of twitter shares from this site.

I've tried using urllib and pyqt to obtain the html of the page, however since the content requires javascript to be generated, its HTML is not present in the response of urllib/pyqt. I am currently using selenium for this task, however it is taking longer than I would like it to.

Is it possible to get access to this data without opening the page in a browser?

This question has already been asked in the past, but the results I found are either c# specific or provide a link to a solution that has since gone dead

Answer 1

Working example :

import urllib
import requests
import json

url = "https://daphnecaruanagalizia.com/2017/10/crook-schembri-court-today-pleading-not-crook/"

encoded = urllib.parse.quote_plus(url)
# encoded = urllib.quote_plus(url) # for python 2 replace previous line by this
j = requests.get('https://count-server.sharethis.com/v2.0/get_counts?url=%s' % encoded).text
obj = json.loads(j)
print(obj['clicks']['twitter'] + obj['shares']['twitter'])

# => 5008

Explanation :

Inspecting the webpage, you can see that it does a request to this :

https://count-server.sharethis.com/v2.0/get_counts?url=https%3A%2F%2Fdaphnecaruanagalizia.com%2F2017%2F10%2Fcrook-schembri-court-today-pleading-not-crook%2F&cb=stButtons.processCB&wd=true

If you paste it in your browser you'll have all your answers. Then playing a bit with the url, you can see that removing extra parameters will give you a nice json.

So as you can see, you just have to replace the url parameter of the request with the url of the page you want to get the twitter counts.

Answer 2

You could do something similar to the following after launching a Selenium web browser, then passing driver.page_source to the BeautifulSoup library (unfortunately cannot test this at work with firewalls in place):

soup = BeautifulSoup(driver.page_source, 'html.parser')

shares = soup.find('span', {'class': 'st_twitter_hcount'}).find('span', {'class': 'stBubble_hcount'})