Get dynamically generated content with python Selenium

Question

This question has been asked before, but I've searched and tried and still can't get it to work. I'm a beginner when it comes to Selenium.

Have a look at: https://finance.yahoo.com/quote/FB

I'm trying to web scrape the "Recommended Rating", which in this case at the time of writing is 2. I've tried:

driver.get('https://finance.yahoo.com/quote/FB')
time.sleep(10)
rating = driver.find_element_by_css_selector('#Col2-4-QuoteModule-Proxy > div > section > div > div > div')
print(rating.text)

...which doesn't give me an error, but doesn't print any text either. I've also tried with xpath, class_name, etc. Instead I tried:

source = driver.page_source
print(source)

This doesn't work either, I'm just getting the actual source without the dynamically generated content. When I click "View Source" in Chrome, it's not there. I tried saving the webpage in chrome. Didn't work.

Then I discovered that if I save the entire webpage, including images and css-files and everything, the source code is different from the one where I just save the HTML.

The HTML-file I get when I save the entire webpage using Chrome DOES contain the information that I need, and at first I was thinking about using pyautogui to just Ctrl + S every webpage, but there must be another way.

The information that I need is obviosly there, in the html-code, but how do I get it without downloading the entire web page?

Answer 1

Try this to execute the dynamically generated content (JavaScript):

driver.execute_script("return document.body.innerHTML")

See similar question: Running javascript in Selenium using Python