Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why render / requests-html doesn't scrape dynamic content?

Long story short : switched from Selenium to Requests(-html).

Works OK but not in every case.

Page : https://www.winamax.fr/paris-sportifs/sports/1/1/1

Upon load it charges dynamic content with english games (example : Sheffield United - West Ham).

But when I try to do this :

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.winamax.fr/paris-sportifs/1/1/1')
r.html.render()
print(r.html.text) # I also tried print(r.html.html)

the games don't show in the output.

Why ? Thanks !

like image 245
jeremoquai Avatar asked Oct 13 '25 02:10

jeremoquai


1 Answers

add timeout, it should work, sorry this must be a comment but I cannot comment..

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.winamax.fr/paris-sportifs/sports/1/1/1')
r.html.render(timeout=20)
print(r.html.html)
session.close()
like image 152
fardV Avatar answered Oct 14 '25 18:10

fardV