How to select "Load more results" button when scraping using Python & lxml

Question

I am scraping a webpage. The webpage consists of 50 entries. After 50 entries it gives a Load more reults button. I need to automatically select it. How can I do it. For scraping I am using Python, Lxml.

Answer 1

Even JavaScript is using http requests to get the data, so one method would be to investigate, what requests are providing the data when user asks to "Load more results" and emulate these requests.

This is not traditional scraping, which is based on plain or rendered html content and detecting further links, but can be working solution.

Next actions:

• visit the page in Google Chrome or Firefox
• press F12 to start up Developer tools or Firebug
• switch to "Network" tab
• click "Load more results"
• check, what http requests have served data for loading more results and what data they return.
• try to emulate these requests from Python

Note, that the data do not necessarily come in HTML or XML form, but could be in JSON. But Python provide enough tools to process this format too.