How to get content of a javascript/ajax -loaded div on a site?

Question

I have a PHP-script that loads page-content from another website by using CURL and simple_html_dom PHP library. This works great. If I echo out the HTML returned I can see the div-content there.

However, if I try to select only that div with the simple_html_dom, the div always returned empty. At first I didn't know why. Now I know that it's because its content apparently is populated with javascript/ajax.

How would I get the content of the site and then be able to select the div-content AFTER the javascript has populated it with the correct content?

Is it even possible? Thanks!

Answer 1

Yes its piece of cake if you are interested only in that particular html which is returned by ajax.

1. Gather information like url, parameters and request type (post/get) from that ajax request.
2. Generate the same request from your php/curl code and you got it.
3. And hope that server logic will not check who sent the request.

Answer 2

For this kind of screen scraping you could try phpQuery or Snoopy.

phpQuery has a web browser plugin and scoopy claims to simulate one