Programmatic Form Submit

Question

I want to scrape the contents of a webpage. The contents are produced after a form on that site has been filled in and submitted.

I've read on how to scrape the end result content/webpage - but how to I programmatically submit the form?

I'm using python and have read that I might need to get the original webpage with the form, parse it, get the form parameters and then do X?

Can anyone point me in the rigth direction?

Answer 1

Using python, I think it takes the following steps:

1. parse the web page that contains the form, find out the form submit address, and the submit method ("post" or "get").

this explains form elements in html file

2. Use urllib2 to submit the form. You may need some functions like "urlencode", "quote" from urllib to generate the url and data for post method. Read the library doc for details.