Manipulating web pages with python

Question

My aim is to be able to automatically manipulate a web page with a script. Filling in information and selecting the correct drop down box. With minimal user input.

So my example here I'm using the national rail website.

import win32com.client
from time import sleep

ie = win32com.client.Dispatch("InternetExplorer.Application")
ie.Visible = 1
ie.navigate("http://www.nationalrail.co.uk/")

while ie.ReadyState != 4: # Wait for browser to finish loading
    sleep(1)
print("Webpage Loaded")

page = ie.Document

links = page.links

If I wanted to change the box When "Leaving" to arriving, fill in the From "Station / Postcode" and hit Go. How would I go about doing this?

Also is win32com the best method for manipulating webpages like this?

Answer 1

While I'm sure every Python user can appreciate you trying to do this in the most difficult way possible, why not make things easier for yourself and use the library Selenium?

Here's your code & what you're trying to do, in Selenium:

from selenium import webdriver
driver = webdriver.Firefox() # Initialize the webdriver session
driver.get('http://www.nationalrail.co.uk/') # replaces "ie.navigate"
driver.find_element_by_id('sltArr').find_elements_by_tag_name('option')[1].click() # Selects the "Arrive" option

See? Much better looking! That last line selects the "Leaving" form, finds the option tags within it, and selects the Arrive option. With this bit of code, you should be able to figure out the rest of what you want to do with this site as well.