Leaking memory when using Selenium webdriver in a for loop

Question

I'm running a Python script which uses Selenium, on an EC2 instance (ubuntu).

Under my AWS plan, I have 2 GB memory. I upgraded to this from the free version after some performance issues with my script. However, when I check the free memory when connected to the Ubuntu server, I'm only seeing 348MB of available memory, and 353MB of free memory!

As of now, I'm only running two Python scripts, once a day, using crontab. The scripts run through a fairly long array of URLs and grabs information from each of them.

base_url = 'https://www.bandsintown.com/en/c/san-francisco-ca?page='

events = []
eventContainerBucket = []

for i in range(1,25):

    #cycle through pages in range
    driver.get(base_url + str(i))
    pageURL = base_url + str(i)
    
    # get events links
    event_list = driver.find_elements_by_css_selector('div[class^=_3buUBPWBhUz9KBQqgXm-gf] a[class^=_3UX9sLQPbNUbfbaigy35li]')
    # collect href attribute of events in even_list
    events.extend(list(event.get_attribute("href") for event in event_list))



    allEvents = []
    for event in events:
    
        driver.get(event)
        //do a bunch of other stuff

    driver.quit()

Is there an inherent problem in this code that would be causing memory leak? I would think free memory should go up again when the script has stopped running, but it doesn't.

I tried to invoke driver.close() within the for-loop, so that after the information is retrieved from each URL, the window closes. I was thinking this would help with memory leak - unfortunately, this gave me an error selenium.common.exceptions.InvalidSessionIdException: Message: invalid session id.

Am I on the right path with driver.close() or is something else entirely the problem?

Answer 1

Instead of driver quit, have you tried driver.close()?

close() - It is used to close the browser

quite() - It is used to shut down the web driver instance.

for i in range(1,25):

    #cycle through pages in range
    driver.get(base_url + str(i))
    pageURL = base_url + str(i)

    # get events links
    event_list = driver.find_elements_by_css_selector('div[class^=_3buUBPWBhUz9KBQqgXm-gf] a[class^=_3UX9sLQPbNUbfbaigy35li]')
    # collect href attribute of events in even_list
    events.extend(list(event.get_attribute("href") for event in event_list))

    allEvents = []
    for event in events:
        driver.get(event)

    driver.close()

I have checked your code its working fine without any issue. Please find below screenshot for more details