Selenium Unable to locate element only when using headless chrome on Aws Lambda (Python)

Question

I wrote a Python code that automate marking my attendance on this online community. I wanted it to run everyday so I've been trying to make it run on Aws Lambda.

It worked without any problems on my computer with a UI but but when I modified the script to run headless, it fails saying "unable to locate" element.

My code is as follows:

try:
    import json
    from selenium.webdriver import Chrome
    from selenium.webdriver.chrome.options import Options
    import os
    import shutil
    import uuid
    import boto3
    from datetime import datetime
    import datetime

    print("All Modules are ok ...")

except Exception as e:

    print("Error in Imports ")



class WebDriver(object):

    def __init__(self):
        self.options = Options()

        self.options.binary_location = '/opt/headless-chromium'
        self.options.add_argument('--headless')
        self.options.add_argument('--no-sandbox')
        self.options.add_argument('--start-maximized')
        self.options.add_argument('--window-size=1920,1080')
        self.options.add_argument('--single-process')
        self.options.add_argument('--disable-dev-shm-usage')

    def get(self):
        driver = Chrome('/opt/chromedriver', options=self.options)
        return driver



def lambda_handler(event, context):

    instance_ = WebDriver()
    driver = instance_.get()
    
    url=''
    id=''
    password=''
    
    driver.get(url)
    
    driver.find_element_by_id('login_id').send_keys(id)
    driver.find_element_by_id('login_pw').send_keys(password)

    driver.find_element_by_xpath('/html/body/div[6]/div/div/div/div/div/div/div[2]/div[1]/div/div/form/button').click() # click the 'login' button using XPATH




    driver.find_element_by_xpath('//*[@id="undefined-sticky-wrapper"]/nav/div[1]/div[1]/div/div/ul/li[7]/a').click() # click the community button, move to attendance

    driver.find_element_by_xpath('//*[@id="talk_submit"]').click() # click the attendance button
    
    
    
    
    return True

and this is the error message below.

"errorMessage": "Message: no such element: Unable to locate element: {\"method\":\"id\",\"selector\":\"login_id\"}\n  (Session info: headless chrome=69.0.3497.81)\n  (Driver info: chromedriver=2.43.600233 (523efee95e3d68b8719b3a1c83051aa63aa6b10d),platform=Linux 4.14.243-194.434.amzn2.x86_64 x86_64)\n",
  "errorType": "NoSuchElementException",

and this is the target HTML

<input type="text" name="mb_id" id="login_id" required="" class="form-control input-sm" size="20" maxlength="20" placeholder="아이디">

I've been googling it and I ran into another person who struggled with similar problems on StackOverFlow. one of the solutions suggests using WebDriverWait but I got a timeout too.

Please let me know if I'm missing something or doing something wrong. thank you

Answer 1

Possible solutions:

1. Try dumping the HTML page while in headless mode to see if there's any difference and change your xpath accordingly.

2. Update Chromedriver & Chrome to the latest versions

• Chromedriver: See the current latest version here. As of writing, this link should take you directly to the download: https://chromedriver.storage.googleapis.com/index.html?path=94.0.4606.41/
• Chrome: (on Linux)

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb

EDIT:

I've recently started a new project that makes use of Selenium. Thus, I have taken a look at this again and found a place to download updated binaries for both chromedriver and Chromium. You can visit this page, find a section correlated with your OS, and click on Archive to download a zip package. Unpack within your project directory and it's ready to use.