Selenium Chrome getting text not work on headless mode

Question

I am experiencing a strange behaviour from selenium when running it on headless mode with Chrome webdriver. Up to now, I did not have this problem before to get text in headless mode, it always worked.

The reproduceble example is given bellow:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
#options.add_argument('--headless')
#options.add_argument('--no-sandbox')

driver = webdriver.Chrome(chrome_options=options)

driver.get("https://www.zoom.com.br/ar-condicionado/todos")

wait = WebDriverWait(driver, 10)

stores = wait.until(
    EC.presence_of_all_elements_located((By.XPATH,
                                        './/span[@class="storeCount-txt"]')))

print(stores[0].text)

When I ran this peace of code the output is:

> em 14 lojas

However, when I run it on headless mode (remove #s), the output is empty:

> ""

Any ideas on what is going on?

Answer 1

I had the same issue when I was running my webcrawling script deployed in heroku running chrome in headless mode. I solved it by adding the following chrome option to my list of options

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--window-size=1920x1080") #I added this

Just as mentioned on the comment in your question, there could be two things that could make some elements not show up

1. The resolution you are on does not display that element(or something like that) which I have solved by adding that option

2. You are searching for the element when it is not yet loaded. I suggest waiting appropriately (which you have done with the stores variable) you could also use this instead

    try:
       # Wait until 'what you specified' is visible
       WebDriverWait(driver, 60) \
           .until(expected_conditions.visibility_of_element_located((By.XPATH, './/span[@class="storeCount-txt"]')))
    except Exception as exp:
       print("Exception occured", exp)
       driver.quit()
   

Hope this helps