playwright headless chromium can't find selector, but finds it in UI mode

Question

What I'm trying to do

I am doing some e2e testing with playwright on a webapp

The problem

I am running into problems whenever I want to save resources with headless mode. My playwright script is working perfectly in chromium UI mode. When encountering the first from (login), this happens:

[2022-03-31 07:57:38,079] [root] [ERROR] [MainThread]: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for selector "#username"

What I tried

I looked for answers, but can't find anything on this issue. I have absolutely no idea why an id selector would work in UI mode, but not in headless mode. It's just html and it's also able to pass other selectors like clicking buttons to get to the login. I'm at a loss here.

Code snippet:

# goto voice client
await page.click(selector='[title=log_in]')
await page.wait_for_timeout(timeout=500)

# enter credentials
await page.type(selector='#username', text=self.username, delay=50)
await page.type(selector='#password', text=self.password, delay=50)
await page.wait_for_timeout(timeout=100)

Answer 1

What I ran into when I saw this behavior is that when running in headless mode, the browser was failing bot detection tests on the site I was accessing. You can see this if you make sure and dump page.content() and inspect it in headless mode.

To get around this, you can use Playwright's Devices support to make sure and emulate a particular browser on a device when in headless mode:

https://playwright.dev/python/docs/api/class-playwright#playwright-devices

Below is some example code from that page that configures webkit to emulate an iPhone 6. Note how you setup a context and call new_page() on it instead of the browser.

    webkit = playwright.webkit
    iphone = playwright.devices["iPhone 6"]
    browser = webkit.launch()
    context = browser.new_context(**iphone)
    page = context.new_page()
    page.goto("http://example.com")

This approach passed the bot test.