Unable to locate elements on webpage with headless chrome

I had the same problem. You could take screenshots to understand whats wrong.

driver.get_screenshot_as_file("screenshot.png")

A few reasons why selenium works when run normally but stops working in headless mode -

1)It might have switched to a mobile template. Can be fixed by changing the window size.

chrome_options.add_argument("--window-size=1920,1080")

2)If its a blank page (screenshot), it might be due to an invalid SSL certificate.(see @Marcel_Wilson post) It should be fixed by -

chrome_options.add_argument('--ignore-certificate-errors')chrome_options.add_argument('--allow-running-insecure-content')

3)Its possible that the website blocks 'headless' mode. (Your screenshots might show errors which you cannot recreate in normal mode)You could try this-

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'options.add_argument(f'user-agent={user_agent}')

However, the above code won't work if the website has a more robust blocking method. You can find more about this here https://intoli.com/blog/making-chrome-headless-undetectable/.

I met the same situation. After studying, the following is correct：

self.chrome_options = webdriver.ChromeOptions()self.chrome_options.add_argument("--window-size=1920,1080")self.chrome_options.add_argument("--disable-extensions")self.chrome_options.add_argument("--proxy-server='direct://'")self.chrome_options.add_argument("--proxy-bypass-list=*")self.chrome_options.add_argument("--start-maximized")self.chrome_options.add_argument('--headless')self.chrome_options.add_argument('--disable-gpu')self.chrome_options.add_argument('--disable-dev-shm-usage')self.chrome_options.add_argument('--no-sandbox')self.chrome_options.add_argument('--ignore-certificate-errors')self.browser = webdriver.Chrome(options=self.chrome_options)

If it's an issue with SSL certs, you can start Chrome without certs using a command line flag (assuming that's how you're starting it). I believe the switch is --allow-running-insecure-content, and I located that from this list found here.