Selenium headless mode causing page source issues after login

Other Questions Headless Browser
1

I’m having trouble with a Python script that uses Selenium to check appointment availability on a website. The code works fine when not in headless mode, but when I enable headless mode, the page source only shows ‘Unavailable’ after logging in.

Here’s a simplified version of what I’m trying to do:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--headless')

driver = webdriver.Chrome(options=chrome_options)
driver.get('https://example.com/login')

# Log in code here

if 'No appointments available' in driver.page_source:
    print('Closed')
else:
    print('Open')

driver.quit()

I’ve tried adding various Chrome options like user agents and wait processes, but nothing seems to work. The page source is still unavailable in headless mode.

Is the website blocking headless browsers? Are there any other solutions I can try to get this working in headless mode? I need to run this script automatically, so using a visible browser isn’t an option.

2

I’ve dealt with similar challenges using Selenium in headless mode. One effective approach I’ve found is utilizing undetected-chromedriver. This specialized fork of ChromeDriver is designed to evade detection mechanisms that some websites employ against automated browsers.

To implement this, you’d need to install the undetected-chromedriver package and modify your code slightly:

import undetected_chromedriver as uc

options = uc.ChromeOptions()
options.headless = True
driver = uc.Chrome(options=options)

This method has consistently worked for me on sites that were previously blocking headless access. It mimics a regular Chrome browser more closely, often bypassing anti-bot measures. If this doesn’t resolve the issue, you might need to investigate whether the site is using more sophisticated detection methods or if there are network-related problems specific to headless mode.

3

I’ve encountered similar issues with headless mode in Selenium, especially on sites with robust anti-bot measures. One trick that’s worked for me is to use a combination of user agent spoofing and additional Chrome flags.

Try adding these options to your chrome_options:

chrome_options.add_argument(‘–no-sandbox’)
chrome_options.add_argument(‘–disable-gpu’)
chrome_options.add_argument(‘–window-size=1920x1080’)
chrome_options.add_argument(‘–user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36’)

Also, consider implementing explicit waits instead of relying on page_source immediately after login. The site might be using dynamic content loading that takes a moment to populate.

If all else fails, you might need to look into using a service like Browserless or Puppeteer, which can sometimes bypass these restrictions more effectively than Selenium in headless mode.

4

hey, have you tried using a proxy? sometimes websites block certain IP ranges for automated access. you could try rotating through different proxies to see if that helps. also, make sure you’re handling any CAPTCHAs or other anti-bot challenges the site might throw at you in headless mode. just a thought!