I’m developing a Python script to log into a website and check for available appointment times. The script functions well in normal Chrome mode, but when I switch to headless mode, I only receive ‘Unavailable’ in the page source instead of the expected content.
Below is my code example:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import time
# Link and credentials
website_url = 'https://appointments.example.com/booking/123'
user_email = '[email protected]'
user_pass = 'mypassword123'
search_message = "No available slots right now"
# Configuring Chrome options
options = Options()
options.add_argument("--headless") # This is causing the issue
# Launching the browser
browser = webdriver.Chrome(options=options)
browser.implicitly_wait(5)
# Visiting the login page
browser.get(website_url)
# Performing login
email_field = browser.find_element('name', 'userEmail')
password_field = browser.find_element('name', 'userPassword')
email_field.send_keys(user_email)
password_field.send_keys(user_pass)
password_field.send_keys(Keys.RETURN)
# Pause and check the page content
time.sleep(5)
if search_message in browser.page_source:
print('No slots available')
else:
print('Slots are open')
browser.quit()
When headless mode is off, I see the correct page after logging in. However, enabling headless mode results in a page source that only shows the text ‘Unavailable’. I’ve attempted to add user agent strings and several Chrome flags, like disable-blink-features and no-sandbox, but nothing seems to help. Is it possible that the site is identifying headless browsers and blocking them? What are some other tactics I can employ to resolve this issue in headless mode?
This happens because websites use JavaScript detection that specifically targets headless browsers. They’re checking for browser properties or behaviors that are different between headless and regular Chrome.
Besides the automation flags, try setting a realistic window size with --window-size=1920,1080. Headless browsers often use weird dimensions that trigger detection. You’ll probably want --disable-dev-shm-usage and --disable-extensions too.
You could also switch to undetected-chromedriver instead of regular selenium. It handles most detection bypasses automatically and updates regularly to beat new detection methods. Just heads up though - some sites now use server-side detection that analyzes request patterns and timing, so they’re much harder to get around no matter what browser config you use.
yep, sounds like bot detection! try adding --disable-blink-features=AutomationControlled and run webdriver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})") right after launching. sites r getting pretty good at blocking headless browsers lately.
Had the exact same problem last month scraping booking sites. It’s not just detection flags - many sites load different content based on what your browser can handle. Try adding --disable-web-security and --allow-running-insecure-content to your options. Also bump up your wait time - headless mode loads way slower than you’d think. What really saved me was adding options.add_experimental_option('useAutomationExtension', False) and options.add_experimental_option('excludeSwitches', ['enable-automation']) before the headless flag. Some sites check for specific navigator properties that only show up in automated browsers. If none of this works, try using a virtual display with xvfb instead of true headless mode. It’s technically not headless but acts the same while dodging most detection methods.