Why does my headless Selenium browser return an access denied error while the regular mode works?

[scarlettturner]

I’m experiencing an issue when trying to scrape a website using Selenium with Chrome in headless mode. When I run it with the browser interface visible, everything functions normally, and I can get all the content from the page. However, when I switch to headless mode, the site denies access and shows an “Access Denied” message.

Code that works (with GUI):

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait

options = Options()
driver = webdriver.Chrome(executable_path="/usr/local/bin/chromedriver", options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
url = "https://finance.yahoo.com/quote/AAPL"
driver.get(url)
wait.until(lambda browser: browser.execute_script('return document.readyState') == "complete")
print(driver.page_source)

Code that fails (headless mode):

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait

options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(executable_path="/usr/local/bin/chromedriver", options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
url = "https://finance.yahoo.com/quote/AAPL"
driver.get(url)
wait.until(lambda browser: browser.execute_script('return document.readyState') == "complete")
print(driver.page_source)

The headless code results in an access denied response instead of loading the page’s content. Can someone explain why there’s a difference between the two modes?

[lucask]

Had this exact problem scraping financial data a few months back. Headless browsers miss certain properties that regular browsers have, so they’re easier to spot. Setting a proper window size fixed it for me - some sites actually check viewport dimensions. Add --window-size=1920,1080 to your options. Also, headless Chrome handles JavaScript timing differently, so detection scripts might fail. Try adding --disable-web-security and --disable-features=VizDisplayCompositor too. Websites target headless environments because they scream automation instead of real users browsing.

[avamtz]

yep, that sounds right! sites like yahoo finance sometimes block headless modes since they can identify them. using --user-agent to mimic a regular browser should help. also, don’t forget to add --disable-blink-features=AutomationControlled to make it less suspicious.

[davidw]

Headless Chrome is missing properties that regular browsers have. Sites check for specific window objects and navigator properties - when they’re not there, you get flagged as a bot. I’ve run into this tons of times scraping financial sites. You need to make headless Chrome look like a real browser session. Add --disable-blink-features=AutomationControlled and use webdriver.Chrome(options=options, service_args=['--verbose']) to see what’s getting detected. Rotate your requests and throw in random delays between actions. Most sites also blacklist the default headless user agent, so spoofing that fixes most access problems.