I’m working on a web scraping project using headless Firefox with Selenium. The script keeps failing when I call browser.get(target_url) with different connection issues.
Sometimes I get [Errno 104] Connection reset by peer and other times [Errno 111] Connection refused. The weird thing is that it works perfectly fine occasionally, maybe once every few days. When I test the same URLs with a regular Firefox browser on my MacBook, everything works without any problems.
I’ve tried several approaches to fix this:
Adding explicit waits for page elements
Setting implicit wait times
Using selenium-requests library with custom headers
Different timeout configurations
My setup includes Python with Selenium running on CentOS 6.5. The strange part is that other JavaScript-heavy sites work fine with the same configuration.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
fox_options = Options()
fox_options.add_argument('--headless')
browser = webdriver.Firefox(options=fox_options)
try:
browser.get('https://example-shopping-site.com/daily-deals')
# Script fails here with connection errors
except Exception as e:
print(f"Error occurred: {e}")
finally:
browser.quit()
Any suggestions on what might be causing these intermittent connection issues?
Check your DNS settings and network config on that CentOS box. Those intermittent errno 104/111 errors are usually network stack problems, not Selenium issues. Try adding --no-sandbox and --disable-dev-shm-usage to your Firefox options - fixed similar networking hiccups for me on older Linux systems running headless.
Those connection errors are definitely your CentOS 6.5 setup, not Firefox. That OS is ancient and has major SSL/TLS issues with modern sites. I’ve hit the same thing running Selenium on old Linux boxes - the network libraries just can’t handle current encryption. Since it works fine on your MacBook, that confirms it’s your environment. Upgrade to newer CentOS or switch to Ubuntu 18.04+ if you can. Quick fix: try setting custom Firefox preferences for network timeouts and SSL handling in the Firefox profile before starting the driver. Fixed my connection refused errors when I was stuck on legacy systems.
Had the same issue scraping e-commerce sites with headless Firefox. Turns out it’s their anti-bot detection targeting headless browsers specifically. These shopping sites have pretty sophisticated fingerprinting that catches headless mode even with spoofed user agents. Fixed it by rotating residential proxies and adding random delays between requests. Also set a real Firefox user agent and disabled automation indicators with options.add_argument('--disable-blink-features=AutomationControlled'). You’re getting intermittent success because the site lets some requests through - they don’t want to look too aggressive with blocking. I’d try undetected-chromedriver instead since some sites are less strict about Chrome automation.