Firefox browser window appears despite using headless mode, causing cron job failures

Other Questions Headless Browser
1

I’m experiencing an issue with my web scraping script that is supposed to run automatically through cron. Even though I’m using a headless browser gem, Firefox keeps launching visually when the code runs.

#!/usr/bin/env ruby
require 'watir'
require 'headless'

# Start headless session
virtual_display = Headless.new
driver = Watir::Browser.new :firefox
driver.goto 'http://dashboard.example.com/login'

# Navigate to login page
driver.link(:css => '.login-button').wait_until(&:present?).click
driver.input(:name => 'username').wait_until(&:present?).type '[email protected]'
driver.button(:class => 'continue-btn').click
driver.input(:name => 'password').wait_until(&:present?).type 'secret123'
driver.button(:class => 'submit-btn').click

# Extract data from dashboard
driver.goto 'http://dashboard.example.com/reports'
report_data = driver.divs(:class => 'metric-value').map(&:text)

# Cleanup
driver.quit
virtual_display.stop

puts "Data extracted at: #{Time.now}"
report_data.each { |item| puts item }

My bash script that executes the Ruby code is:

#!/bin/bash
today=$(date +"%Y_%m_%d")
ruby scraper.rb > ~/reports_$today.txt

The script works well when I run it manually, but it fails in the cron job because Firefox attempts to open a GUI window. I thought headless mode would stop this behavior. Where am I going wrong?

2

You’re creating the headless display but never actually starting it. You’ve got virtual_display = Headless.new but you’re missing virtual_display.start - that’s why Firefox still shows up on screen. Call virtual_display.start before you create the Watir browser. Also check that your cron job doesn’t have DISPLAY set, or make sure it points to your virtual display. I’ve hit this exact issue with scrapers before - the headless session wasn’t running when the browser launched.

3

This is exactly why I ditched browser automation years ago. Headless gem conflicts, cron environment issues, browser-specific implementations - it’s a maintenance nightmare.

Skip debugging Ruby gems and virtual displays. You can automate this whole thing without any browser code. Just set up scraping logic that runs on schedule, handles login, grabs your data, and saves everything automatically.

I’ve migrated dozens of scraping jobs from flaky browser automation to reliable cloud setups. No more cron failures, headless display problems, or broken dependencies after updates.

It’ll handle your login, hit the reports page, pull those metrics, and output them just like your current script. But it runs in a managed environment where headless actually works.

Same data extraction, zero browser headaches. Plus it scales way better for multiple sites or frequent jobs.

4

You’re initializing the Headless instance but not configuring the browser to actually use it. Skip the headless gem and pass the headless option straight to Watir: driver = Watir::Browser.new :firefox, headless: true. Remove the headless gem dependency completely. This works better because it uses Firefox’s built-in headless mode instead of creating a virtual display. I’ve seen the headless gem conflict with browser-specific headless modes, especially in cron jobs where display settings work differently than normal sessions.