Firefox browser window appears when using Headless gem, causing cron job failure

Other Questions Headless Browser
1

I’m trying to create an automated script that runs through cron, but I’m having issues with the Headless gem. Even though I’m using it, Firefox still opens a visible window when the script executes.

require 'watir-webdriver'
require 'headless'

# start headless session
virtual_display = Headless.new
virtual_display.start

# navigate to login page
driver = Watir::Browser.new :firefox
driver.goto 'https://portal.example.com/login'

# perform login steps
driver.link(:css => '.auth-link').wait_until_present
driver.link(:css => '.auth-link').click
driver.text_field(:name => 'username').set '[email protected]' 
driver.button(:class => 'continue-btn').click
driver.text_field(:name => 'password').set 'mypassword'
driver.button(:class => 'login-btn').click

# navigate to main page
driver.goto 'https://portal.example.com/reports'

# extract data here
data_collection = []
# data extraction logic

# cleanup
driver.quit
virtual_display.destroy

print "Data extracted at: #{Time.now}\n"
data_collection.each { |item| puts item }

My shell script is:

#!/bin/bash
today=$(date +"%Y%m%d")
ruby data_scraper.rb > ~/reports/extract_$today.log

The script works fine when I run it manually, but fails in cron. I think the visible Firefox window is the problem. Shouldn’t Headless prevent any GUI from appearing? I tried PhantomJS before but it couldn’t handle the Google authentication properly.

2

I ran into the same thing with watir scripts in cron. Turns out it wasn’t the display - Firefox was trying to access profile directories that don’t exist when running through cron. I fixed it by creating a dedicated Firefox profile for the script and setting it explicitly: driver = Watir::Browser.new :firefox, profile: '/path/to/your/profile'. Check your cron logs too with tail -f /var/log/cron - you might see permission errors. Sometimes what looks like a GUI problem is actually just filesystem access issues.

3

Your cron environment probably doesn’t have the display variables set up. When you run headless stuff through cron, you’ve got to configure the virtual display before starting the browser. Try adding ENV['DISPLAY'] = ":#{virtual_display.display}" right after virtual_display.start in your ruby script. Also make sure cron can find the firefox binary - either add the full path to your environment variables or just use absolute paths. I ran into the exact same thing with selenium scripts bombing out in cron. Turned out the PATH variables were totally different between cron and my regular user session.

4

yeah, try adding export DISPLAY=:99 to your script or crontab - that should fix it. also, make sure xvfb is actually installed since you need it for headless mode.