HtmlUnitDriver returns blank page instead of loading website

Other Questions Headless Browser
1

I’m working with Selenium WebDriver for my automation testing. I want to run tests without opening a visible browser window using HtmlUnitDriver but I’m having issues.

When I try to load a webpage with HtmlUnitDriver, it doesn’t work correctly. Here’s my test code:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class HeadlessBrowserTest 
{
    public static void main(String[] args) throws InterruptedException
    {
        HtmlUnitDriver browserDriver = new HtmlUnitDriver();
        //browserDriver.setJavascriptEnabled(true);

       // WebDriver browserDriver = new ChromeDriver();

        browserDriver.get("https://www.google.com"); 

        /*WebDriverWait waitForElement = new WebDriverWait(browserDriver, 120);
        waitForElement.until(ExpectedConditions.presenceOfElementLocated(By.id("search_input")));
        */
        Thread.sleep(50000);
        System.out.println("Current URL: " + browserDriver.getCurrentUrl());

        System.out.println("Title of page: " + browserDriver.getTitle());     
    }
}

The result I get is:

Current URL: about:blank
Title of page:

When I switch to ChromeDriver it works perfectly fine. What am I missing with HtmlUnitDriver setup?

2

Yeah, the commented out JavaScript line is definitely causing issues, but there’s more to it. HtmlUnitDriver gets really picky with SSL certificates and modern security stuff. I’ve seen sites return completely blank pages because of certificate validation errors or security headers that HtmlUnitDriver just can’t handle.

Try adding these lines after you create your driver:
browserDriver.getWebClient().getOptions().setThrowExceptionOnScriptError(false);
browserDriver.getWebClient().getOptions().setUseInsecureSSL(true);

These settings saved me from a bunch of headaches by bypassing the stricter security checks that were blocking pages from loading. Also, Google’s got some pretty aggressive bot detection that might be messing with HtmlUnitDriver requests.

3

uncomment that setJavascriptEnabled(true) line - that’s what’s causing your blank page. HtmlUnitDriver disables JS by default, but Google needs it to load. Also, throw in a user agent string since some sites block requests with missing headers.

4

I faced similar problems when I tried using HtmlUnitDriver for projects relying on modern JavaScript. The issue is that its JavaScript support is not robust enough to handle dynamic content effectively. Websites like Google load content dynamically, and HtmlUnitDriver might only retrieve the initial HTML but fail to execute the JavaScript that populates the page with content. Consequently, you end up with a blank page. For headless testing, consider using ChromeDriver with headless mode enabled, which better supports modern web technologies and will give you accurate rendering.