HtmlUnitDriver element location issues with By.name selector in headless testing

Other Questions Headless Browser
1

I’m working on a headless browser automation script but running into problems with element detection. My code is supposed to open a search page, grab the title, enter a search term, and check the title again.

package automationTest;

import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

public class SearchTest {

    public static void main(String[] args) {
        
        HtmlUnitDriver driver = new HtmlUnitDriver();
        
        driver.get("http://google.com");
        
        System.out.println("Page title: " + driver.getTitle());
        
        WebElement inputField = driver.findElement(By.name("q"));
        
        inputField.sendKeys("Selenium");
        
        WebElement submitBtn = driver.findElement(By.name("gbqfba"));
        
        submitBtn.click();
        
        System.out.println("New page title: " + driver.getTitle());
    }
}

When I run this I get:

Page title: 
Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element with name: q

The page title comes back empty and it can’t find the search input element. I checked the page source and the element with name=“q” is definitely there. The HTML shows something like this:

<input name="q" type="text" class="search-input" id="search-field" autocomplete="off" maxlength="2048" />

Running this on Windows with Eclipse. Anyone know why the headless driver might not be seeing these elements even though they exist in the DOM?

2

HtmlUnitDriver struggles with modern web pages because it doesn’t handle JavaScript and timing well. Google’s homepage loads content dynamically, which creates timing problems.

I ran into this same issue last year testing search functionality. Adding an implicit wait after driver initialization fixed it for me - driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS). This gives page elements time to load before your script tries to use them.

That submit button selector By.name("gbqfba") is probably outdated - Google changes their element names all the time. Try By.name("btnK") instead, or just use inputField.sendKeys(Keys.RETURN) after entering your search term.

The empty page title confirms the page isn’t fully loaded when your script runs.

3

I’ve encountered a similar issue with HtmlUnitDriver when dealing with modern websites. Google’s search page dynamically loads its elements using JavaScript, and HtmlUnitDriver often struggles with that. The empty title you see usually indicates that the page hasn’t fully rendered.

You might want to call driver.setJavascriptEnabled(true) right after initializing your driver. However, HtmlUnitDriver can still face challenges with such complex pages. I recommend incorporating a brief wait after the page loads to give the JavaScript some time to process.

If these suggestions don’t resolve your issue, consider switching to ChromeDriver operating in headless mode. It tends to be much more robust for contemporary web applications compared to HtmlUnitDriver.

4

Yeah, HTMLUnitDriver’s pretty outdated now. Google’s page probably isn’t loading right, which is why you’re getting an empty title. Switch to ChromeDriver with headless mode - it’s way more reliable for modern sites. Just use ChromeOptions options = new ChromeOptions(); options.addArgument("--headless"); and you’ll have much better luck.