HtmlUnit Java Library Not Working Properly - Getting Same Page After Click

Other Questions Headless Browser
1

I’m having trouble with HtmlUnit while trying to automate a ticket booking website. When I select a quantity and click the submit button, I expect to get a captcha page like what happens in a real browser. But HtmlUnit just returns the same page instead.

Here’s my setup code:

WebClient client = new WebClient(BrowserVersion.CHROME);
client.getOptions().setJavaScriptEnabled(true);
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setCssEnabled(true);
client.getOptions().setUseInsecureSSL(true);
client.getOptions().setRedirectEnabled(true);
client.setAjaxController(new NicelyResynchronizingAjaxController());

// Get the page
HtmlPage mainPage = client.getPage("https://example-tickets.com/event/12345");

// Select quantity
HtmlSelect quantityDropdown = mainPage.getHtmlElementById("qty_selector");
quantityDropdown.setSelectedAttribute("2", true);

// Click submit
HtmlInput submitBtn = mainPage.getHtmlElementById("submit_btn");
HtmlPage resultPage = submitBtn.click();

// Wait for response
Thread.sleep(30000);

System.out.println("Result page: " + resultPage.asXml());

The problem is that resultPage is identical to mainPage. In a real browser, this would show me a captcha verification page. Why isn’t HtmlUnit getting the same response that a regular browser would get? What am I missing in my configuration?

2

Booking sites usually have invisible captcha that kicks in based on how you interact with the page. HTMLUnit probably isn’t triggering the right events like mouseover or focus before you click. Try adding page.getWebClient().waitForBackgroundJavaScript(5000) after you select quantity, then use fireEvent() to simulate mouse interactions before hitting submit.

3

This is probably an AJAX form submission issue. Most booking sites use async requests to validate your selections before moving to the next step. Your code waits after clicking, but HtmlUnit might not be catching the AJAX response properly. Try client.waitForBackgroundJavaScript(10000) right after the click instead of Thread.sleep(). Also check if the form submission triggers any network requests by monitoring the WebClient’s response. I’ve seen this with similar sites - the page transition happens through JavaScript after form validation finishes. You’ll probably need to watch for specific DOM changes or wait for certain elements to show up rather than expecting an immediate redirect. Some sites also need the referer header to match the current page URL for security.

4

This issue often arises when the site recognizes automated tools, serving different content compared to a real browser. It’s likely that the site is checking headers, user agent fingerprints, or JavaScript behavior that HtmlUnit struggles to replicate adequately. I’ve faced similar challenges with booking sites. Even with BrowserVersion.CHROME, HtmlUnit can be easily detected. Consider modifying headers like Accept-Language, Accept-Encoding, or Cache-Control to appear more realistic. Remember, such sites usually employ complex bot detection methods, including monitoring timing patterns and mouse movements. Additionally, the form may rely on specific cookies or session tokens generated by JavaScript after loading. Your 30-second pause might be counterproductive because the JavaScript could still be executing when you trigger the click. A shorter wait could ensure that everything loads correctly before you submit.