Chrome headless returns blank HTML when using BrowserMob Proxy - need solution

Other Questions Headless Browser
1

I’m working on capturing network traffic from my automated tests using BrowserMobProxy. Everything works perfectly with regular Chrome browser but I’m having issues with headless mode.

Working setup (regular Chrome):

Proxy webDriverProxy = ClientUtil.createSeleniumProxy(mobProxy);
webDriverProxy.setNoProxy("<-loopback>");
webDriverProxy.setHttpProxy("localhost" + ":" + mobProxy.getPort());
webDriverProxy.setSslProxy("localhost" + ":" + mobProxy.getPort());

Capabilities caps = DesiredCapabilities.chrome();
caps.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.IGNORE);
caps.setCapability(CapabilityType.ACCEPT_SSL_CERTS, Boolean.TRUE);
caps.setAcceptInsecureCerts(Boolean.TRUE);

ChromeOptions chromeOpts = setupChromeOptions(config.getUserAgent(), Boolean.TRUE);
chromeOpts.setCapability(CapabilityType.PROXY, webDriverProxy);
caps.setCapability(ChromeOptions.CAPABILITY, chromeOpts);

return new RemoteWebDriver(remoteUrl, caps);

Problem setup (headless Chrome):

ChromeOptions chromeOpts = new ChromeOptions();
chromeOpts.addArguments("--user-agent=" + "HeadlessChrome");
chromeOpts.setAcceptInsecureCerts(Boolean.TRUE);
chromeOpts.setHeadless(true);
chromeOpts.addArguments("--allow-insecure-localhost", "--no-sandbox", "--disable-extensions", "--window-size=1920,1080");

List<String> proxyArgs = Arrays.asList("--ignore-certificate-errors", "--proxy-bypass-list=<-loopback>",
    "--proxy-server=http://localhost:" + proxyInstance.getPort(), "--remote-debugging-port=9222");
chromeOpts.addArguments("--ssl-protocol=any");
chromeOpts.addArguments("--allow-running-insecure-content");
chromeOpts.addArguments(proxyArgs);

desiredCaps.setCapability(ChromeOptions.CAPABILITY, chromeOpts);
webDriver = new RemoteWebDriver(driverEndpoint, desiredCaps);

The headless version only shows blank pages and returns empty HTML like <html><head></head><body></body></html>. I tried adding wait times but nothing helps.

I’m using browsermob-proxy 2.1.5 and ChromeDriver 92.0.4515.159 on Linux. The proxy starts with:

System.setProperty("bmp.allowNativeDnsFallback", "true");
BrowserMobProxy proxyInstance = new BrowserMobProxyServer();
proxyInstance.setTrustAllServers(Boolean.TRUE);

Any ideas what might be causing this issue with headless mode?

2

Had this exact problem last year - wasted way too much time on it. Headless Chrome bypasses proxy configs that work fine in regular mode, especially SSL stuff. What fixed it: add --disable-web-security to your headless Chrome options. Also, set the proxy through Chrome arguments only - don’t mix WebDriver proxy capabilities with Chrome arguments. Pick one. Ditch the DesiredCapabilities proxy setting completely and handle everything through Chrome arguments. Your --proxy-server looks right, but add --disable-web-security and --disable-features=VizDisplayCompositor. One more thing - you’re using different user agents between setups. Keep the same user agent string. Sites sometimes serve different content based on user agent, which could explain those blank pages. SSL certificate issues in headless mode cause silent failures all the time. Those extra security flags should help Chrome accept the proxy’s certificates properly.

3

Been there, done that. Ditched this complex proxy setup months ago for exactly this reason.

You’re hitting a super common issue - headless Chrome handles proxy settings differently than regular mode. SSL certificate validation gets wonky and proxy tunneling breaks silently.

I wasted hours debugging similar setups until I found a cleaner approach. Instead of wrestling with BrowserMobProxy and headless Chrome, I moved all network monitoring into a proper automation workflow.

Now I set up monitoring scenarios that capture network data without any proxy config. The workflow handles browser automation, captures network requests, processes data, and stores results in one flow.

No proxy server management, no SSL certificate headaches, no headless vs regular mode differences. Just clean automation that works consistently.

The debugging time savings alone make this worth it. Plus you get better reporting and easy integration with other testing tools.

Check out Latenode for these automation workflows: https://latenode.com

4

Ugh, this bug again… Add --disable-gpu and --disable-dev-shm-usage to your headless args. Chrome has memory issues on Linux that cause blank pages. Your BrowserMob version is ancient too - 2.1.5 breaks with newer Chrome versions. Upgrade to the latest BrowserMob or switch to LittleProxy. Works way better with headless mode.

5

This exact combo of BrowserMobProxy + headless Chrome has burned me too. Headless mode is way stricter about certificate validation through proxies than regular Chrome.

You’re missing proper certificate handling. Add --ignore-ssl-errors=yes and --ignore-ssl-errors-list to your Chrome arguments. Also, ditch the mixed proxy config - you’re setting proxy through both WebDriver capabilities AND Chrome arguments, which creates conflicts.

Here’s what worked for me: remove ALL proxy settings from DesiredCapabilities and handle everything through Chrome arguments only. Keep your --proxy-server argument but add --ignore-certificate-errors-spki-list and --ignore-certificate-errors-whitelist.

For BrowserMobProxy, start it with different trust settings for headless mode. Set proxyInstance.setMitmDisabled(false) and make sure the CA certificate is configured properly.

That blank HTML? It’s SSL handshake failures, not actual page loading issues. Your proxy is rejecting connections because of certificate validation problems that headless mode doesn’t report clearly.