Chrome headless driver shows blank pages when using BrowserMob proxy - need troubleshooting help

Other Questions Headless Browser
1

Chrome Headless Proxy Issue

I’m working on network traffic monitoring for my automated tests using BrowserMob proxy. Everything works great with regular Chrome browser but I’m getting empty pages when switching to headless mode.

Working Setup (Regular Chrome)

Proxy webDriverProxy = ClientUtil.createSeleniumProxy(mobProxy);
webDriverProxy.setNoProxy("<-loopback>");
webDriverProxy.setHttpProxy("localhost" + ":" + mobProxy.getPort());
webDriverProxy.setSslProxy("localhost" + ":" + mobProxy.getPort());

DesiredCapabilities caps = DesiredCapabilities.chrome();
caps.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.IGNORE);
caps.setCapability(CapabilityType.ACCEPT_SSL_CERTS, Boolean.TRUE);
caps.setAcceptInsecureCerts(Boolean.TRUE);

ChromeOptions chromeOpts = buildChromeOptions(config.getUserAgent(), Boolean.TRUE);
chromeOpts.setCapability(CapabilityType.PROXY, webDriverProxy);
caps.setCapability(ChromeOptions.CAPABILITY, chromeOpts);

return new RemoteWebDriver(serverUrl, caps);

Broken Setup (Headless Chrome)

ChromeOptions headlessOpts = new ChromeOptions();
headlessOpts.addArguments("--user-agent=" + "HeadlessChrome");
headlessOpts.setAcceptInsecureCerts(Boolean.TRUE);
headlessOpts.setHeadless(true);
headlessOpts.addArguments("--allow-insecure-localhost", "--no-sandbox", "--disable-extensions", "--window-size=1920,1080");

List<String> proxyArgs = ImmutableList.of("--ignore-certificate-errors", "--proxy-bypass-list=<-loopback>",
    "--proxy-server=http://localhost:" + proxyInstance.getPort(), "--remote-debugging-port=9222");
headlessOpts.addArguments("--ssl-protocol=any");
headlessOpts.addArguments("--allow-running-insecure-content");
headlessOpts.addArguments(proxyArgs);

capabilities.setCapability(ChromeOptions.CAPABILITY, headlessOpts);
webDriver = new RemoteWebDriver(remoteUrl, capabilities);

Current Configuration

The Problem

Pages load as completely blank with this HTML:

<html><head></head><body></body></html>

I’ve tried adding waits and delays but nothing helps. When I use --proxy-bypass-list=* the pages load fine but then I can’t capture network traffic.

Environment Details

  • browsermob-proxy version: 2.1.5
  • ChromeDriver: 92.0.4515.159
  • Running on remote Linux server

Proxy Initialization

System.setProperty("bmp.allowNativeDnsFallback", "true");
BrowserMobProxy proxyInstance = new BrowserMobProxyServer();
proxyInstance.setTrustAllServers(Boolean.TRUE);

I’ve been stuck on this for days and really need to get it working soon. Has anyone faced similar issues with headless Chrome and proxy setup?

2

Classic headless Chrome proxy issue. Headless mode has stricter security by default, and you’re missing key arguments. Add --disable-web-security and --disable-features=VizDisplayCompositor to your Chrome options. Try changing --proxy-server=http://localhost: to just --proxy-server=localhost: - drop the http prefix. You’re setting the proxy in both Chrome arguments AND capabilities, which causes conflicts. Pick one - I’d go with Chrome arguments for headless. Also add --disable-gpu and --no-first-run - these helped me. The remote debugging port can mess with proxy traffic too, so remove it unless you actually need it for debugging.

3

Had the exact same issue with BrowserMob proxy and headless Chrome last year. It’s usually the proxy config and SSL certificate handling that’s messed up in headless mode. Start your BrowserMob proxy with explicit binding to all interfaces - don’t just use localhost. Set the proxy before you start HAR recording too. Headless Chrome handles certificate validation differently than regular Chrome, so add --ignore-ssl-errors=yes and --ignore-ssl-errors-list to your Chrome options. Try proxyInstance.setConnectTimeout(30, TimeUnit.SECONDS) on your proxy instance - the default timeout’s often too short for headless environments. Check if your remote Linux server has proper DNS resolution. I’ve seen headless Chrome fail to resolve hostnames through the proxy while regular Chrome works fine.

4

had this same issue a few months ago. it’s usually the ssl handshake failing silently when running headless. Add --disable-dev-shm-usage and --disable-background-timer-throttling to your arguments - that’s what fixed the blank pages for me. and make sure you start the proxy before creating the webdriver instance.