Testing internal site with authentication using headless browser

Other Questions Headless Browser
1

Hey guys, I’m trying to test a website on our internal network. The tricky part is it uses basic auth instead of a normal login page. I’ve been using URLs like https://user:[email protected] in my scripts.

I’m wondering if there’s a headless browser that can handle this type of URL. I’ve been using HtmlUnitDriver with Selenium, but it’s not working. Firefox Driver logs in fine, but HtmlUnitDriver gives me a 401 error.

Here’s what I’m doing:

HtmlUnitDriver tester = new HtmlUnitDriver();
String siteAddress = "https://" + username + ":" + password + "@" + testSite;
tester.get(siteAddress);
System.out.println(tester.getTitle());

Any ideas on how to make this work with a headless browser? Thanks!

2

I’ve been in your shoes, Noah. Headless browsers can be finicky with basic auth. Have you considered using Selenium with Chrome in headless mode? It’s been my go-to for internal site testing lately.

Here’s a snippet that’s worked wonders for me:

ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
WebDriver driver = new ChromeDriver(options);
driver.get("https://" + username + ":" + password + "@" + testSite);
System.out.println(driver.getTitle());

This approach has been rock-solid for me, handling basic auth without a hitch. Plus, it’s closer to real browser behavior, which can be crucial for certain tests. Give it a shot and see if it resolves your 401 issues.

3

I’ve encountered similar issues when testing internal sites with basic auth. While PhantomJS is an option, it’s no longer actively maintained. I’d recommend using Puppeteer with Chrome in headless mode instead. It handles basic auth seamlessly and provides a more modern, actively supported solution. Here’s a basic example:

const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage();
await page.authenticate({username: 'user', password: 'pass'});
await page.goto('https://site.com');
console.log(await page.title());
await browser.close();

This approach has worked well for me in similar scenarios, offering better performance and compatibility with modern web standards.

4

hey Noah, have u tried using PhantomJS? it’s a headless browser that supports basic auth. just set the auth header before making the request:

phantomjs.customHeaders = {‘Authorization’: 'Basic ’ + btoa(username + ‘:’ + password)}

might work better than HtmlUnitDriver for ur use case. lmk if u need more help!