I need to change the document.referrer property before JavaScript runs on pages loaded through headless browser automation. The specific browser tool doesn’t matter much to me.
I’ve been testing different headless browsers but can’t get this working properly. With PhantomJS, even when I set the Referrer header, the document.referrer still shows up as an empty string instead of my custom value. I also tried Zombie.js but had similar issues.
Has anyone managed to successfully override document.referrer in headless browsers? I need this to happen before any page scripts execute so the website thinks it came from a specific referrer URL. Any working solution or alternative approach would be really helpful.
puppeteer is the way to go. try using page.setExtraHTTPHeaders() along with page.evaluateOnNewDocument() to set your custom referrer before any scripts kick in. had a similar issue and this worked for me.
Selenium WebDriver with Chrome handles this through CDP commands. I fought with this same issue for weeks before finding the solution - use execute_cdp_cmd to modify the referrer policy, then inject JavaScript before document ready. The key is combining Page.addScriptToEvaluateOnNewDocument with setting the referrer through navigation parameters. Way more reliable than those older tools you mentioned. The document.referrer sets correctly and sticks for the whole session. Just configure your CDP session before navigating to the target page.
Hit this same issue last year. Playwright crushes PhantomJS and Zombie for referrer stuff. Use the referer parameter in page.goto() plus set the HTTP header: await page.goto(url, { referer: 'https://example.com' }). Make sure your cookies and headers match what a real browser sends from that domain - some sites will catch you otherwise. This sets document.referrer properly and keeps it through the whole page lifecycle.