I’m having trouble accessing complete HTTP headers while testing with puppeteer. When I intercept requests in my test, I only see basic headers like referrer and user-agent. However, when I check the same requests using Chrome developer tools, there are many additional headers including Cookie and others.
Here’s my test code:
const chromeBrowser = await puppeteer.launch();
const testPage = await chromeBrowser.newPage();
testPage.on('request', function(interceptedRequest) {
console.log(JSON.stringify(interceptedRequest.headers, null, 2));
});
await testPage.goto('https://example.com/', {waitUntil: 'networkidle'});
await chromeBrowser.close();
Are there specific limitations on which HTTP headers can be accessed through puppeteer? Is this a Chrome browser restriction or something specific to the puppeteer library itself? Any help would be appreciated!
This is actually a known behavior with Puppeteer’s request interception. The issue occurs because Puppeteer intercepts requests at the network layer before Chrome adds certain headers automatically. Headers like Cookie, Authorization, and some others are added by Chrome’s internal mechanisms after the interception point. I’ve encountered this same problem in my automation projects. What worked for me was switching to using the Chrome DevTools Protocol directly through page.on('response') instead of request interception when I needed complete header information. You can also try enabling request interception with page.setRequestInterception(true) which gives you access to more headers, though it comes with performance overhead. Another approach is to use page.evaluateOnNewDocument() to inject JavaScript that captures headers client-side if you specifically need the complete set that Chrome normally sends.
I encountered this exact limitation during load testing scenarios where header analysis was critical. The fundamental issue is architectural - Puppeteer’s request interception occurs before Chrome’s header enrichment phase completes. What I found effective was using page.on('requestfinished') events combined with accessing the underlying CDP session. This approach captures requests after Chrome has fully processed them, including all automatically added headers. You can access the complete header set by calling request.response().headers() on finished requests rather than intercepting them early. Another technique that worked well was enabling both Network domain logging and using page.setRequestInterception(false) to avoid interfering with Chrome’s natural request flow. The trade-off is you lose the ability to modify requests, but gain complete visibility into actual transmitted headers. This method revealed headers that were completely invisible during standard request interception, particularly authentication and session-related headers that Chrome manages internally.
yea this is super annoying lol. puppeteer catches requests before chrome adds extra headers. try switching to ‘response’ events and set request interception to false. i had the same issue and this helped me get all the headers including cookies.
I ran into this exact scenario while building automated tests for a financial application. The root cause is timing - Puppeteer captures the request before Chrome’s networking stack has fully processed it. What solved this for me was using the Network domain events through CDP (Chrome DevTools Protocol) instead of the basic request listener. Try replacing your current approach with await testPage._client.send('Network.enable') followed by listening to Network.requestWillBeSent events. This gives you access to the complete header set including cookies and authentication headers that Chrome adds during its request processing phase. The headers object in these events contains everything you’d see in DevTools. One gotcha though - this method requires you to work with the underlying CDP client rather than Puppeteer’s abstracted API, so the code becomes slightly more complex but you get complete header visibility.
The behavior you’re experiencing stems from how Puppeteer hooks into Chrome’s request pipeline. During my work on API testing automation, I discovered that certain headers are genuinely unavailable at the interception stage because they haven’t been computed yet. Chrome dynamically generates headers like Accept-Encoding and Cookie based on browser state that occurs after Puppeteer’s interception point. A workaround that proved reliable was combining both request and response monitoring - use request events for outgoing headers you control, then capture the actual sent headers from response events using page.on('response', response => response.request().headers()). This approach gives you visibility into what was actually transmitted. Additionally, if you need to verify specific header behavior, consider using page.setExtraHTTPHeaders() to explicitly set headers you want to test, ensuring they appear in your interception logic. The limitation exists because Puppeteer operates at a lower level than the complete browser networking stack.