How to capture all browser network activity including requests and responses with Puppeteer?

Other Questions Puppeteer
1

I’m working on a project where I need to collect clickable items from a webpage and simulate clicks on them to track redirections and capture the final destination URLs. My current approach works but I’m missing some network traffic that happens before I can set up the request listeners.

const newTab = await browserInstance.newPage();
const targetUrl = "example-website-url";

await newTab.goto(targetUrl, {
    waitUntil: 'networkidle2',
    timeout: 25000
});

// Find clickable items on page
let linkElements, clickButtons, allClickables;
linkElements = await newTab.$$('a[href]');
clickButtons = await newTab.$$('input[type="submit"], button');
allClickables = linkElements.concat(clickButtons);

// Process each clickable item
for (let index = allClickables.length - 1; index >= 0; index--) {
    // Listen for new tab creation
    const tabCreationPromise = new Promise(resolve => 
        browserInstance.once('targetcreated', target => resolve(target.page()))
    );
    
    allClickables[index].click({button: 'middle'});
    
    // Get the newly created tab
    const createdTab = await tabCreationPromise;
    if(createdTab) {
        createdTab.on('request', networkRequest => {
            console.log(`Captured URL: ${networkRequest.url}`);
        });
    }
}

The problem is that initial redirects occur before I can attach the request listener using the page.on('request') method. Is there a way to set up global network monitoring for all new pages created by the browser instance? I need to catch all network activity from the very beginning.

2

You’ve hit a classic timing issue - network activity starts before your listener attaches. Fix this by monitoring at the browser context level instead of individual pages. Set up a CDP session on your browser instance and enable the network domain globally:

const client = await browserInstance.target().createCDPSession();
await client.send('Network.enable');

client.on('Network.requestWillBeSent', (params) => {
    console.log(`Global request: ${params.request.url}`);
});

client.on('Network.responseReceived', (params) => {
    console.log(`Global response: ${params.response.url}`);
});

This catches everything - initial navigation requests, redirects during page creation, all of it. I’ve used this on similar scraping projects and it reliably grabs traffic that page-level listeners miss. The CDP session sticks around for all pages from that browser instance, so you won’t lose any early network activity.

3

you can also try page.setRequestInterception(true) before navigating - this makes puppeteer intercept requests right from the start. pair it with page.on('response') to catch both sides of network activity. i used this when cdP was too much for basic redirect tracking.