Fetch request remains unresolved when used in Puppeteer's page context

[Isaac_Cosmos]

I’m tasked with scraping a website where I need to log in and then perform an API call while within the page context. However, my fetch request doesn’t seem to finish and remains in a pending state.

Here’s my approach:

1. Navigate to the login URL
2. Input login details and submit the form
3. Wait for the page to redirect to the dashboard
4. Perform a POST request using fetch within the page context

Here’s my current implementation:

await browser.goto('https://example-login.com/signin')
await Promise.all([
    browser.evaluate(() => {
        (function() {
            const setInputValue = Object.getOwnPropertyDescriptor(
                window.HTMLInputElement.prototype,
                "value").set;
            const fillField = (field, value) => {
                const fieldElement = document.querySelector(`[name="${field}"]`)
                setInputValue.call(fieldElement, value)
                fieldElement.dispatchEvent(new Event('change', { bubbles: true}))
            };
            fillField('email', "[email protected]");
            fillField('pass', "mypassword123");
            document.getElementById("login-btn").click();
        })()
    }),
    browser.waitForNavigation({waitUntil: 'domcontentloaded'}),
]);

await browser.evaluate(() => {
    fetch('https://api.example.com/data', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify(this.requestPayload)
    }).then(response => {
        return response.json();
    }).then(data => {
        console.log(data);
    })
})

The issue: The fetch call does not resolve, causing the script to hang indefinitely. It seems related to how promises function within page.evaluate(). Any suggestions on resolving this?

[sophiac]

The problem is that page.evaluate() doesn’t wait for your fetch call to finish. It completes while the fetch is still running. To fix this, return the fetch promise from inside evaluate. Also, don’t use this.requestPayload - pass it as an argument instead. Try await page.evaluate((payload) => { return fetch(...).then(...) }, requestPayload). This makes Puppeteer wait for the fetch to complete before moving on.

[Bob_Clever]

your fetch prob isn’t resolving due to how page.evaluate() handles promises. try using an async function or consider page.evaluateHandle(). also, this.requestPayload won’t work there; you’ll need to pass it in as an arg.

[miat]

I experienced the same issue with authenticated endpoints. The root of the problem is that your evaluate function isn’t waiting for the fetch call to complete, which leads to the script hanging. To resolve this, make your evaluate function asynchronous and return the fetch promise. For example, wrap the code in an async function and use await with the fetch call. Also, remember that this.requestPayload won’t work, as this is not defined in the page context. Instead, pass your payload directly as an argument. The modified code should look something like await browser.evaluate(async (payload) => { const response = await fetch(...); return response.json(); }, yourPayload). This way, Puppeteer will wait for the fetch to finish before executing the next steps.