Extending navigation timeout in Puppeteer for Node.js

Other Questions Puppeteer
1

Hey everyone, I’m trying to scrape some data using Puppeteer in Node.js. I’ve run into a problem with really big web pages. They’re taking too long to load and I keep getting this annoying timeout error:

TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded

It happens when I use this code:

await page.goTo('https://example.com', { waitUntil: 'load' });

Does anyone know how to make Puppeteer wait longer? Or maybe there’s a way to just ignore the timeout completely? I’d really appreciate any help or tips on this. Thanks!

2

yo, i had the same issue! try using page.setDefaultTimeout(0) before ur goto. it’ll disable the timeout completely. just be careful cuz it might hang if the page never loads. also, check ur internet connection. slow internet can cause timeouts too. good luck!

3

I’ve dealt with similar timeout issues in my Puppeteer projects. One approach that’s worked well for me is combining a longer timeout with a custom function to check if the page has finished loading. Here’s a snippet I use:

const waitForNetworkIdle = async (page, timeout = 60000) => {
  await page.goto('https://example.com', {
    waitUntil: 'networkidle0',
    timeout: timeout
  });
  await page.waitForFunction(() => {
    return document.readyState === 'complete';
  }, { timeout: timeout });
};

This function waits for network idle and checks if the page is fully loaded. It’s been more reliable than just increasing the timeout. You might need to tweak the timeout value based on your specific use case. Hope this helps!

4

I’ve encountered this issue before when working with Puppeteer. The default navigation timeout is indeed 30 seconds, which can be insufficient for larger pages or slower connections. You can extend this timeout by modifying the page.setDefaultNavigationTimeout() method. Here’s how you can do it:

await page.setDefaultNavigationTimeout(60000);

This sets the timeout to 60 seconds (60000 milliseconds). You can adjust this value as needed. Place this line before your page.goto() call.

Alternatively, you can pass the timeout option directly in the goto() method:

await page.goto(‘https://example.com’, { waitUntil: ‘load’, timeout: 60000 });

Remember, while increasing the timeout can help, it’s also worth considering optimizing your scraping process or the website’s load time if possible. Excessive timeouts might indicate underlying performance issues.