Troubleshooting Puppeteer Timeout Issue with Selector Wait

Other Questions Puppeteer
1

I’m stuck with a Puppeteer problem in my Node.js project. The script keeps timing out when I use waitForSelector. Here’s what I’ve tried:

const puppeteer = require('puppeteer');

async function scrapeData() {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();
  
  const targetUrl = 'https://example.com/jobs';
  const rowSelector = '#jobTable tr.job-listing';
  
  await page.goto(targetUrl);
  await page.waitForSelector(rowSelector, { timeout: 60000 });
  
  const jobListings = await page.$$eval(rowSelector, rows => rows.map(row => row.textContent));
  
  console.log(jobListings);
  await browser.close();
}

scrapeData().catch(console.error);

I’ve increased the timeout, but it didn’t help. Taking a screenshot before waitForSelector shows only the page header. The same code works fine on other sites. Any ideas what could be causing this? Maybe it’s a dynamic content issue?

2

Have you considered the possibility of JavaScript rendering on the client side? Some websites use frameworks like React or Vue.js that dynamically populate content after the initial page load. In such cases, Puppeteer might not detect the elements immediately.

One approach you could try is to wait for a network request that likely triggers the content load. For example:

await page.goto(targetUrl);
await page.waitForResponse(response => response.url().includes(‘api/jobs’));
await page.waitForSelector(rowSelector, { timeout: 60000 });

This assumes there’s an API call to fetch job listings. You’d need to adjust the URL based on the actual network requests you observe.

Also, ensure you’re not being blocked by any anti-bot measures. Some sites implement sophisticated detection methods that could interfere with scraping attempts.

3

I’ve encountered similar issues with Puppeteer before. One thing that often helps is setting a custom user agent. Some sites might be blocking or behaving differently for headless browsers. Try this:

await page.setUserAgent(‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36’);

await page.goto(targetUrl);

Also, have you checked if there are any iframes on the page? If the content is inside an iframe, you might need to switch to it first:

const frame = page.frames().find(frame => frame.name() === ‘contentFrame’);
await frame.waitForSelector(rowSelector, { timeout: 60000 });

Lastly, some sites use lazy loading. You might need to scroll the page to trigger content loading:

await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));

Hope one of these suggestions helps!

4

sounds like dynamic content loading. try adding a delay before waitForSelector:\n\nawait page.goto(targetUrl);\nawait page.waitForTimeout(5000); // wait 5 seconds\nawait page.waitForSelector(rowSelector, { timeout: 60000 });\n\nif that doesnt work, the site might be using ajax to load content. you could try waiting for network idle instead:\n\nawait page.goto(targetUrl, {waitUntil: ‘networkidle0’});