Web scraping issue: Empty objects returned from Puppeteer query

Other Questions Puppeteer
1

I’m new to web scraping and I’m having trouble with Puppeteer. I’m trying to get a list of elements from a website, but my code is giving me an array of empty objects instead of the actual data.

Here’s what I’m doing:

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');

const elementList = await page.evaluate(() => {
  return Array.from(document.querySelectorAll('.content-item'));
});

console.log(JSON.stringify(elementList));

await browser.close();

When I run this, I get something like:

[{},{},{},{},{},{}]

Instead of the actual content I’m looking for. What am I doing wrong here? Is there a problem with how I’m using querySelectorAll or evaluate? Any help would be really appreciated!

2

hey there! i’ve dealt with this before. the problem is puppeteer can’t serialize DOM elements directly. try extracting specific data like this:

const elementList = await page.evaluate(() => {
return Array.from(document.querySelectorAll(‘.content-item’)).map(el => ({
innerText: el.innerText,
attributes: Object.fromEntries([…el.attributes].map(attr => [attr.name, attr.value]))
}));
});

this should give u the text and attributes for each element. goodluck!

3

I encountered a similar issue when I first started using Puppeteer. The problem is likely due to how JavaScript handles DOM elements during serialization. When you return DOM elements directly from page.evaluate(), they’re stripped down to empty objects.

To fix this, you need to extract the specific data you want from the elements before returning them. Here’s how I modified my code to make it work:

const elementList = await page.evaluate(() => {
  return Array.from(document.querySelectorAll('.content-item')).map(el => ({
    text: el.textContent,
    href: el.querySelector('a')?.href,
    // Add more properties as needed
  }));
});

This approach extracts the text content and any links from each element, which you can then use in your script. Remember to adjust the properties based on what data you actually need from the page. Hope this helps solve your issue!

4

The issue you’re facing is quite common when working with Puppeteer. The problem stems from the fact that DOM elements can’t be directly serialized and passed between contexts. To resolve this, you need to extract the specific data you want from the elements before returning them.

Try modifying your code like this:

const elementList = await page.evaluate(() => {
  return Array.from(document.querySelectorAll('.content-item')).map(el => ({
    text: el.textContent.trim(),
    className: el.className,
    id: el.id
  }));
});

This approach will return an array of objects containing the text content, class name, and ID of each element. Adjust the properties based on what data you actually need. Also, ensure that ‘.content-item’ is the correct selector for the elements you’re targeting on the page.