Using XPath in Puppeteer's evaluate() function with Chrome headless

Other Questions Puppeteer
1

I’m trying to figure out how to use XPath expressions inside the page.evaluate() function when working with Puppeteer and Chrome headless. I know I can use $x() in Chrome dev tools, but it’s not working in this context.

Here’s what I’ve tried:

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com');

const result = await page.evaluate(() => {
  return $x('//div[@class=\"example\"]')[0].textContent;
});

console.log(result);

But this just causes the script to time out. Any ideas on how to make XPath work within page.evaluate()? I’m pretty new to Puppeteer, so I might be missing something obvious. Thanks for any help!

2

I’ve encountered this issue as well. The solution lies in utilizing Puppeteer’s built-in methods for XPath evaluation. Instead of using page.evaluate(), you can leverage page.$x() directly. This method returns an array of ElementHandles matching the XPath expression. Here’s how you could modify your code:

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com');

const elements = await page.$x('//div[@class=\"example\"]');
if (elements.length > 0) {
    const textContent = await elements[0].evaluate(el => el.textContent);
    console.log(textContent);
} else {
    console.log('Element not found');
}

This approach is more efficient and aligns with Puppeteer’s recommended practices for DOM manipulation.

3

I’ve been using Puppeteer for a while now, and XPath can be tricky. One approach that’s worked well for me is to inject a helper function into the page. You can do this before your evaluate call:

await page.evaluate(() => {
window.xpath = (xpath) => {
const result = document.evaluate(xpath, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null);
return result.singleNodeValue;
};
});

Then in your evaluate function, you can use it like this:

const result = await page.evaluate(() => {
const element = xpath(‘//div[@class=“example”]’);
return element ? element.textContent : null;
});

This method gives you the flexibility of XPath without the limitations of $x(). It’s been reliable for me across different projects. Just remember to handle cases where the element might not exist to avoid errors.

4

hey there! i’ve run into this before. the $x() function isn’t available in page.evaluate(). try using document.evaluate() instead. it’s a bit more verbose but works great for xpath in puppeteer. you’ll need to tweak your code a bit, but it should solve the timeout issue. good luck!