Implementing XPath selectors within Puppeteer's evaluate function for headless browser automation

Other Questions Puppeteer
1

I’m working on a web scraping project using Puppeteer and need to execute XPath queries inside the evaluate method. When I try to run XPath expressions using the $x() function within browser.evaluate(), my script hangs and eventually times out.

The main issue seems to be that the context inside the evaluate function doesn’t have access to the same Chrome DevTools methods that work in the browser console. I can use $x() successfully when testing in Chrome’s developer tools, but the same approach fails when executed through Puppeteer’s page context.

const result = await browser.evaluate(() => {
    // This doesn't work - causes timeout
    return $x('//div[@class="content"]');
});

What’s the proper way to execute XPath selectors within Puppeteer’s evaluate function? Is there an alternative approach or do I need to include additional libraries in the page context?

2

The $x() function is a DevTools utility that won’t work in page.evaluate(). You’ve got two good options here. First, use the native document.evaluate() method that’s part of the standard DOM API:

const result = await page.evaluate(() => {
    const iterator = document.evaluate('//div[@class="content"]', document, null, XPathResult.ORDERED_NODE_ITERATOR_TYPE, null);
    const nodes = [];
    let node = iterator.iterateNext();
    while (node) {
        nodes.push(node);
        node = iterator.iterateNext();
    }
    return nodes;
});

Or just use Puppeteer’s page.$x() method directly without wrapping it in evaluate. This is usually cleaner for XPath queries. I’ve found it more reliable in production since it handles XPath execution outside the browser context entirely.

3

yea, puppeteer’s page.$x() is def the way to go. I wasted hours tryin to get $x() workin in evaluate b4 realizing it’s just a devtools thing. way simpler and u don’t gotta mess with document.evaluate syntax, which gets messy fast.

4

Had this exact problem last year. The $x() function doesn’t exist in page context when you use evaluate() - it’s only available in DevTools console. I switched to page.$x() instead of wrapping XPath in evaluate and it worked perfectly:

const elements = await page.$x('//div[@class="content"]');
// Now you can interact with the elements or extract data
const textContent = await Promise.all(
    elements.map(el => el.evaluate(node => node.textContent))
);

This completely fixes the timeout issues since you’re not trying to access methods that don’t exist. You can use the element handles for clicking, extracting text, whatever you need. Way cleaner than fighting with document.evaluate() syntax.