Web Scraping Element Selection - Need Help Clicking Specific HTML Element

Other Questions Puppeteer
1

I’m trying to automate clicking on a specific link element that should take me to a documentation section, but my current selector approach isn’t working properly.

Here’s what I have so far:

const webdriver = require('puppeteer');
const targetUrl = process.argv[2];
if (!targetUrl) {
    throw "URL argument is required";
}
async function execute() {
    const browserInstance = await webdriver.launch({
        headless: false,
        defaultViewport: null,
        slowMo: 15,
        args: ['--start-maximized', '--disable-notifications']
    });
    const currentPage = await browserInstance.newPage();
    await currentPage.goto(targetUrl);

    await currentPage.waitForSelector(".main-nav .menu-wrapper .nav-item.fourth a", {
        visible: true
    });
    await currentPage.click(".main-nav .menu-wrapper .nav-item.fourth a");

    await currentPage.waitForSelector(".header-section .action-buttons ", {
        visible: true
    });
    const linkElement = await currentPage.$$('.header-section .action-buttons a');
    await linkElement[0].click();

    await currentPage.waitForSelector(".content-grid", {
        visible: true
    });
    linkElement = await currentPage.$$('.content-grid')[0].$$('tbody')[0].$$('tr')[0].$$('td')[0].$$('p')[1];
    await linkElement.click();

    await currentPage.pdf({path: 'output.pdf', format:"A4"});

    console.log("Process completed");
    browserInstance.close();
}
execute();

The main issue is with this specific line: linkElement = await currentPage.$$('.content-grid')[0].$$('tbody')[0].$$('tr')[0].$$('td')[0].$$('p')[1];

What would be a better way to target and click this nested element? Any suggestions for a more reliable selector approach?

2

Your selector chain breaks because you can’t chain $$() calls like that - $$() returns an array, not a single element that supports more $$() calls. When you do currentPage.$$('.content-grid')[0].$$('tbody'), you’re trying to call $$() on an ElementHandle, which won’t work.

Try $eval() or evaluate() to work in the page context instead:

const targetElement = await currentPage.evaluate(() => {
  const grid = document.querySelector('.content-grid');
  return grid.querySelector('tbody tr td p:nth-child(2)');
});

Or just use one CSS selector: .content-grid tbody tr:first-child td:first-child p:nth-child(2) and call currentPage.click() directly on it. Way cleaner and easier to debug.

3

Had the same problem scraping dynamic sites. Your chained selectors break because each $$() returns an array, not something you can chain. Plus you’re building selectors that’ll fall apart if the DOM changes even slightly.

I’ve had way better luck with page.waitForFunction() - wait for the exact element you want, then grab it with one clean selector:

await currentPage.waitForFunction(() => {
  const targetP = document.querySelector('.content-grid tbody tr td p:nth-child(2)');
  return targetP && targetP.offsetParent !== null;
});
await currentPage.click('.content-grid tbody tr td p:nth-child(2)');

This actually waits until the element exists and is visible before clicking. Way more reliable than chaining selectors or using setTimeout, especially with async-loading sites.

4

That selector chain is a nightmare to maintain. Every time the DOM changes, you’ll be back fixing selectors.

I’ve hit this same issue at work. Instead of fighting brittle CSS selectors and custom Puppeteer code, I switched to Latenode for web scraping.

Latenode has visual element selection - you literally click what you want in their interface and it handles the selector logic. No more guessing with nested selectors or debugging xpath.

When the site structure changes, you just update visually instead of diving back into code. Saved me hours on maintenance.

For your case, Latenode would handle the navigation, clicking, and PDF generation without any Puppeteer code. Way more reliable than hardcoded selectors.

5

that nested selector chain’s pretty fragile. I’d go with xpath like //div[@class='content-grid']//p[2] or mix CSS with nth-of-type. you’re also turning linkElement into an array then calling .click() on it - that’s gonna break things.