Extract text content from div element using Puppeteer

[JollyMusic3]

I’m trying to scrape text from a div element but keep running into issues. Here’s my current code:

const puppeteer = require('puppeteer');

(async function() {
    try {
        const browser = await puppeteer.launch({headless: false});
        const webpage = await browser.newPage();
        await webpage.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36');

        await webpage.goto('https://example-ranking-site.com/apps/category/games', {waitUntil: 'networkidle2'});
        await webpage.waitForSelector('.main-content');
        
        const freeAppsLink = await webpage.$('[data-filter="free-apps"]');
        await freeAppsLink.click();
        
        await webpage.waitForSelector('.app-list-table');
        const tableRows = await webpage.$$('.app-list-table tbody tr td');
        
        for (const cell of tableRows) {
            const titleElement = await cell.$('div.app-title');
            const appTitle = await webpage.evaluate(el => el.textContent, titleElement);
            console.log('App Title:', appTitle);
        }
        
        console.log('Scraping completed');
        
    } catch (error) {
        console.log('Error occurred:', error);
    }
})();

I keep getting the error TypeError: Cannot read property 'textContent' of null when trying to extract text from <div class="app-title">. I’ve tried different approaches but nothing seems to work. What’s the proper way to get text content from div elements in Puppeteer?

[olivias]

This is super common with dynamic content scraping. Beyond ameliat’s null check suggestion, try waitForFunction instead of just waitForSelector. Sometimes the selector exists but the content isn’t actually loaded yet.

Add this after your click:

await webpage.waitForFunction(() => {
    const rows = document.querySelectorAll('.app-list-table tbody tr td');
    return rows.length > 0 && rows[0].querySelector('div.app-title');
});

Also, $$eval works better for bulk extraction:

const appTitles = await webpage.$$eval('.app-list-table tbody tr td div.app-title', elements => 
    elements.map(el => el.textContent.trim())
);

This approach is way more reliable with tables since it grabs all matching elements at once instead of looping through each cell.

[Bob_Clever]

Looks like a timing issue. Add a delay after clicking before you search for elements - Puppeteer often moves faster than the DOM updates. Try await webpage.waitForTimeout(2000); right after the click. Also double-check your selector against the actual page source first.

[ameliat]

You’re getting that null error because some table cells don’t have the div.app-title element. Just add a null check before trying to access textContent:

for (const cell of tableRows) {
    const titleElement = await cell.$('div.app-title');
    if (titleElement) {
        const appTitle = await webpage.evaluate(el => el.textContent, titleElement);
        console.log('App Title:', appTitle);
    }
}

Or you can use $eval with a try-catch, which is cleaner:

for (const cell of tableRows) {
    try {
        const appTitle = await cell.$eval('div.app-title', el => el.textContent);
        console.log('App Title:', appTitle);
    } catch (e) {
        // Element not found in this cell, skip
    }
}

I’ve hit this same issue scraping dynamic pages. The second approach is better since it combines selection and evaluation in one step.