Extracting Text from DOM Elements Using Puppeteer with Headless Chrome

Other Questions Puppeteer
1

I am attempting to utilize Puppeteer and headless Chrome to execute our JavaScript test cases, but I’m struggling to retrieve the information from the webpage. I found a reference suggesting the use of page.evaluate(), which includes an illustrative example that seems relevant to my requirements.

const mainHandle = await page.$('main');
const content = await page.evaluate(main => main.textContent, mainHandle);
await mainHandle.dispose();

As a complete example, I converted this to a script to fetch my username from my Stack Overflow profile. Since our project is based on Node 6, I modified the await usage to incorporate .then().

const puppeteer = require('puppeteer');

puppeteer.launch().then(function(browser) {
    browser.newPage().then(function(page) {
        page.goto('https://stackoverflow.com/users/4794').then(function() {
            page.$('h2.profile-name').then(function(nameHandle) {
                page.evaluate(function(name) {
                    return name.textContent;
                }, nameHandle).then(function(userName) {
                    console.log(userName);
                    browser.close();
                }, function(err) {
                    console.error(err);
                    browser.close();
                });
            });
        });
    });
});

However, executing this results in the following error message:

$ node get_user_info.js 
TypeError: Converting circular structure to JSON
    at Object.stringify (...)
    at ...

It seems that the issue arises when trying to serialize the argument passed to page.evaluate(). While I can send strings and numbers, element handles appear to cause a problem. Is there an error in the example provided, or is it specific to Node 6? How can I correctly extract the text from a DOM element?

2

hey! you may need to ensure the element exists first. Sometimes elements can take longer to load. Try adding a waitfor method, like await page.waitForSelector('h2.profile-name') before querying the element. This can help ensure the right element is captured.