Getting return values from page.evaluate() function in Puppeteer

Other Questions Puppeteer
1

I’m working on a web scraper using Puppeteer and I’m having trouble extracting data from the page.evaluate() method. I can see the results in the browser console but I can’t access them in my main script. Here’s what I’m trying to do:

let videoElements = []
const fetchContent = async() => {
    return await page.evaluate(async () => {
        console.log(await new Promise(resolve => {
            var currentScroll = 0
            var stepSize = 150
            var scrollTimer = setInterval(() => {
                items = document.querySelectorAll("div.video-container .video-item")
                console.log(`Found ${items.length} items`)
                var totalHeight = document.body.scrollHeight
                window.scrollBy(0, stepSize)
                currentScroll += stepSize
                if(currentScroll >= totalHeight || items.length >= 100){
                    clearInterval(scrollTimer)
                    resolve(Array.from(items))
                }
            }, 600)
        }))
    })
}
videoElements = await fetchContent()
console.log(videoElements)

The console.log inside the promise shows the array correctly in the browser console. However, when I try to assign the result to videoElements variable, it comes back undefined. What am I doing wrong here? Any help would be great.

2

I hit this exact issue a few months ago. Your page.evaluate() isn’t returning anything - that’s the problem. You’re logging the resolved promise inside page.evaluate(), but you need to actually return the result to get it back in your main script. Here’s the fix: javascript const fetchContent = async() => { return await page.evaluate(async () => { const result = await new Promise(resolve => { var currentScroll = 0 var stepSize = 150 var scrollTimer = setInterval(() => { items = document.querySelectorAll("div.video-container .video-item") var totalHeight = document.body.scrollHeight window.scrollBy(0, stepSize) currentScroll += stepSize if(currentScroll >= totalHeight || items.length >= 100){ clearInterval(scrollTimer) resolve(Array.from(items)) } }, 600) }) return result; // This is what you're missing }) } console.log() doesn’t return the value it logs, so your evaluate function was returning undefined. Add that explicit return statement and you’re good to go.

3

Yeah, that’s a classic Puppeteer gotcha. You’re missing the return statement after your promise resolves. The console.log won’t send anything back to Node.js - you’ve got to explicitly return the data. Store your result first, then return it: const elements = await new Promise(...); return elements; at the end of your evaluate function.

4

You’re not returning the promise result from page.evaluate(). You’re console.logging the value but that doesn’t send it back to Node.js. page.evaluate() only returns serializable data through explicit returns - without one, you get undefined. Capture your promise result in a variable first: const items = await new Promise(resolve => { ... }); then return items; at the end. I hit this exact same issue when I started with Puppeteer. The browser and Node contexts are totally separate, so you’ve got to explicitly return anything you want back.