Web scraping with Puppeteer returns textContent null error intermittently

Other Questions Puppeteer
1

I’m building a stock price scraper using Puppeteer and Node.js that searches Google for stock information. Users input a stock symbol which gets added to a Google search URL, then I extract the price change data.

The problem is inconsistent behavior - sometimes it works perfectly, other times I get this error: Error: Evaluation failed: TypeError: Cannot read property 'textContent' of null.

I tried using waitForSelector but it times out. Also attempted waitUntil: "domcontentloaded" with no success.

My code checks three different CSS selectors since Google shows different elements for positive, negative, or neutral price changes:

const browser = await puppeteer.launch({ args: ["--no-sandbox"] });
const pageInstance = await browser.newPage();
const stockSymbol = parseStock(userInput);
const searchUrl = "https://www.google.com/search?q=" + stockSymbol.symbol;
await pageInstance.goto(searchUrl, { waitUntil: "networkidle2"});

// Check for negative change first
var priceChange = await pageInstance.$(
    "#knowledge-finance-wholepage__entity-summary > div > g-card-section > div > g-card-section > div.wGt0Bc > div:nth-child(1) > span.WlRRw.IsqQVc.fw-price-dn > span:nth-child(1)"
);
if (priceChange == null) {
  // Check for positive change
  priceChange = await pageInstance.$(
    "#knowledge-finance-wholepage__entity-summary > div > g-card-section > div > g-card-section > div.wGt0Bc > div:nth-child(1) > span.WlRRw.IsqQVc.fw-price-up > span:nth-child(1)"
  );
  if (priceChange == null) {
    // Check for no change
    priceChange = await pageInstance.$(
    "#knowledge-finance-wholepage__entity-summary > div > g-card-section > div > g-card-section > div.wGt0Bc > div:nth-child(1) > span.WlRRw.IsqQVc.fw-price-nc > span:nth-child(1)"
    );
}}

const changeText = await pageInstance.evaluate(
  (element) => element.textContent,
  priceChange
);

How can I make this more reliable?

2

Your evaluation runs even when all three selectors fail - that’s the problem. You need better error handling around the whole selector block. I’ve hit this same issue scraping dynamic content. page.waitForFunction() works way better than waitForSelector here. Try await pageInstance.waitForFunction(() => document.querySelector('your-selector-1') || document.querySelector('your-selector-2') || document.querySelector('your-selector-3')) before your existing code. This waits until at least one selector actually finds something. You could also move the textContent extraction inside the evaluation function and handle the null check in the browser context. Google’s finance widget loads async and the DOM isn’t always ready after networkidle2, especially for obscure stock symbols.

3

Google constantly changes their DOM structure and has anti-bot measures - that’s why you’re getting intermittent failures. Your main issue is passing a potentially null element to evaluate() without checking if any of your three selectors actually found something. Add this validation after trying all selectors: if (priceChange === null) { throw new Error(‘Price change element not found with any selector’); } I’d also add retry logic with exponential backoff. I’ve scraped financial data before and rotating user agents plus random delays between requests helped avoid detection. But honestly, Google’s finance structure is a nightmare for scraping - it’s super unstable. You might be better off switching to Yahoo Finance or Alpha Vantage APIs. Yeah, you’d need to tweak your workflow, but they’re way more reliable.

4

yeah, it can be frustrating! maybe try page.waitForTimeout(2000) before the selectors—sometimes content takes longer to load. also, think about using a stable stock price API instead of scraping Google, it could save u a lot of trouble in the long run.