Puppeteer-Core and @sparticuz/chromium Bundle: Issues with Dynamic Script and Element Detection

Other Questions Puppeteer
1

My Problem:

I switched from regular Puppeteer to Puppeteer-Core combined with @sparticuz/chromium because of memory issues. I need to verify if a script loads dynamically and if a specific UI component appears on web pages.

My Current Approach:

  1. Block image loading to speed things up
  2. Get all script URLs from the page to verify script installation
  3. Look for a specific UI component (.my-widget-element)
  4. Changed from ‘load’ to ‘networkidle2’ wait condition for better dynamic content detection

The Problem:
• Script detection works fine but UI element detection is unreliable
• UI elements seem to need more time to load even with 30 second timeout
• Right now I navigate to the same page twice (first with ‘load’ then ‘networkidle2’) which feels wrong

My Code:

const puppeteer = require("puppeteer-core");
const chromium = require("@sparticuz/chromium");

const WIDGET_SCRIPT_URL = "https://cdn.example.com/widget/js/"; // Sample URL

class DynamicContentDetector {
  async disableImageLoading(page) {
    await page.setRequestInterception(true);
    page.on("request", (req) => {
      if (req.resourceType() === "image") {
        req.abort();
      } else {
        req.continue();
      }
    });
  }

  checkForWidgetScript(scriptList, widgetCode) {
    let widgetScriptUrl = WIDGET_SCRIPT_URL + widgetCode;
    return scriptList.some((scriptSrc) => scriptSrc && scriptSrc.includes(widgetScriptUrl));
  }

  async extractScriptSources(page) {
    return await page.evaluate(() => {
      return Array.from(document.querySelectorAll("script"))
        .map((script) => script.getAttribute("src"))
        .filter(Boolean);
    });
  }

  async checkWidgetInstallation(siteUrl, widgetCode) {
    let scriptFound = false;
    let elementFound = false;
    let browser;

    try {
      browser = await puppeteer.launch({
        args: [...chromium.args, "--no-sandbox", "--disable-setuid-sandbox"],
        executablePath: await chromium.executablePath(),
        headless: true,
        ignoreHTTPSErrors: true,
      });
      console.log("Browser started");

      const page = await browser.newPage();
      await this.disableImageLoading(page);

      await page.goto(siteUrl, { waitUntil: "load" });
      console.log("Page loaded");

      const scriptSources = await this.extractScriptSources(page);
      scriptFound = this.checkForWidgetScript(scriptSources, widgetCode);
      console.log("Script found:", scriptFound);

      await page.goto(siteUrl, { waitUntil: "networkidle2" });
      console.log("Switched to networkidle2 for element detection");

      try {
        await page.waitForSelector(".my-widget-element", { timeout: 30000 });
        elementFound = true;
        console.log("Widget element found:", elementFound);
      } catch (error) {
        console.log("Widget element not detected within timeout");
      }

      await browser.close();
    } catch (error) {
      console.error("Detection error:", error);
      if (browser) await browser.close();
    }

    return { scriptFound, elementFound };
  }
}

Is there a better way to handle dynamic content detection without navigating twice?

2

You’re overcomplicating this - don’t navigate twice. Just use networkidle2 from the start since it waits for network activity to settle, which handles both script loading and initial rendering. The real problem is that modern widgets often use lazy initialization or wait for specific events before they actually render DOM elements. After your page loads with networkidle2, try adding page.evaluate(() => window.dispatchEvent(new Event('resize'))) to trigger any resize-based widget initializations. Also, check if the widget script creates any global variables or functions that show it’s actually initialized - don’t just rely on DOM elements appearing. Some widgets only render when they’re in viewport or after user interactions.

3

I’ve hit this same issue with dynamic widgets. Waiting for network idle isn’t enough - most modern widgets use async initialization that happens after all network requests finish.

Skip the double navigation. Instead, set up a MutationObserver right after your initial page load with networkidle2. Have it watch for when your widget element actually gets added to the DOM, then wait for either the element or your timeout.

What works even better for me is checking if the widget’s JavaScript API is available instead of just looking for DOM elements. Most widgets expose global functions or objects once they’re fully loaded. You can poll for these with page.evaluate() - way more reliable than watching for DOM changes.

4

Try page.waitForLoadState('networkidle') instead of navigating twice - it waits for network activity to settle. Also throw in a small delay after networkidle before checking for elements. JS frameworks sometimes need extra time to render components even when the network’s quiet.