Dynamic content missing when scraping pages with Puppeteer on Linux headless mode

Other Questions Puppeteer
1

I’m having trouble with web scraping using Puppeteer on Linux systems. When I run my script in headless mode on Linux, the scraped content is incomplete - it only captures the basic HTML structure but misses all the dynamic elements that get loaded by JavaScript, web fonts, and images.

This issue is really weird because it works perfectly fine on Windows (both headless and headfull modes) and also works when I run it on Linux with headfull mode. I’ve been debugging this for hours and noticed that when running headless on Linux, there are fewer network requests being made compared to other configurations. I even tried adding a stealth plugin to avoid detection but that didn’t help.

Here’s the code I’m working with:

const antiDetect = StealthPlugin();
puppeteer.use(antiDetect);

const browserInstance = await puppeteer.launch({
  headless: true,
  args: [
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--font-render-hinting=none",
    "--disable-gpu",
    "--no-first-run",
  ],
});

const webPage = await browserInstance.newPage();

webPage.setDefaultNavigationTimeout(30 * 1000);
let response = await webPage.goto(settings.targetUrl, { waitUntil: "networkidle0" });

let htmlContent = await webPage.content();
fs.writeFileSync("scraped.html", htmlContent);

Has anyone else experienced this Linux-specific issue with headless mode? Any suggestions on how to fix this would be greatly appreciated.

2

Had the same issue about a year ago when I moved my scrapers to production Linux servers. Turns out Chrome was missing system dependencies it needs for headless rendering. Puppeteer launches fine, but some JavaScript just fails silently.

Install these packages: libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxi6 libxtst6 libnss3 libxss1 libgconf-2-4 libxrandr2 libasound2 libatk1.0-0 libdrm2 libxkbcommon0 libgtk-3-0. Also throw --disable-dev-shm-usage in your launch args since the default shared memory is usually too small in containers.

One more thing - set your viewport size and user agent to match what you’re getting on Windows. The responsive design can trigger differently on headless Linux and break certain JavaScript modules.

3

check your display settings - headless linux often needs --disable-extensions and --disable-background-timer-throttling flags. try setting page.setViewport({width: 1920, height: 1080}) before you navigate. i’ve seen js frameworks completely skip rendering when there’s no display detected.

4

This is usually a timing issue, not missing dependencies. networkidle0 fires too early on headless Linux because JavaScript behaves differently without a display server. I’ve had way better luck with explicit waits than network idle states. Try await webPage.waitForSelector('specific-dynamic-element') instead, or just add a fixed delay before extracting content. You can catch JavaScript errors with webPage.on('console', msg => console.log(msg.text())) to see if something’s breaking the dynamic content loading. I also had success enabling request interception to log what’s actually getting blocked or failing. Sometimes headless Linux environments give different response codes for certain resources.