Getting 'Target closed' protocol error when using Puppeteer page navigation

Other Questions Puppeteer
1

I’m working on a web scraping project using Puppeteer in Node.js with worker processes to capture website screenshots. The setup works most of the time, but I keep running into protocol errors with certain websites that seem to load fine in regular browsers.

const cluster = require('cluster');
const express = require('express');
const bodyParser = require('body-parser');
const puppeteer = require('puppeteer');

async function captureWebsiteImage(url) {
    let imageData;
    const browserInstance = await puppeteer.launch({ 
        args: ['--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage'] 
    });
    const newPage = await browserInstance.newPage();

    try {
        await newPage.goto('http://' + url + '/', { timeout: 60000, waitUntil: 'networkidle2' });
    } catch (err) {
        try {
            await newPage.goto('http://' + url + '/', { timeout: 120000, waitUntil: 'networkidle2' });
            imageData = await newPage.screenshot({ type: 'png', encoding: 'base64' });
        } catch (err) {
            console.error('Failed to load: ' + url + ' error: ' + err);
        }
    }

    await newPage.close();
    await browserInstance.close();

    return imageData;
}

if (cluster.isMaster) {
    const workerCount = require('os').cpus().length;
    for (let i = 0; i < workerCount; i++) {
        cluster.fork();
    }

    cluster.on('exit', function (worker, code, signal) {
        console.log('Worker ' + worker.process.pid + ' exited with code: ' + code);
        cluster.fork();
    });
} else {
    const server = express();
    server.use(bodyParser.json());
    server.listen(80);

    server.post('/capture', (req, res) => {
        const websiteUrl = req.body.url;

        captureWebsiteImage(websiteUrl)
            .then((image) => {
                res.status(200).json({ image: image });
            })
            .catch((err) => {
                res.status(500).json({ error: err });
            });
    });
}

The error messages I keep seeing are:

Error: Protocol error (Page.navigate): Target closed.
Error: Protocol error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed.

I’m not sure what causes this. Could it be related to redirects or some other page behavior? Has anyone encountered similar issues with Puppeteer navigation?

2

I’ve hit this before - usually happens when sites throw popups or redirect chains that mess with puppeteer. Add --disable-popup-blocking to your launch args and check newPage.url() after goto to make sure it actually landed where you expected.

3

I’ve hit this exact problem in production. It’s usually race conditions between your page operations and browser lifecycle. You’re spinning up and tearing down browser instances too fast, which creates timing issues where pages get closed before operations finish. Switch to browser instance pooling instead of launching new browsers per request. Keep one browser instance per worker and reuse pages, or even better, use a page pool. You’re only catching navigation errors but missing screenshot failures - add proper error handling around the screenshot operation too. That networkidle2 wait condition is problematic with sites that have persistent connections. Try ‘load’ or ‘domcontentloaded’ instead for more reliable results. Also check if you’re hitting memory limits. That --disable-dev-shm-usage flag tells me you might be in a constrained environment where Chrome’s getting killed unexpectedly.

4

This looks like Chrome’s crashing, not your Puppeteer code. I’ve seen this when Chrome gets killed unexpectedly or runs out of memory. Your cluster setup might have multiple workers fighting for resources. Check your system logs for Chrome crashes or OOM kills - that ‘target closed’ error means Chrome died, not that navigation failed. Try adding --single-process to your launch args to test if it helps. Also add delays between requests and watch memory usage per worker. If you’re using Docker, bump up the memory limits. The error hits before your catch blocks because Chrome just dies suddenly, leaving your page instance hanging.