How can I implement an endless loop using puppeteer-cluster?

Other Questions Puppeteer
1

I’m working on a project using puppeteer-cluster and I need help setting up an endless loop for my tasks. I wrote some initial code, but after about 30 seconds of execution, the process becomes idle and stops responding as expected. My goal is to ensure that the cluster continuously processes tasks without interruption. Below is an example of my current approach, and I would appreciate any ideas on how to modify it so that the loop runs seamlessly without idling.

const { Cluster } = require('puppeteer-cluster');

(async () => {
  const endlessCluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_PAGE,
    maxConcurrency: 4
  });

  endlessCluster.task(async ({ page, data: targetUrl }) => {
    await page.goto(targetUrl);
    console.log(`Navigated to ${targetUrl}`);
  });

  // Running an infinite loop to continuously queue tasks
  while (true) {
    endlessCluster.queue('https://example.org');
    await new Promise(resolve => setTimeout(resolve, 1000));
  }
})();
2

hey, try overriding the idle timeout settings in puppeteer-cluster and pump tasks faster. looks like the cluster stops when it thinks there’s nothing left for it to do. maybe bumping the concurrency or tweaking timers can help keep it running non-stop.

3

In my experience, ensuring a continuous flow of tasks in puppeteer-cluster requires careful handling of the idle state. I found that setting up a monitoring function to detect when the cluster is approaching idleness was beneficial. This function immediately schedules additional jobs or checks if any pending errors have caused the shutdown. Also, it’s important to make sure error handling in each task is robust to prevent an unhandled exception from stopping the entire process. Adjusting the cluster configuration and incorporating such checks maintained an ongoing loop without interruption.