Integrating puppeteer plugins with cluster functionality for web scraping

Other Questions Puppeteer
1

Hey everyone! I’m working on a web scraping project and I’ve hit a roadblock. I’m using puppeteer with some plugins to avoid detection and block ads, plus I’m routing through TOR for IP rotation. It works okay, but it’s not super reliable.

I heard about puppeteer-cluster and I think it could help me manage multiple pages and keep track of my scraping tasks better. The thing is, I’m not sure how to combine it with my current setup.

Here’s a simplified version of what I’m doing now:

const puppeteer = require('puppeteer-extra');
const stealthPlugin = require('puppeteer-extra-plugin-stealth');
const adBlockPlugin = require('puppeteer-extra-plugin-adblocker');

puppeteer.use(stealthPlugin());
puppeteer.use(adBlockPlugin());

async function scrapePage(url, torPort) {
  const browser = await puppeteer.launch({
    args: [`--proxy-server=socks5://127.0.0.1:${torPort}`, '--no-sandbox'],
    ignoreHTTPSErrors: true
  });
  
  const page = await browser.newPage();
  await page.goto(url);
  
  // Scraping logic here
  
  await browser.close();
}

Can anyone help me figure out how to use this with puppeteer-cluster? I saw an example in their docs, but it didn’t really click for me. Thanks in advance!

2

hey man, i’ve used puppeteer-cluster before and it’s pretty sweet for scaling up scraping. here’s a quick tip - when setting up ur cluster, make sure to pass in the puppeteer instance with plugins:

const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
puppeteer,
// other options
});

then just queue ur URLs and let it rip. way easier than managing everything urself!

3

I’ve been down this road before, and integrating puppeteer-cluster with plugins can be tricky but totally worth it. Here’s what worked for me:

First, set up your cluster with the puppeteer instance that has your plugins:

const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_CONTEXT,
  maxConcurrency: 3,
  puppeteer,
  puppeteerOptions: {
    args: ['--proxy-server=socks5://127.0.0.1:9050', '--no-sandbox'],
    ignoreHTTPSErrors: true
  }
});

Then, define your task:

await cluster.task(async ({ page, data: url }) => {
  await page.goto(url);
  // Your scraping logic here
});

Queue your URLs and let the cluster handle the rest. This approach maintains your plugins and TOR setup while leveraging cluster’s concurrency management. Just remember to adjust maxConcurrency based on your system’s capabilities and TOR’s limits.

4

As someone who’s been in the trenches with puppeteer and clustering, I can tell you it’s a game-changer for web scraping at scale. Here’s what I’ve found works well:

First, set up your cluster with your existing puppeteer instance:

const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 5, // Adjust based on your system and needs
puppeteer,
puppeteerOptions: {
args: [‘–proxy-server=socks5://127.0.0.1:9050’, ‘–no-sandbox’],
ignoreHTTPSErrors: true
}
});

Then define your task:

await cluster.task(async ({ page, data: url }) => {
await page.goto(url);
// Your scraping logic here
});

Now you can just queue up your URLs:

cluster.queue(‘https://example.com’);
cluster.queue(‘https://example.org’);

This approach maintains your stealth and ad-blocking plugins while leveraging cluster’s concurrency management. Just remember to close the cluster when you’re done:

await cluster.idle();
await cluster.close();

In my experience, this setup significantly improved reliability and throughput. Hope this helps!