Navigating Cloudflare Protection with Puppeteer and FlareSolver

Other Questions Puppeteer
1

Hey everyone! I’m having trouble with web scraping on a watch site. The site uses Cloudflare and it’s giving me a headache. I’ve tried a bunch of things like Puppeteer with rotating proxies and stealth plugins, but nothing seems to work consistently.

Right now, I’m using FlareSolverr to make a pre-request and get cookies and user agent info. Then I pass these to Puppeteer. But I’m running into two big problems:

  1. FlareSolverr often can’t solve the Cloudflare challenge. It just times out.
  2. Even when FlareSolverr doesn’t see a challenge, Puppeteer still hits one when it tries to load the page.

I’m really stuck here. Does anyone have ideas on what I might be doing wrong? Or maybe a better way to get Puppeteer past Cloudflare?

Here’s a simplified version of what I’m trying:

async function scrapePage(url) {
  const flareData = await getFlareSolverData(url);
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  
  await page.setCookie(...flareData.cookies);
  await page.setUserAgent(flareData.userAgent);
  
  await page.goto(url);
  const content = await page.content();
  
  await browser.close();
  return content;
}

Any help would be awesome. Thanks!

2

Have you considered using a dedicated anti-detection browser like Incogniton or Multilogin? These tools are designed to create unique browser fingerprints and often come with built-in proxy integration, which can be more effective at bypassing Cloudflare than standard automation libraries.

Another approach worth exploring is using a residential proxy network. These proxies use real IP addresses from ISPs, making them much harder for Cloudflare to detect and block compared to data center IPs.

If you’re open to paid solutions, services like CaptchaAI or 2captcha can solve Cloudflare challenges programmatically. You could integrate these with your existing setup to handle the challenges when they appear.

Remember, Cloudflare constantly updates its detection methods, so staying current with anti-detection techniques is crucial for long-term success in web scraping projects.

3

hey charlieLion, cloudflare’s a real pain! have u tried using puppeteer-extra with stealth plugin? it might help bypass those pesky challenges. also, make sure ur using a good proxy rotation service. sometimes the issue is with IP reputation, not just headers. good luck mate!

4

I’ve dealt with similar Cloudflare issues in my scraping projects. One approach that’s worked well for me is using a headless browser like Playwright instead of Puppeteer. Playwright seems to handle Cloudflare challenges more reliably in my experience.

Another trick I’ve found helpful is implementing a retry mechanism with exponential backoff. Sometimes, Cloudflare’s challenges are temporary, and retrying after a short delay can often succeed.

You might also want to look into services like ScrapingBee or ScraperAPI. They handle a lot of the Cloudflare bypassing complexity for you, which can save a ton of headaches.

Lastly, consider reaching out to the site owners directly. Sometimes they’re open to providing API access for legitimate use cases, which can bypass these scraping hurdles altogether.