I’ve been using Selenium for a while but now I need a headless browser that can handle lots of instances at once. PhantomJS worked okay for a small test but it eats up too much memory when I try to run hundreds of instances.
I’m looking for something that:
- Supports JavaScript, Ajax, and HTML5
- Works with proxies
- Uses minimal memory and CPU so I can run 100+ instances at the same time
- Runs on Windows
It would be awesome if it also:
- Has a C# .NET wrapper (not required)
- Doesn’t need installation
- Has good docs
- Is Webkit-based
I’m thinking about trying ZombieJs or HTMLUnit but I’m not sure if they’re the best options. Has anyone done something similar? What would you recommend for this kind of setup? I’d really appreciate any advice before I dive into testing these out.
hey, have u tried cypress? it’s pretty cool for what ur after. handles js n ajax like a boss, works with proxies, and doesn’t hog resources. i’ve run tons of instances no prob. it’s got decent docs too. only thing is it needs some setup, but totally worth it. give it a go, might solve ur headless browser headache!
Having worked extensively with headless browsers, I’d recommend giving HtmlAgilityPack a try. It’s lightweight, fast, and integrates seamlessly with C# .NET projects. While it doesn’t have full JavaScript support out of the box, you can pair it with AngleSharp for JS execution if needed.
HtmlAgilityPack is memory-efficient, allowing you to run numerous instances concurrently without significant resource drain. It’s particularly adept at handling HTML parsing and manipulation tasks.
For proxy support, you can easily combine it with HttpClient. The documentation is comprehensive, and there’s a large community behind it, which means plenty of resources and examples available online.
If you absolutely need full browser capabilities, CefSharp might be worth considering. It’s Chromium-based and offers more features, but at the cost of a larger footprint.
I’ve been in a similar situation, and after trying various options, I found that Puppeteer-sharp worked wonders for my needs. It’s a .NET port of Puppeteer, which is built on Chromium. The memory footprint is surprisingly low, and it handles multiple instances like a champ.
What I love about Puppeteer-sharp is its robust JavaScript support and how seamlessly it integrates with proxies. I’ve successfully run over 100 instances simultaneously on a decent machine without breaking a sweat.
The documentation is comprehensive, and the community support is solid. Plus, being Chromium-based, it’s got great compatibility with modern web standards.
One caveat: it does require some setup, but it’s worth it for the performance gains. If you’re comfortable with C#, you’ll find the learning curve pretty manageable. Give it a shot – it might just be the solution you’re looking for.