I need recommendations for a browser automation solution that works with .NET applications running multiple threads concurrently.
Key requirements:
- Portable deployment - no server setup required, just a library I can package with my app
- Modern web support - handles JavaScript, AJAX calls, and HTML5 features with DOM manipulation capabilities
- Session management - proper handling of multiple cookie responses and session persistence
- User agent flexibility - ability to customize browser identification
- Thread safety - supports thousands of simultaneous user sessions without shared state conflicts
- Performance optimized - fast execution for high-volume operations
- SSL compatibility - works with self-signed certificates
I’ve researched several options but unsure which fits best. Looking for practical advice and code examples from developers who’ve implemented similar solutions in production environments.
playwright-dotnet might be what you’re looking for. I’ve been using it for 8 months and it handles concurrency really well - ran 3k+ sessions without major issues. Unlike Puppeteer, it supports Chrome, Firefox, and WebKit which is great for testing. Deployment’s clean since it auto-downloads browser binaries on first run. SSL cert handling works out of the box and user agent spoofing is dead simple.
HtmlAgilityPack with HttpClientFactory is worth checking out if you want something lighter. I used this combo for a financial scraping app that handled 5,000 concurrent requests - worked surprisingly well.
Biggest win is way less memory usage than browser automation. HttpClientFactory handles connection pooling automatically, and you can set custom user agents and SSL handling through HttpClientHandler. Cookie management works great with CookieContainer, plus it’s thread-safe when configured right.
Downside is it can’t handle heavy JavaScript sites since it doesn’t run scripts. I ended up mixing HtmlAgilityPack for simple stuff and falling back to headless Selenium for JS-heavy pages. This hybrid cut resource usage by 60% while keeping everything functional. Performance was solid for form submissions and basic DOM work, though you’ll need something like AngleSharp for complex HTML parsing.
PuppeteerSharp is an excellent choice for your needs. I’ve successfully used it in a production environment with approximately 2,000 concurrent sessions for the past 18 months. The library includes Chromium, simplifying deployment since you just need to add the NuGet package without any server setup.
To ensure thread safety, make sure to isolate browser instances for each thread. Each instance maintains its own cookie jar and session state, preventing shared state conflicts. While the performance remains solid, be aware that memory usage may increase significantly with high concurrency, so proper disposal is essential.
For handling SSL certificates, the IgnoreHTTPSErrors option works well, and user agent customization is straightforward via page settings. As for JavaScript support, it performs excellently, leveraging full Chrome. One thing to consider is that PuppeteerSharp adds about 280MB to your deployment size due to the bundled Chromium, but if that is manageable for you, it’s worth exploring.