Seeking multithreaded web automation tool with proxy and JavaScript support

Other Questions Headless Browser
1

Hey everyone,

I’m trying to find a tool for web automation that can handle multiple threads, use different proxies for each thread, and work well with JavaScript and AJAX. I’ve tried a bunch of options like Awesomium and WatiN, but none of them quite fit the bill.

Here’s what I need:

  • Multithreading support
  • Ability to set different proxies for each thread
  • Good JavaScript and AJAX handling
  • Preferably easy to use with C#

I managed to get something working with Selenium WebDriver and ChromeDriver, but it uses a ton of memory when running multiple threads. Here’s a snippet of what I’ve got:

foreach (var proxy in proxyList)
{
    tasks.Add(Task.Run(() => RunAutomation(proxy)));
}

void RunAutomation(string proxy)
{
    var options = new ChromeOptions();
    options.AddArgument($"--proxy-server={proxy}");
    using var driver = new ChromeDriver(options);
    driver.Navigate().GoToUrl("https://example.com");
    // Do stuff
}

Any ideas on how to make this more efficient or suggestions for better tools? I’m trying to automate tests on a local network site. Thanks!

2

I’ve been in a similar situation, and I found that Puppeteer Sharp might be what you’re looking for. It’s a .NET port of the popular Puppeteer library and ticks all your boxes.

Here’s why I think it could work well for you:

  1. It supports multithreading out of the box.
  2. You can easily set different proxies for each instance.
  3. It has excellent JavaScript and AJAX support since it uses a real browser (Chromium) under the hood.
  4. It’s designed to be used with C#, so it should fit right into your existing stack.

In my experience, Puppeteer Sharp is more memory-efficient than Selenium when running multiple instances. It also has a smaller footprint because it uses headless Chrome by default.

Here’s a quick example of how you might use it:

using var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Args = new[] { "$\"--proxy-server={proxy}\"" }
});
var page = await browser.NewPageAsync();
await page.GoToAsync("https://example.com");
// Do stuff

Hope this helps! Let me know if you need any more info on setting it up.

3

Have you considered using CefSharp? It’s a .NET wrapper for the Chromium Embedded Framework that might suit your needs. I’ve used it for similar projects and found it quite effective.

CefSharp offers good multithreading support, allows proxy configuration per instance, and handles JavaScript and AJAX well. It’s also designed to work seamlessly with C#.

One advantage of CefSharp is its memory efficiency when running multiple instances. You can create a separate CefSharp browser for each thread, each with its own proxy settings. Here’s a basic example:

var settings = new CefSettings();
settings.CefCommandLineArgs.Add("proxy-server", proxy);
Cef.Initialize(settings);

using (var browser = new ChromiumWebBrowser("https://example.com"))
{
    // Perform actions
}

This approach should provide better performance than your current Selenium setup. Give it a try and see if it meets your requirements.

4

have u tried phantomjs? its pretty lightweight and supports multithreading + proxies. i use it for similar stuff and it works great. heres a quick example:

var driver = new PhantomJSDriver();
driver.Proxy = new Proxy { HttpProxy = proxyAddress };
driver.Navigate().GoToUrl("https://example.com");
// do stuff

way less memory usage than selenium in my experience