Headless Browser for .NET Without a UI

Other Questions Headless Browser
1

I’m currently building a web crawler using .NET C#. Here’s the process I’m following: Initially, I visit the primary page, which I’ll refer to as MainPage.aspx. Then, I employ HttpWebRequest to retrieve a secondary page, which I will call FormPage.aspx. Finally, I submit the form to another endpoint and collect the output from ResultsPage.aspx. The challenge I’m facing is accessing FormPage.aspx since it requires multiple cookies generated by the JavaScript in MainPage.aspx. If I attempt to access FormPage.aspx directly, I am redirected back to MainPage.aspx. The script that creates the necessary cookies is over 20KB and extremely convoluted, making it difficult for me to use tools like JINT or Javascript.net effectively. After extensive exploration, I realized that a headless browser might be the solution I need. However, I’ve tried several options and found them overly complicated. I have a library project containing my web crawlers, and I want to integrate a straightforward DLL to enable this function. Can anyone recommend a suitable headless browser? I’m open to clarifications if needed, so please ask questions instead of leaving negative feedback.

2

Try using Playwright for .NET. It’s a modern library ideal for headless browsing with support for JavaScript execution. Integration is straightforward with a .DLL, making it perfect for your project setup. Here’s a basic setup:

using Microsoft.Playwright;

var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions { Headless = true });
var page = await browser.NewPageAsync();
await page.GotoAsync("http://yourdomain/MainPage.aspx");
// Extract cookies and continue your crawler logic

Give it a shot!

3

In your scenario, Puppeteer Sharp could be a feasible solution. Puppeteer Sharp is a .NET port of the popular Node.js library, Puppeteer, and is quite adept at handling JavaScript-heavy pages in headless mode. It's relatively simple to integrate and supports full browser automation, which includes managing complex cookie scenarios like yours.

using PuppeteerSharp;

async Task HeadlessCrawl()
{
    await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultChromiumRevision);
    using (var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true }))
    using (var page = await browser.NewPageAsync())
    {
        await page.GoToAsync("http://yourdomain/MainPage.aspx");
        // JavaScript execution and cookie handling occurs seamlessly here

        // Optionally store cookies for future requests
        var cookies = await page.GetCookiesAsync();
        
        await page.GoToAsync("http://yourdomain/FormPage.aspx");

        // Continue with form submission logic
    }
}

This approach allows you to interact with pages just as a real browser would, coping well with JavaScript-based redirections and cookie manipulations. If the library fits your environment, it would be less cumbersome than manual cookie management through HTTP requests. This could be just what you need to streamline your web crawling tasks.

4

For your web crawling task, I recommend considering AngleSharp along with FluentAutomation. Although AngleSharp is more of an HTML/XML parsing library, together with FluentAutomation, it can serve headless purposes effectively by simulating a browser environment without real UI rendering. Here's a sample setup:

using AngleSharp;
using FluentAutomation;

public async Task CrawlWithAngleSharp()
{
    var config = Configuration.Default.WithDefaultLoader().WithCookies();
    var context = BrowsingContext.New(config);
    var document = await context.OpenAsync("http://yourdomain/MainPage.aspx");

    // Simulate JavaScript execution via FluentAutomation, if needed
    using (var browser = new Router())
    {
        browser.Run("http://yourdomain/MainPage.aspx")
            .Expect().Url("http://yourdomain/MainPage.aspx")
            .Navigate("http://yourdomain/FormPage.aspx");

        // Continue with your form submission and extract your desired data
    }
}

This setup enables you to handle cookies and simulate browser interactions efficiently, matching the complexities of JavaScript execution without dealing with heavy libraries. Check if this combination works seamlessly within your existing .NET project workflow.

5

If you're looking for a straightforward headless browser solution for .NET, consider ZennoPoster automation framework. It's more known for its GUI capabilities but offers a headless API as well, which can handle JavaScript and cookies effectively.

using ZennoLab.CommandCenter;

var instance = new ZennoPoster();
instance.Open("http://yourdomain/MainPage.aspx");
// Manage cookies, execute JavaScript
var cookies = instance.GetCookies();
instance.Navigate("http://yourdomain/FormPage.aspx");

It's simple to set up with your existing library and less cumbersome than heavy libraries.