About two weeks ago, I embarked on an ambitious project to create a headless browser using the Go programming language. My initial steps involved exploring options for an HTML parser and finding a way to integrate the V8 JavaScript engine, allowing native Go objects to be accessible as JavaScript entities. Although I initially tried implementing a custom HTML parser, I transitioned to using the x/net/html package for two-step parsing, thanks to a suggestion from a helpful user in the community. Simultaneously, I discovered v8go, which allows embedding V8 in Go applications, but I needed to fork it to incorporate some essential features missing in the original version.
In the subsequent week, I enhanced the DOM model and established JavaScript bindings to native objects, addressing the absent features from v8go while getting inline JavaScript to run during the DOM tree construction, ensuring the right DOM was available for execution.
Recently, I reached a significant milestone: the browser can now download and execute JavaScript from external sources, such as retrieving a script from a specified source in an HTML document.
As my goal is to create a testing tool for Go applications, I designed the browser to directly work with an http.Handler instance to bypass the TCP stack overhead. Below is an internal test demonstrating this functionality (using Ginkgo/Gomega):
It("Should correctly load and run JavaScript from script tags", func() {
// Set up a simple server that delivers an HTML page and a JS script
mux := http.NewServeMux()
mux.HandleFunc(
"GET /webpage.html",
func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(`<html><head><script src="/scripts/main.js"></script></head><body>Hello, Universe!</body>`))
},
)
// The JavaScript performs a simple operation, setting a global variable
mux.HandleFunc(
"GET /scripts/main.js",
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/javascript")
w.Write([]byte(`var scriptExecuted = true`))
},
)
// Create a browser that interacts with our server. Open the webpage and check the global JS variable.
testBrowser := ctx.NewBrowserFromHandler(mux)
Expect(testBrowser.OpenWindow("/webpage.html")).To(Succeed())
Expect(ctx.RunTestScript("window.scriptExecuted")).To(BeTrue())
})
Upcoming Goals
This project originated from my keen interest in integrating a Go and HTMX setup, so my next goal is to implement a basic HTMX-driven application. This will necessitate the development of additional browser APIs, such as XMLHttpRequest (currently in progress), XPathEvaluator (which I can initially implement in JavaScript, but first, the DOM must support the required methods), and the location API, which I intend to address next after minor refactoring.
Explore the Project
Please note that this development is in the very early stages and not yet user-ready. You can find more details about the project at my GitHub repository. The original v8go project appears to be somewhat inactive, but it has been taken over by Tommie, whose fork contains updated dependencies. Be sure to check out his repository as well.
Perhaps consider looking into headless browser libraries in other languages, like Puppeteer for node.js or Selenium, for feature inspo? Also, when dealing with XMLHttpRequest, be mindful of potential cross-origin requests challenges. Massive respect for tackling such a complex topic in Go!
working directly with an