I’m working on a Linux system and wondering if it’s possible to configure a headless browser (like Chromium, Firefox, or through WebDriver) to act as a proxy server for terminal commands such as wget or curl.
The idea is that when I run these command line tools, they would route their requests through the headless browser instead of making direct HTTP requests. This could be useful for handling JavaScript-heavy sites or bypassing certain restrictions.
Has anyone successfully implemented this kind of setup? What would be the best approach to make command line utilities use a headless browser as their proxy? Are there any specific tools or configurations that make this easier to accomplish?
Selenium Wire sounds perfect for this. I’ve been running something similar for six months - Chrome headless with Selenium, configured to listen on a specific port as a proxy. Set up your webdriver with custom Chrome options to enable proxy functionality, then point curl/wget to that local proxy port. Performance isn’t great compared to direct requests, but it handles JavaScript rendering and beats most anti-bot measures. Just watch your browser instances - they’ll eat memory if you don’t manage them properly. Works great for scraping SPAs that need JavaScript to load content.
Been there, done that. I ended up using mitmproxy with a headless browser backend. Here’s how it works: run mitmproxy as your proxy server and have it forward requests to a headless browser when you need JavaScript rendering. Write custom scripts to detect which requests actually need browser rendering vs. simple HTTP calls. Set HTTP_PROXY and HTTPS_PROXY to point to mitmproxy, then configure the logic to route JS-heavy domains through your headless instance. Way better than pure Selenium - you get granular control and don’t waste resources rendering simple API calls through a full browser. Takes some upfront scripting but performs much better.
puppeteer works gr8 for this! i used it with headless Chrome and a Node script for proxying. it’s kinda slow at times, but gets the job done on many sites. just be careful with resource limits.