I’ve been thinking about how to handle scraping from pages that render everything in JavaScript. The traditional approach—just hit the endpoint and parse—doesn’t work because the data doesn’t exist until the browser renders it.
I’ve read about using multiple autonomous agents working together. Like, one agent handles the rendering part (actually loading the page, waiting for content), and another agent handles extraction and structuring. The idea is that they coordinate to break down the problem instead of one monolithic script trying to do everything.
But I’m wondering if this actually works in practice or if it just trades one complexity for another. Does an agent really understand when to hand off to the next agent? Can they actually communicate state in a way that doesn’t lose context? And if something fails partway through, how do you even debug that?
Has anyone actually run this kind of setup where a Renderer Agent and a Scraper Agent work together on dynamically rendered content? What actually worked and where did it get messy?