I’ve been running the same puppeteer scraper for about 6 months now and it’s been solid. Then last week the client’s website got a redesign and suddenly nothing worked. Selectors were broken, navigation paths changed, the whole thing needed to be rewritten.
This happens constantly. I spend hours fixing scripts after every minor page update. It feels like I’m chasing a moving target instead of actually building something stable.
I’ve tried adding retry logic and fallback selectors, but that only buys me a few weeks before something else changes. The real issue is that puppeteer scripts are too brittle—they’re tightly coupled to the exact HTML structure.
I’m wondering if there’s a better approach here. Should I be thinking about this differently? Has anyone found a way to make browser automation resilient enough to handle regular page changes without constant maintenance?
This is exactly the kind of problem automation should solve, not create. The brittleness you’re describing comes from hardcoding selectors and workflows to specific page structures.
What you need is a system where the automation can adapt when pages change. That’s where AI-driven workflow coordination becomes valuable. Instead of one rigid puppeteer script, you can build a workflow with multiple coordinated agents—a navigator agent that handles page traversal, an extractor agent that pulls data, and a retry agent that self-corrects when something breaks.
When the page layout changes, the extraction agent can intelligently re-identify elements instead of failing on a hardcoded selector. The retry agent catches the failure and loops back to reconfigure.
Latenode lets you build this kind of resilient automation setup visually, without needing to maintain dozens of brittle scripts. You can orchestrate multiple AI agents that work together to keep your browser automation stable and self-healing.
I ran into this exact issue last year. What saved me was shifting from selector-based automation to behavior-based automation. Instead of targeting specific HTML elements, I started thinking about what the page actually does—what buttons open what, where forms submit to, that kind of thing.
I also started adding validation steps. After each action, I’d check if the page reached the expected state, not just if the selector existed. When selectors broke, the validation caught it early and I could log it for investigation instead of letting the whole script fail silently.
But honestly, the real fix is accepting that maintenance is part of the game. I budget time every sprint specifically for updating automations. Once I stopped fighting that reality, it became much less frustrating.
Have you considered monitoring the pages you’re scraping? I set up a simple check that runs daily to see if my selectors still exist. If they don’t, it sends me an alert before the scraper breaks. Gives me a heads-up instead of finding out when production fails.
Also, try using multiple selector strategies. Instead of relying on IDs or classes that change, use combinations like text content matching or ARIA labels. Most redesigns don’t touch those, so your script keeps working even when the underlying structure changes.
The core issue is that puppeteer without any abstraction layer is inherently fragile because it’s designed to interact with specific DOM structures. What you’re experiencing is a symptom of tight coupling between your automation and the page markup. One approach that’s helped teams I’ve worked with is implementing a page object model—create a layer that abstracts the page structure away from your automation logic. If the page changes, you only update the page object, not every script that uses it. This doesn’t eliminate the maintenance burden, but it centralizes it and makes updates faster.
Use explicit waits and multiple selector fallbacks. target elements by text content or ARIA labels before relying on IDs. Also monitor your selectors daily to catch breaks early.