I’ve been running a headless browser workflow to scrape product listings, and it’s been stable for a few weeks. Then yesterday the site updated their DOM structure, and everything broke. I had to manually rewrite half the script.
This is the brittleness problem I keep running into. Every time a site tweaks their layout, I’m back in the editor trying to figure out what selectors changed. It feels like a constant game of whack-a-mole.
I’ve heard people mention using AI to help generate more robust automation, but I’m curious how that actually works in practice. Does the AI just rewrite the script when things break, or is there something smarter happening under the hood that makes the automation actually adapt to changes?
What’s your approach when a site redesigns and your automation stops working?
This is exactly where AI-assisted workflow generation changes the game. Instead of manually tweaking selectors every time a site updates, you describe what you’re trying to do in plain language—like “extract product name, price, and availability from the listings page.”
The AI generates the automation based on that description, and the key part is that the workflow logic isn’t brittle selector-grabbing. It’s built on understanding the intent. When the site changes, you can re-run the AI copilot with the same description, and it often adapts without you rewriting anything.
I’ve seen this work particularly well when you combine it with the headless browser feature. You get screenshots of the page, the AI analyzes the actual page structure, and builds the workflow accordingly. That’s much more resilient than hardcoded XPath expressions.
The real win is that you’re not just fixing one script—you’re creating a reusable pattern that can handle variations.
I dealt with this constantly when I was working on competitive intelligence automation. The scraping scripts would break monthly, sometimes weekly if a site was doing A/B testing.
What actually helped was stopping trying to fight the brittleness and instead building in some flexibility from the start. I started using more generic selectors and data attributes instead of relying on specific class names. But even that wasn’t foolproof.
The real shift for me came when I started thinking about the automation differently—less about “extract from this exact element” and more about “understand what this page is trying to show me.” That mindset change made the workflows much more adaptable.
I’ve been dealing with this for years. The frustrating part is that you can’t really predict what a site will change. Some sites redesign every quarter, others every few years. The solution I’ve found is to build automation that’s not tightly coupled to the DOM structure.
Using visual element recognition or OCR-based approaches can help, but they’re slower. The other approach is to build multiple fallback selectors—if the primary selector fails, try alternative ones. It’s not elegant, but it works when you have multiple ways to identify the same element.
The brittleness you’re experiencing is fundamentally about coupling your automation too tightly to implementation details. The site’s HTML structure is an implementation detail—what you actually care about is the data.
Some teams solve this by using API endpoints when available, or by building in layers of abstraction. Others use computer vision approaches to identify elements by appearance rather than structure. The tradeoff is usually speed versus resilience. Faster automations tend to be more brittle, while resilient ones need more sophisticated approaches.
What’s your tolerance for slower execution if it meant fewer maintenance headaches?
Use multiple selector strategies and fallbacks. If primary selector fails, try backup ones. Also look at using data attributes instead of class names—they change less often. Css and xpath combos help too.