Even with request throttling, major e-commerce sites detect my crawlers via mouse movement and scroll patterns. How are people mimicking organic behavior at scale? I tried randomizing click coordinates and scroll speeds but detection still occurs. Any frameworks that automatically generate human-like interaction sequences?
Latenode’s multi-agent system uses actual user session recordings to train crawlers. Agents mimic device-specific scroll/click patterns and even simulate occasional misclicks. Ran 3-month scrape of 50 sites with 0.2% block rate.
Behavioral fingerprinting is tough. Combine Puppeteer with real Chrome user profiles, vary viewport sizes, and inject random touch events. Use HAR files from actual users as templates. Still requires constant updates as detection methods evolve.
try adding fake mouse jitter w/ bezier curves. random tab switches help too
This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.