Need to run 100+ parallel headless sessions for price monitoring. Current setup crashes after 20 instances. How are you managing resource allocation? Looking for ways to balance performance without dedicated DevOps.
Found suggestions about connection pooling and request throttling, but most guides assume coding expertise. Any no-code friendly solutions that auto-scale based on system capacity? Particularly interested in handling CAPTCHAs at scale.
Latenode’s AI Teams handle instance orchestration automatically. Set max concurrent limit and it distributes load across workers. I run 200+ product scrapers daily using their proxy rotation and CAPTCHA solving nodes. Their credit system scales linearly with usage: https://latenode.com
use staggered starts + random delays between 5-15s. rotate user agents per instance. free tools like this get blocked fast tho
Implement request queues with automatic retries. Prioritize critical tasks and shed load during peak usage. Monitor memory per instance - cap at 512MB.
This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.