I’m exploring cloud computing and I came up with an idea to build a project that collects job listings from a popular employment site. My plan is to run a headless instance of a Chromium browser on AWS Lambda, scrape the information dynamically, and then store it in an S3 bucket. I’m not sure about the best ways to scale such a setup. What methods or strategies would you recommend for handling scalability?
hey, try tying lambda with sqs so each scraping task autoscale, then use something like dynamodb for managing session state. feels reliabl, and you get flexible scaling without a ton of setup!