Hey everyone,
I’m on the hunt for a skilled coder to build a powerful tool that can grab product info from big retail sites. The tricky part? It needs to be smart enough to dodge those pesky bot blockers.
Here’s what I’m after:
- Tool should work with main category pages
- Need to pull stuff like product names, prices, ratings, and stock levels
- Gotta be sneaky to avoid getting shut out (you know, mix up IPs, act human-like)
- Prefer it spits out data in a neat CSV or JSON format
If you’ve got experience with this kind of thing, I’d love to hear from you! Drop a comment or shoot me a message. Let me know your price, how long it might take, and maybe share some similar stuff you’ve done before.
Thanks for checking this out!
I’ve actually tackled a similar project before, and I can tell you it’s quite the challenge. Retail sites are getting savvier about blocking scrapers, so you’ll need some pretty advanced techniques to stay under the radar. In my experience, rotating proxies and mimicking human behavior patterns (like randomized delays between requests) are crucial.
One thing to keep in mind is that this kind of tool requires constant maintenance. Retail sites frequently change their layouts and anti-bot measures, so you’ll need to be prepared for ongoing updates.
As for the ethical concerns others have raised - it’s definitely a gray area. I’d suggest thoroughly reviewing the terms of service for any sites you’re targeting. Some explicitly forbid scraping, while others may be more lenient.
If you decide to proceed, I’d estimate at least 2-3 weeks of development time for a robust solution, potentially more depending on the specific sites and data points you’re after. Feel free to reach out if you want to discuss further details.
While I understand the desire for efficient data collection, I would caution against developing tools specifically designed to circumvent website security measures or terms of service. Many retailers have policies against automated scraping for good reasons. Perhaps we could explore ethical alternatives that respect site policies while still meeting your business needs? There may be official APIs or data partnerships available that could provide the product information you’re seeking through approved channels. I’d be happy to discuss more above-board approaches if you’d like to share more about your specific use case and goals.
hey, i’ve done similar stuff before. it’s tricky but doable. you’ll need rotating proxies, user-agent spoofing, and request throttling to avoid detection. might take a couple weeks. PM me if u wanna chat more about it. just be careful with TOS stuff, some sites don’t allow scraping.