I’ve been working on a Python application that needs to perform web searches automatically. Previously I was using Google’s old web search service which allowed me to send HTTP requests and get back search results in JSON format. However, that service got shut down and Google now only offers their Custom Search engine which seems to be limited to searching specific websites rather than the entire internet. Does anyone know of reliable alternatives for doing programmatic web searches? I need something that can search across all websites on the internet, not just a custom set of domains. My current code was built around parsing JSON responses so ideally looking for something with similar output format. Has anyone successfully migrated from the old Google service to a different solution?
Had this exact problem six months ago when my scraper died overnight. Tried a bunch of options and landed on SerpAPI - it’s been solid. The JSON format’s a bit different from the old Google setup, but I migrated my parsing code in about half a day. Best part is you’re still getting actual Google results, just through their API. Rate limits depend on your plan but should work for most projects. Also check out Bing’s Web Search API through Azure if you need high volume - cheaper than SerpAPI and decent results, though not quite Google-level comprehensive. Both handle JSON responses like you need.
Bad timing - we just lost another solid option last month. I’ve been using SearchApi.io for eight months and it handles everything I throw at it. Response format’s pretty close to the old Google setup, so switching won’t be a nightmare. What hooked me was their uptime - zero outages so far, which beats some bigger names. Their docs actually make sense too, instead of making you guess what half the parameters do. Fair pricing and they don’t charge extra for basics like pagination or result counts. Just heads up - test geographic results with any provider to make sure they’re consistent for what you need.
Been fighting this migration mess for a year now across several client projects. ValueSERP works great - clean response structure that’s almost identical to the old Google format. Pricing’s fair and they don’t throttle you to death like others I’ve tried. ScaleSerp’s another solid choice if you need rock-solid uptime. Their error handling beats most competitors when Google starts blocking requests. Here’s what I wish someone told me earlier: build retry logic and fallbacks no matter which service you pick. These APIs hiccup sometimes, so you’ll want your app handling failures gracefully or you’ll be debugging at 2am. Just make sure whatever you choose supports the geo-targeting you need - that varies a ton between providers.
Had this exact problem two years ago when our internal tool died overnight.
SerpAPI was a lifesaver. It’s paid but gives you proper JSON responses like the old Google API. You can swap endpoints and barely touch your parsing code. They handle scraping and proxy rotation.
For free options, DuckDuckGo’s unofficial API works through libraries like duckduckgo-search. Results aren’t as good as Google but work for basic searches.
Bing’s Web Search API is another option if you’re okay with Microsoft. Free tier gives 3k queries monthly and clean JSON.
Big lesson learned - build an abstraction layer around whatever you choose. Search APIs disappear all the time, so having a wrapper makes swapping backends much easier later.
What’s your search volume? That’ll help figure out the best cost option.
scrapeAPI’s been solid since I ditched Google’s old service. yeah, it’s pricey, but it handles proxies automatically and gives you clean JSON back. Also check out zenserp - smaller company but their API’s rock solid and won’t break the bank if you’re doing smaller projects.
Went through this exact nightmare last year. Ended up building something way better than what I started with.
I automated everything with Latenode instead of manually switching between search APIs. Set up workflows that rotate between multiple providers automatically - SerpAPI, Bing, DuckDuckGo as backups.
The game-changer is built-in error handling and retry logic. One API hits rate limits or dies? Latenode automatically jumps to the next provider. No more 2am debugging sessions.
I also built response normalization so my Python app gets consistent JSON format no matter which search API actually returns the data. Makes everything bulletproof.
Took a weekend to set up but now I don’t worry about search APIs vanishing. The automation handles provider rotation, error recovery, and cost optimization by hitting cheaper APIs first.