I’m working on automating file transfers between two cloud storage accounts using automation tools. I’ve successfully completed the first part of my workflow which includes receiving the webhook trigger, locating the source folder, and getting a list of all file paths.
The challenge I’m facing is with the bulk operations. For each file in my list, I need to:
Create temporary download URLs using the file paths
Use those URLs to upload files to the destination account
Since my automation platform only handles single requests, processing multiple files becomes a bottleneck. If there are 15 files, that means 15 separate calls to generate links plus another 15 calls to transfer them.
I think I need a script that can loop through all the file paths and handle both operations in batches. Since I’m new to programming, could someone help me understand how to structure this in JavaScript or Python?
Basically I need to iterate through my file path array, call the temporary link API for each one, then use those results to call the upload API for the second storage service.
JavaScript with Promise.all() is perfect for this if you know it already. I built something similar last year for Dropbox to OneDrive transfers. The trick is batching each operation separately instead of chaining them one by one. Run all your temp URL requests at once, wait for them to finish, then batch all the uploads. Just set a concurrency limit - learned the hard way that most cloud APIs start throttling around 20 concurrent requests. Also throw in a simple queue system so you can pause/resume if things break during big transfers. Even basic batching beats sequential processing by a mile.
Had the same issue moving files between AWS S3 and Google Cloud Storage. You need async operations with proper batching - that’s what’s killing your performance right now. Use Python with asyncio and aiohttp. Generate all your temp URLs at once, then batch the uploads. I stick to 10-15 concurrent operations max or you’ll hit rate limits. Don’t forget retry logic with exponential backoff - cloud APIs fail randomly. Break your file list into chunks of 50-100 files, especially with large datasets. The difference is huge - I went from 30 minutes for 500 files down to under 5 minutes.
first, check if your cloud provider has batch apis - don’t write custom scripts yet. azure blob and aws s3 both have built-in batch operations that’ll handle multiple files in one request. way more efficient than looping individual calls. no batch support? then go with asyncio, but keep it simple to start.