I’m working with a content management system that has about 2 million files stored in a SQL Server database. These files include PDFs, Word documents, Excel spreadsheets and other formats. Each file is roughly 2MB on average.
My plan is to transfer all these documents to Google Drive and then save the Google Drive file IDs back in my SQL Server database as references instead of storing the actual files.
Has anyone tried something similar before? Are there any potential issues or limitations I should be aware of when dealing with this volume of data? Any advice would be really helpful.
Thanks in advance for your input.
I migrated about 800k files from our legacy system to cloud storage. Upload speed will be your biggest headache - even with multiple threads, you’re looking at weeks or months. Google Drive’s API caps you at 1000 requests per 100 seconds per user, so build in proper throttling and retry logic. Failed uploads are another pain point. You need solid tracking to resume where you left off without starting over. Test with 10k files first to get realistic timeframes and spot bottlenecks. Don’t forget about bandwidth costs and the server resources you’ll need running this thing 24/7.
The volume you’re dealing with creates serious challenges beyond just the technical migration. We tried something similar with 1.5 million files and hit dependency issues we didn’t see coming. Your app performance will definitely suffer during the transition - you’re switching from direct database access to API calls for every file request. Network latency becomes a real problem when users access documents frequently. Data integrity is another headache. You need rock-solid error handling because partial failures at this scale are guaranteed. I’d go with a hybrid approach - migrate in batches while keeping the old system running, then gradually switch over sections of your app. That way you can validate everything works before fully committing. Also budget for serious bandwidth costs. Moving 4TB isn’t cheap, especially if you need to retry failed uploads.
honestly, sounds like a nightmere waiting to happen. google drive has api limits and ur gonna hit them hard with 2 mill files. also, what happens when google changes their pricing or terms? id consider cloud storage like s3 instead - way more reliable for bulk data.