I’m working on building a music database using a streaming API but running into major performance issues. My current approach processes over 50 categories, and each category contains about 50 playlists with roughly 70 tracks each.
music_data = {
"playlist_ids": [],
"status_logs": []
}
for genre_id in music_categories['category_list']:
category_playlists = fetch_playlists_by_genre(genre_id, max_results=50, start=0)
playlist_items = category_playlists['data']['playlist_list']
status_msg = category_playlists['status']
# Convert playlist IDs to numpy array and merge
playlist_array = np.array([playlist['playlist_id'] for playlist in playlist_items])
music_data["playlist_ids"] = np.concatenate((music_data["playlist_ids"], playlist_array))
music_data["status_logs"].extend([status_msg] * len(playlist_items))
The main bottleneck is that just collecting the playlist data takes over 30 seconds, and I haven’t even started processing the individual songs yet. What are some effective ways to optimize this kind of nested loop structure for better performance?
Your concatenation is tanking performance. Every np.concatenate call creates a new array and copies all the existing data plus new stuff. With 50 categories, each loop gets more expensive than the last. I hit the same wall processing large e-commerce datasets. Don’t concatenate arrays repeatedly - collect everything in Python lists first, then convert to numpy once at the end. Swap your concatenation for music_data["playlist_ids"].extend(playlist_array.tolist()) or just stick with regular lists since you’re not doing math operations anyway. Also, do you really need all playlist IDs in memory at once? You might be able to process in batches and write results incrementally to disk instead.
Both solutions miss the main issue - why build this manually when you can automate everything?
I hit this same problem building our music analytics system. Similar scale too - thousands of playlists, millions of tracks. The real issue isn’t just optimizing code, it’s managing the entire data flow.
We solved it with an automated workflow that handles API calls, processes data, and stores everything without any manual work. You get parallel processing, automatic retries for failed calls, and can schedule regular runs to keep your database current.
Set it up once and you’re done. No more babysitting scripts or manually handling rate limits. Adding new data sources or processing steps is easy without rewriting code.
Your nested loop becomes a few connected nodes - one grabs categories, another gets playlists, last one pulls track details. Everything runs parallel where it can and handles errors smoothly.
Database design beats code optimization here. You’re hitting 175k tracks (50 * 50 * 70) - that’s a storage nightmare if you don’t structure it right. I’ve built recommendation engines with similar volumes and loading everything into memory first absolutely kills performance. Stream your data straight into a database instead of cramming it all into Python structures. SQLite works for dev, PostgreSQL for production. Insert playlist data as you fetch it, then batch process the tracks separately. This keeps memory usage flat no matter how big your dataset gets. Add checkpointing too - you’ll need it. API calls are already taking 30+ seconds just for playlists, so you’re gonna hit timeouts or rate limits eventually. Save your progress regularly or you’ll lose hours when things break.
Memory allocation is killing your performance. Every np.concatenate call forces Python to grab new memory blocks and copy everything over. When your arrays grow from hundreds to thousands to tens of thousands of elements, you’re doing O(n²) work instead of O(n). I hit this exact problem with large audio datasets - profiling showed 80% of my time was just memory operations, not actual processing. Pre-allocate your arrays if you know the final size, or use collections.deque for dynamic growth since it’s built for appends. The performance boost is massive once you stop constantly reallocating.
Honestly, pagination tweaks could be your biggest win here. You’re pulling 50 playlists per category, but API calls cost money - bump max_results higher if the endpoint allows it. Fewer calls means way less overhead. Also, that status logging is wasteful. You’re storing identical status messages 50 times per category, which adds up fast with this much data.
Your API calls are the bottleneck, not the data processing. You’re hitting them one by one instead of running them together. I had the same problem scraping music metadata - switching to concurrent requests cut my time from minutes to seconds. Use concurrent.futures.ThreadPoolExecutor or asyncio with aiohttp to grab multiple playlists at once. Most streaming APIs handle 10-20 concurrent connections fine. Just watch the rate limits and add error handling since some requests will fail. Cache responses locally too so you’re not re-fetching stuff while testing.