How to handle concurrent file downloads in Telegram bot using python-telegram-bot library

Other Questions Telegram bot
1

I have a Telegram bot that processes video files sent by users. The bot works perfectly when handling one or two files, but crashes when multiple large files are sent simultaneously.

Current setup:

I register the handler like this:

app.add_handler(MessageHandler(filters.VIDEO, process_video_handler, filters.User(username="@myuser")))

My handler function:

async def process_video_handler(update, context):
    try:
        video_name = update.message.video.file_name
        video_name = sanitize_name(video_name)
        
        status_msg = f"<b>{video_name}</b> received successfully"
        await update.message.reply_text(status_msg, parse_mode=ParseMode.HTML)
        
        video_file = await context.bot.get_file(update.message.video)
        if not video_name:
            parsed_url = urlparse(video_file.file_path)
            video_name = os.path.basename(parsed_url.path)
            
        await update.message.reply_text(f"<b>{video_name}</b> ready for download", parse_mode=ParseMode.HTML)
        
        await video_file.download_to_drive(video_name)
        await update.message.reply_text(f"<b>{video_name}</b> downloaded successfully", parse_mode=ParseMode.HTML)
        
        file_exists = await check_remote_file(video_name, update)
        status = "already exists" if file_exists else "needs upload"
        await update.message.reply_text(f"<b>{video_name}</b> {status}", parse_mode=ParseMode.HTML)
        
        if not file_exists:
            upload_success = await upload_to_server(video_name, update)
            result = "uploaded successfully" if upload_success else "upload failed"
            await update.message.reply_text(f"<b>{video_name}</b> {result}", parse_mode=ParseMode.HTML)
            
    except Exception as error:
        error_msg = f"Processing failed: {str(error)}"
        await update.message.reply_text(error_msg, parse_mode=ParseMode.HTML)

The problem: When users send multiple large videos at once, the bot goes offline and downloads fail. I tried using multiprocessing but couldn’t get it working properly.

What’s the best way to handle multiple concurrent downloads without crashing the bot? Should I use queues, limit concurrent downloads, or is there a better approach?

2

The issue you’re experiencing is likely caused by memory exhaustion when handling multiple large files simultaneously. I faced similar problems with my media processing bot and found that implementing a proper task queue system works much better than just limiting concurrent operations. Consider using asyncio.Queue to manage your download tasks sequentially rather than processing them all at once. Create a background worker that processes one video at a time from the queue while your handler just adds new tasks to it. This approach prevents the bot from becoming overwhelmed and maintains stability even with heavy usage. Also worth checking your server’s memory limits and file size restrictions since large video downloads can quickly consume available resources. You might want to add file size validation before starting the download process to avoid attempting downloads that will inevitably fail.

3

I ran into this exact same issue last year when building a file processing bot. The root cause is usually timeout issues combined with blocking I/O operations that pile up. What worked for me was implementing a combination approach - first, add connection timeout handling to your download operations since Telegram’s API can be unreliable with large files. Second, consider breaking down your process into smaller chunks using asyncio.create_task() for each major operation instead of running everything sequentially in one handler. This prevents the entire handler from blocking. Also check if your server has sufficient disk space and bandwidth because multiple simultaneous downloads can exceed your hosting limits even if the code handles concurrency properly. I found that adding proper error recovery and retry logic was just as important as managing concurrency since network interruptions are common with large file transfers.

4

try using asyncio.Semaphore to limit concurrent downloads. something like semaphore = asyncio.Semaphore(2) then wrap your download logic with async with semaphore:. this way only 2 files download at once and prevents memory overload that crashes ur bot.