Airflow DAG execution causes Python Telegram bot to hang

Other Questions Telegram bot
1

Issue with Python Telegram Bot in Airflow DAG

I’m running into a problem with my Python Telegram Bot when I try to run it through an Airflow DAG. The DAG starts and launches the bot, but it doesn’t respond to any commands. Oddly enough, the bot works fine when I run the file manually.

I’m using Astro CLI 1.31.0 to run Airflow in a Docker container. I’m pretty new to Airflow, so I’m not sure what’s going wrong.

Here’s a simplified version of my code:

import telebot
import os

API_TOKEN = 'MY_BOT_TOKEN'
bot = telebot.TeleBot(API_TOKEN)

def run_bot():
    @bot.message_handler(commands=['start'])
    def send_welcome(message):
        bot.reply_to(message, 'Hello! How can I help you?')

    bot.infinity_polling()

if __name__ == '__main__':
    run_bot()

I’ve tried setting NO_PROXY = '*' in the environment variables, but it didn’t help. Any ideas on what might be causing this issue or how to fix it?

2

I encountered a similar issue when integrating a Telegram bot with Airflow. One solution that worked for me was to use the ‘PythonOperator’ in Airflow instead of directly running the bot script. This allows better control over the execution.

Try modifying your DAG to use a PythonOperator that calls your bot function. Also, ensure your Airflow worker has the necessary network access to communicate with the Telegram API. You might need to configure your Docker network settings or proxy if you’re behind a corporate firewall.

Additionally, consider implementing logging in your bot script to help diagnose where it’s hanging. This can provide valuable insights into what’s happening when the DAG runs.

3

As someone who’s dealt with similar Telegram bot issues in Airflow, I can offer a few suggestions. First, make sure your Airflow environment has the necessary network permissions to connect to Telegram’s servers. Sometimes, container networking can be tricky.

Also, consider using a non-blocking approach. Instead of infinity_polling(), you could set up a webhook or use a timeout as others suggested. This prevents your DAG from getting stuck.

Another thing to check is your Airflow executor. If you’re using LocalExecutor or SequentialExecutor, it might not handle long-running tasks well. Try switching to CeleryExecutor if possible.

Lastly, implement proper error handling and logging in your bot code. This will help you pinpoint where exactly the issue occurs when running in Airflow. It’s saved me countless hours of debugging in the past.

4

hey there, i had a similar issue. try adding a timeout parameter to your infinity_polling() call, like bot.infinity_polling(timeout=10). this helped me when my bot was hanging in airflow. also, make sure your airflow task has enough time to run before timing out. good luck!