Hi everyone!
I’m working on a B2B application using Python FastAPI and Celery with Redis. I want to add workflow automation features similar to what you’d find in Zapier but built directly into my platform.
Basically I need things like background job scheduling, connecting different APIs together (like Slack, Notion, email services), handling delays between steps, and automatic retries when things fail.
I really like the approach that Trigger.dev takes with their clean interface and git-based configuration, but that’s all Node.js focused. I want to stick with Python instead of adding another language to my stack.
My current setup includes:
- FastAPI for the main API
- Celery with Redis as the message broker
- Considering APScheduler for better cron job handling
- Using Flower for monitoring but it looks pretty old
What I’m looking for:
How do you build polished automation systems using Python? What approaches or libraries help make Celery-based workflows feel more modern? Specifically interested in:
- Better visibility into running workflows and debugging
- Smart retry handling and task chaining
- Nice dashboards for non-technical users
- Clean ways to define workflows in code
Any suggestions for tools, architectural patterns, or open source projects would be super helpful!
Check out Prefect or Airflow instead of modernizing Celery from scratch. I switched from Celery to Prefect 8 months ago - night and day difference. Prefect’s got clean workflow syntax, a UI that doesn’t look ancient, smart retry logic by default, and proper versioning built-in. For B2B stuff, Prefect Cloud has solid user management and easy API access for workflow triggers. Want something closer to your current setup? Dramatiq feels way more modern than Celery but keeps Redis. Better monitoring and handles failures more gracefully. Either way, you’ll skip months of custom dashboard work and get real workflow orchestration without rebuilding everything.
check out temporal - their python sdk handles retries and state management automaticly, plus great observability features. I switched from celery 6 months ago after hitting too many limitations. the workflow definitions are clean and you get distributed tracing built-in, which beats debugging with flower by miles.
I built something like this last year and ended up wrapping Celery with custom code that made everything way more maintainable. The trick was treating workflows as state machines instead of just chaining tasks together. Used SQLAlchemy to save workflow states and threw together a simple REST API so non-technical folks could use it through a React dashboard. For monitoring, I completely ditched Flower and built my own endpoints that spit out workflow progress as JSON. Way cleaner visualizations and real-time updates. Debugging got infinitely easier once I started logging every state change with proper context. Python dataclasses for workflow definitions was a game changer - keeps everything type-safe but other devs can still read it. Added a simple plugin system too, so hooking up new APIs became dead simple. The whole thing feels way more solid than vanilla Celery and gives you that modern workflow engine vibe without having to switch languages.