What's the best way to manage extended tasks in n8n while providing progress updates?

Other Questions N8n
1

Hi folks! I’m working on an n8n workflow that takes a while to finish. It involves web scraping and using OpenAI. My main issue is letting users know what’s happening during the process.

Here’s what I’ve got so far:

1. Start with webhook
2. Send 'Started processing' message
3. Do the main work (3-5 min)
4. Send final result

But my app needs to show progress. I tried using Cloudflare KV for status updates, but it was too slow. Then I tried waiting for the full response, but that’s not great for longer tasks.

What do you use for this? A special database? Some webhook trick? Polling? Maybe websockets?

I’d really appreciate any real-world advice, especially for tasks that run for several minutes. Thanks for your help!

2

hey there JumpingMountain! have u considered using a message queue like RabbitMQ? it’s pretty solid for long-running tasks. u can send status updates to the queue as ur workflow progresses, and have a separate worker process that reads from the queue and updates ur frontend. it’s scalable and works great for tasks that take a while

3

I have faced similar challenges in my n8n projects. In my experience, a combination of webhook callbacks and a fast in-memory data store such as Redis offers a reliable solution. I begin by generating a unique task identifier and storing an initial status in Redis. As the task progresses, I update the status in Redis accordingly. Separately, I created an endpoint that the frontend polls for updates, ensuring users receive consistent progress information. Employing automatic expiration for status entries also helps maintain system efficiency. This approach works well even when tasks run for extended periods.

4

One approach I’ve found effective is implementing a progress tracking system using a lightweight database like SQLite.

Here’s how it works:

  1. Create a table to store task progress.
  2. At the start of your workflow, insert a new record with an initial status.
  3. Update this record at key points in your process.
  4. Set up a separate endpoint that queries this database for the latest status.

Your frontend can then periodically check this endpoint for updates. This method is fast, scalable, and doesn’t significantly impact your main workflow’s performance. It’s particularly useful for those longer tasks you mentioned, providing real-time updates without the overhead of constant API calls or the complexity of websockets.