Asynchronous Task Processing in Agentic Workflows

[rakro101]

🚀 Feature

Support for Asynchronous Task Processing in Agentic Workflows.

Provide functionality for asynchronous task management, enabling clients to submit tasks that require extended processing time (e.g., > 60 seconds) and receive responses in an asynchronous manner. This includes mechanisms for queuing tasks, notifying clients upon task completion, and ensuring smooth client-server interaction.

Motivation

In workflows where prediction steps involve significant computation time, it becomes impractical to block client requests for extended durations. This feature would enable the following:

1. Efficient task queuing and processing for long-running tasks.
2. Improved client experience by offering non-blocking task submission and flexible result delivery options (polling or push notifications).
3. Better resource utilization on the server-side, allowing for scalable management of incoming tasks.

Currently, there is no native support in the litserve package to handle such asynchronous workflows, which limits its applicability to real-world, complex AI systems requiring prolonged computation.

Pitch

The proposed functionality would introduce:

1. Task Submission API:

   • Clients submit requests containing input data (e.g., ID and Document).
   • The server acknowledges the request with an immediate response containing a unique task ID or tracking URL.

2. Task Processing:

   • The server processes the task asynchronously in the background.

3. Result Retrieval Options:

   • Option A: Clients can poll the server using the task ID to retrieve the status and result.
   • Option B: Clients can register a callback URL for the server to push the results once processing is complete.

4. Server Response API:

   • The server sends the final result (e.g., task ID and encoded model output) back to the client.
   • The client acknowledges the result to complete the workflow.

Additional context

This feature would align litserve with modern REST API patterns for long-running workflows, making it more suitable for real-world AI applications. Below is a simple diagram of the workflow:

1. Client-Server Workflow:

   • Client submits a request: { "id": "123", "document": "text" }.
   • Server responds: { "status": "accepted", "task_id": "123-task" }.

2. Option A (Polling):

   • Client polls: /status/123-task.
   • Server responds: { "status": "completed", "result": "encoded output" }.

3. Option B (Webhook):

   • Server sends: { "task_id": "123-task", "result": "encoded output" } to a client-specified URL.

This enhancement would allow seamless integration into systems with asynchronous requirements and complex workflows.
(This request was support by GenAI.)

[aniketmaurya]

related to #348

[aniketmaurya]

Thank you for writing such a detailed feature request @rakro101! We are planning to add this sometime soon. Will keep you posted about the progress.