Flowlitics Backend

This project is a Node.js RESTful backend API server designed to manage and execute complex workflows (Flows). It provides a robust API for defining, managing, and running workflows composed of various node types, including those interacting with AI agents and requiring human intervention. The system supports asynchronous, event-driven execution with persistence and scheduling capabilities.

⚠️ Not Production-Ready

This project is currently under active development and is not ready for production use. Expect frequent and potentially breaking changes to the codebase, features, and APIs. Use with caution and be prepared for instability.

❤️ Interested in collaborating or sponsoring? You're welcome to join! For more information, please see the bottom of this README.

Objective

The primary goal is to create a flexible and scalable engine for orchestrating complex processes that require coordination between automated tasks (like AI agent interactions) and manual steps (human-in-the-loop).

Features

• Workflow Definition: Define flows composed of interconnected nodes (Start, Action, Decision, Delay, End, Custom) and edges with optional conditions.

• Agent Integration: Integrate various AI agents and define 'human' agents to represent manual steps. See the Available Agents section for details.

• Asynchronous Execution: Execute workflows asynchronously using a queue-based, event-driven model. The executor processes nodes until a waiting state is encountered (e.g., human node pending), freeing up resources. Execution resumes upon external events (e.g., human node resolution).

• State Management: Persistent storage of flow definitions, executions, and execution steps using MongoDB.

• Human-in-the-Loop: Explicit support for nodes requiring manual input or approval, pausing execution until resolved via API.

• Pluggable Node Types: Extend functionality by adding custom node types via a plugin system.

• Scheduling: Schedule flow executions using cron expressions.

• API: Comprehensive RESTful API for managing flows, nodes, agents, executions, plugins, and schedules.

• Logging: Structured logging for monitoring system activity and execution progress.

• TypeScript: Developed using TypeScript for improved code quality and maintainability.

Architecture Highlights

The system employs an event-driven architecture centered around the queueManager and flowExecutionService.

• Queueing: New executions or resumed executions are added to a queue.

• Execution: A limited number of concurrent workers (flowExecutionService) pick up jobs from the queue.

• Partial Processing: The worker executes nodes sequentially within a flow until it encounters a node that cannot proceed (e.g., a 'pending' human node or a node waiting for input from a pending upstream node).

• Passive Waiting: When a flow is blocked, the worker releases it. The execution status remains 'running' in the database, but it's passively waiting for an external event.

• Event Trigger: An external event (e.g. an API call resolving a human node) updates the state of a specific node step.

• Re-queueing/Resumption: The event processing logic checks if the flow can now proceed. If so, the execution is re-enqueued, and a worker will pick it up to continue processing from the newly unblocked node(s).

• Finalization: When no more nodes can be processed and no nodes are pending, the execution is finalized as 'completed' or 'failed'.

This model ensures efficient resource utilization by not blocking workers on long-waiting tasks.

Key Components

• Example API Client: The ApiClient class in client-demo.ts provides methods to interact with the backend APIs for managing agents, flows, nodes, and executions.
• Flow Execution: Demonstrates creating a complex workflow with AI and human-controlled nodes, starting execution, and monitoring its progress.
• Mermaid Syntax: Thanks to Mermaid Live Editor Syntax, it is also possible to graphically represent the flow of all involved processes.

How to Run

1. Install Dependencies:

   git clone [email protected]:flowlitics/backend.git flowlitics-backend
   cd flowlitics-backend
   npm install

2. Set Up Environment:

   • Create a .env file with the necessary configuration. See .env.example for a template of all required variables.

3. Start the Server:

   npm start # or npm run dev

4. Run the Client Demo:

   Optionally, run the client-demo.ts to test workflow creation and execution:

   npm run client

Tech Stack Overview

Flowlitics Backend is built on a robust, modern tech stack that ensures scalability, maintainability, and efficient performance. Our core technologies include:

• Node.js, Express.js & TypeScript: For a fast, asynchronous, and strongly-typed backend.
• MongoDB & Mongoose: For NoSQL document storage and data modeling.
• Advanced Agent Integrations: Supporting AI SDKs (Google Gemini, Mistral AI, OpenAI, Anthropic, etc.).
• Workflow Management & Scheduling: Utilizing a custom queue system and node-cron.
• Utilities & Middleware: Including Joi for validation, Winston for logging, and more.

For a complete overview of all the technologies and integrations used, please see the full Tech Stack Documentation.

Available Agents

Our backend integrates multiple agent types to automate AI tasks and support human interactions within workflows.

The following agents are supported out-of-the-box:

• Human: Requires manual intervention for workflows.
• Gemini: Integrates with Google's Gemini AI language models.
• Codestral: Provides coding assistance via Mistral AI.
• Mistral: Access to open-source language model capabilities.
• Ollama: Local execution of AI models.
• Tavily Search: AI-powered search engine.
• Tavily Extract: Extract web page content from one or more specified URL.
• OpenAI: A variety of models from OpenAI.
• Anthropic Claude: Utilizes Anthropic's Claude models.
• Groq: Integrates with advanced models via the Groq API (compound, llama, deepseek, gemma, playai, qwen, mistral, allam ..)

For detailed information on each agent, please see the Agents Documentation.

Example Workflow Execution

The client-demo.ts script provides a practical example of how to use the API to:

1. Create AI and Human agents.
2. Define a complex flow structure with various node types (start, custom, decision, action, end).
3. Connect nodes with edges, including conditional edges.
4. Start an execution of the flow with initial data.
5. Simulate the resolution of a human-controlled node via an API call, triggering the flow to resume.
6. Poll the execution status until completion.
7. Retrieve the flow definition in Mermaid syntax for visualization.

Here you can find More Examples

Visual Representation Example

Flows can be visualized using Mermaid which can be retrieved via the API.

Thanks to Mermaid Live Editor Syntax.

Gigantic Flow Example

Here you can find More Visual Examples

Available Node Types

Flowlitics Backend supports several node types that form the building blocks of workflows. Below is a brief overview:

• Start (start): Begins the flow execution by receiving initial data.
• Action (action): Executes tasks (via AI agents or built-in logic) and may require manual intervention.
• Decision (decision): Facilitates conditional branching within a flow.
• Delay (delay): Pauses the flow for a specified duration.
• End (end): Marks the conclusion of a flow path.
• Custom (custom): Executes user-defined code or a custom handler.

For a detailed description of each node type, please refer to the Node Types Documentation.

API Endpoints Overview

The backend exposes a comprehensive set of RESTful API endpoints for managing flows, nodes, agents, executions, plugins, and schedules. Here’s a quick summary:

• Agents: Manage AI and human agents.
• Flows: Create, update, and visualize workflows.
• Nodes: Create, update, delete, and resolve workflow nodes.
• Executions: Start, monitor, and cancel flow executions.
• Plugins: Manage custom functionality enhancements.
• Schedules: Handle scheduled flow executions.

For complete details on the API endpoints, please refer to the API Endpoints Documentation.

Future Enhancements

Based on the project roadmap and current structure, planned enhancements include:

• Add authentication and authorization mechanisms
• Improved error handling, validation, and logging
• Enhanced monitoring and real-time updates (e.g. WebSocket integration)
• Integration with external services (e.g. Telegram bot for human task notifications)
• Advanced workflow features (e.g. fallback nodes, loops).
• Workflow templating, export, and import

Contributing

We welcome contributions to help improve Flowlitics Backend. Whether you’d like to donate financially or contribute code, your support is greatly appreciated. For detailed guidelines on how to contribute, please see our CONTRIBUTING.md file.

Thank you for your support! ❤️

License

This project is licensed under the MIT License.