Adding Workspace Support to LinghtRAG Server

[danielaskdd]

Background

• Support multiple document sets at one server
• Create and destroy document set easy and quick
• Temporary or private document set
• Single file RAG
• Ollama models emulation for multiple document sets
• Act as a Tool(MCP Server) to AI agent with multiple document sets

Challenge

• API Compatible
• Minimize UI changes at transition
• Method of storage isolation for each storage implentation
• Cache isolation
• Parallel document processing for difference document sets
• LLM sessions reserved for query to avoid frontend blockage

[ParisNeo]

Hi there.
Good idea.
This would mean a substantial upgrades.
And it won't be stable right away, so I suggest creating a new branch on which we can start this work.

We will need alot of changes.
We need to add a new storage to store user credentials and make a robust and secure architecture.

[niallam22]

Alternative Approach: API Key-Bound Collections

Background

Utilize the existing workspace field already present in database tables (I have only checked postgres)
Map collections to API keys rather than user credentials
Maintain backend service model with simpler authentication requirements

Benefits

Minimal schema changes (only needs a collections-to-API-key mapping table)
No need for user credential management within LightRAG
Preserves LightRAG as a backend service that can be integrated with any authentication system

Implementation Strategy

Create a lightweight collection management layer that maps collection IDs to workspaces
Extend API endpoints to accept collection ID as a parameter
Add validation to ensure collections are only accessible by their associated API key

Challenges Addressed

API Compatible: Extends existing API with minimal breaking changes
Minimize UI Changes: UI only needs to pass collection ID parameter
Storage Isolation: Uses existing workspace field in all tables

This approach would require significantly less architectural change than implementing a full user credential system, while still providing robust multi-document set capabilities. It maintains LightRAG as a flexible backend service that can be integrated with various frontend authentication systems rather than building that directly into the core.

[danielaskdd]

Support rapid creation of temporary Workspaces of previously indexed files, by utilizing LLM caching, and maintain document chunks. This means the creation process will be quick.