framework to be used by RAG for pipeline

Signed-off-by: Ryan Cook <[email protected]>

Author: cooktheryan

docs/retrieval-augmented-generation/rag-framework.md

@@ -0,0 +1,80 @@
+# RAG  will use Haystack
+
+| Created  | Dec 4, 2024 |
+| -------- | -------- |
+| Authors |  Ryan Cook, Ilya Kolchinsky, Hema Veeradhi |
+| Replaces | N/A |
+| Replaced by | N/A |
+
+## What
+
+This ADR defines the decision on the framework used to support the RAG pipeline for InstructLab. The Emerging Technologies team is pushing for the adoption of Haystack to be used for the framework of the RAG offering. Haystack will handle the data ingestion and retrieval processes for the to be productized RAG solution.
+
+NOTE: Speaking with Peter Staar on Dec 3, 2025 the docling team is aware of the efforts to potentially use Haystack and are already looking into adding the functionality for support of Haystack.
+
+## Why
+
+Multiple options for frameworks currently exist. During our initial analysis, the following options were the only ones to satisfy the basic requirements in terms of functionality, reliability and open-source availability:
+
+- [Haystack](https://haystack.deepset.ai/)
+- [Langchain/Langgraph](https://www.langchain.com/)
+- [LlamaIndex](https://www.llamaindex.ai/)
+- [RAGFlow](https://ragflow.io/)
+
+All of the above offer a variation of a modular pipeline architecture, where users can chain together components (like retrievers, readers, and generators) to process data in different stages.
+
+Out of those, we propose to use Haystack for the following reasons:
+
+1. Focus on RAG. Haystack is a framework specifically targeting RAG use cases and sophisticated RAG indexing and retrieval pipelines. While Langchain and LlamaIndex shine in their own areas, the former is a generalist framework and the latter has a different focus, namely building custom indices over data. Haystack provides functionality that is strongly tailored for RAG and includes a comprehensive library of out-of-the-box solutions for advanced RAG scenarios. As a result, many essential or soon-to-be-essential RAG capabilities can be implemented in a few lines in Haystack but require considerable work to be supported over Langchain or LlamaIndex. Some examples include hybrid retrieval, iterative RAG, HyDE, combining multiple ingestion sources, custom data preprocessing and metadata augmentation. As the decision discussed in this document involves only the RAG component of RHEL AI, we believe that choosing the best RAG framework, as opposed to the best general LLM serving framework, would be more strategically correct.
+
+1. Maturity and stability. Haystack is the most mature, established and stable product among the considered alternatives. It has been around for more time overall (since 2017) and accumulated more mileage. Haystack has an active, sizable and steadily growing community.
+
+2. Extensive vendor support. Haystack natively supports all currently popular vector DBs and provides dedicated backends for incorporating them into its pipelines. Additionally, Haystack supports multiple models and model providers out-of-the-box.
+
+3. Enterprise-level performance. Haystack is designed for production-grade scalability, supporting distributed systems and high-throughput applications. Moreover, and in contrast to the alternatives (of which only LlamaIndex showcases similar performance and scalability), Haystack is specifically optimized for efficient search and retrieval in the RAG setting.
+
+4. Ease of use and documentation. Being strictly focused on RAG as opposed to taking a generalist approach, the learning curve of Haystack is less steep than that of Langchain. At the same time, Haystack offers extensive documentation and tutorials which are more well-organized and easy to use than those of LlamaIndex.
+
+1. Extending the previous point, Haystack can be seen as a middle ground between Langchain and LlamaIndex, sharing their benefits while only partially inheriting their drawbacks. Like the former, Haystack enables building custom flows and pipelines. Unlike Langchain though, Haystack does not try to be too abstract and general, strictly focusing on RAG and document search instead. As a result, Haystack is more straightforward to use, especially for users looking to implement custom and highly non-standard scenarios. On the other hand, like LlamaIndex, Haystack's performance is optimized towards data retrieval and indexing, but it offers a higher degree of flexibility and better interfaces for custom use cases.
+
+## Goals
+
+- We need to identify a framework that will be used to help support the RAG work stream. This is one of many pieces involved in the RAG work stream but identifying the framework used to index and retrieve data is step 1 of the larger picture.
+
+## Non-goals
+
+- What can we ignore when making this decision?
+
+## Decision
+
+Upon acceptance of this integration our next step is to include additional ADRs for the subsequent components required for the RAG pipeline. Next up will be decisions on Milvus and containerized/non-containerized offerings of that solution.
+
+Upon denial of this integration the team will need to go back and evaluate alternative technologies and ensure they meet the needs of the project goal and ensure they meet the larger project plan goals of a configurable RAG pipeline.
+
+## How
+
+A downstream should be generated of the [https://github.com/deepset-ai/haystack](https://github.com/deepset-ai/haystack) project.
+
+## Alternatives
+
+- Langchain/Langgraph. Drawbacks include:
+  - A generalist framework (as opposed to a RAG-focused solution)
+  - Complicated as compared to the alternatives, steep learning curve
+  - Lower performance in large-scale production environments than LlamaIndex and Haystack
+
+- LlamaIndex
+  - Limited flexibility and customization options as compared to the alternatives
+  - Limited out-of-the-box support for building complex, multi-component pipelines as compared to Langchain and Haystack
+  - Documentation is less well-maintained and more difficult to use as compared to the alternatives
+
+- RAGFlow
+  - Limited support for many of the mainstream vector DB providers
+  - Limited scalability as compared to the alternatives
+
+## Risks
+
+N/A
+
+## References
+
+- [https://github.com/deepset-ai/haystack](https://github.com/deepset-ai/haystack)