Common implementation of Dense and Sparse vector (BM25, TFIDF, SPLADE ...) for Hybrid Search

[Freezaa9]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Propose a common way to specify if a vector db is made to contain Dense or Sparse vector (or both, depending on implementation).

Propose a list of Sparse embedding solution to be used in Sparse vector db : BM25, TFIDF, SPLADE ...

Motivation

Using a mix of dense and sparse vector is becoming a a must to achieve better RAG, and Langchain is lacking of a common approach to achieve this.

Multiple vector db are implementing Dense and Sparse vector in their own way
https://python.langchain.com/v0.2/docs/integrations/retrievers/milvus_hybrid_search/
https://python.langchain.com/docs/integrations/retrievers/qdrant-sparse/

And more are compatible but not implemented yet in Langchain: PGvector, Google Vertex Search ...

Proposal (If applicable)

A Hybrid Search workflow would be:

1. Create vector db A on chroma with Dense embedding and Vertex embedding
2. Create vector db B on pgVector with Sparse embedding and BM25
3. Add rerank with the Ensemble retriever with vector db A and vector db B
4. retrieve