Availability for embedding BGE-M3 model for LanChain Hybrid Search Retrieval

[kagaho]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

With a database in Milvus created with sparse and vector embeddings, as Description:

from milvus_model.hybrid import BGEM3EmbeddingFunction

embedding_function = BGEM3EmbeddingFunction(
    model_name = "BAAI/bge-m3",
    batch_size = 32,
    normalize_embeddings = True,
    use_fp16 = False,
    return_dense = True,
    return_sparse = True,
    return_colbert_vecs = False,

docs = [
    "Artificial intelligence was founded as an academic discipline in 1956.",
    "Alan Turing was the first person to conduct substantial research in AI.",
    "Born in Maida Vale, London, Turing was raised in southern England.",
]

docs_embeddings = embedding_function.encode_documents(docs)

from pymilvus import (
    Collection,
    CollectionSchema,
    DataType,
    FieldSchema,
    WeightedRanker,
    connections,
)

<...>

<Collection>:
-------------
<name>: LangChain_fan
<description>: 
<schema>: {'auto_id': True, 'description': '', 'fields': [{'name': 'doc_id', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}, 'is_primary': True, 'auto_id': True}, {'name': 'dense_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 1024}}, {'name': 'sparse_vector', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 5000}}], 'enable_dynamic_field': False}

pk_field = "doc_id"
dense_field = "dense_vector"
sparse_field = "sparse_vector"
text_field = "text"
sparse_search_params = {"metric_type": "IP"}
dense_search_params = {"metric_type": "IP", "params": {}}

from langchain_milvus.retrievers import MilvusCollectionHybridSearchRetriever 

retriever = MilvusCollectionHybridSearchRetriever(
    collection=collection,
    rerank=WeightedRanker(0.5, 0.5),
    anns_fields=[dense_field, sparse_field],
    field_embeddings=[embedding_function, embedding_function],
    field_search_params=[dense_search_params, sparse_search_params],
    top_k=3,
    text_field=text_field,
)

Description

Hi team,
I need some help trying to use embed model BGE-M3 for Hybrid Search in RAG with MilvusCollectionHybridSearchRetriever class for the Retrieval. I tried to create subclasses but not succeeded. I am using Milvus 2.4.0 as my vectorstore.

BGE-M3 actually creates both sparse and dense vectors:

print("Dense document dim:", embedding_function.dim["dense"], docs_embeddings["dense"][0].shape)
Dense document dim: 1024 (1024,)

print("Sparse document dim:", embedding_function.dim["sparse"], list(docs_embeddings["sparse"])[0].shape)
Sparse document dim: 250002 (1, 250002)

list(docs_embeddings.keys())
['dense', 'sparse']

print(collection)
<Collection>:
-------------
<name>: LangChain_fan
<description>: 
<schema>: {'auto_id': True, 'description': '', 'fields': [{'name': 'doc_id', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}, 'is_primary': True, 'auto_id': True}, {'name': 'dense_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 1024}}, {'name': 'sparse_vector', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 5000}}], 'enable_dynamic_field': False}

Now when we try to create the retriever, I could not find BGE-M3:

from langchain_milvus.retrievers import MilvusCollectionHybridSearchRetriever 

retriever = MilvusCollectionHybridSearchRetriever(
    collection=collection,
    rerank=WeightedRanker(0.5, 0.5),
    anns_fields=[dense_field, sparse_field],
    field_embeddings=[embedding_function, embedding_function],
    field_search_params=[dense_search_params, sparse_search_params],
    top_k=3,
    text_field=text_field,
)

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[23], line 3
      1 from langchain_milvus.retrievers import MilvusCollectionHybridSearchRetriever 
----> 3 retriever = MilvusCollectionHybridSearchRetriever(
      4     collection=collection,
      5     rerank=WeightedRanker(0.5, 0.5),
      6     anns_fields=[dense_field, sparse_field],
      7     field_embeddings=[embedding_function, embedding_function],
      8     field_search_params=[dense_search_params, sparse_search_params],
      9     top_k=3,
     10     text_field=text_field,
     11 )

File [~/anaconda3/lib/python3.8/site-packages/langchain_milvus/retrievers/milvus_hybrid_search.py:49](http://localhost:8888/lab/tree/ELITA/product/ENERGON/anaconda3/lib/python3.8/site-packages/langchain_milvus/retrievers/milvus_hybrid_search.py#line=48), in MilvusCollectionHybridSearchRetriever.__init__(self, **kwargs)
     48 def __init__(self, **kwargs: Any):
---> 49     super().__init__(**kwargs)
     51     # If some parameters are not specified, set default values
     52     if self.field_search_params is None:

File [~/anaconda3/lib/python3.8/site-packages/langchain_core/load/serializable.py:113](http://localhost:8888/lab/tree/ELITA/product/ENERGON/anaconda3/lib/python3.8/site-packages/langchain_core/load/serializable.py#line=112), in Serializable.__init__(self, *args, **kwargs)
    111 def __init__(self, *args: Any, **kwargs: Any) -> None:
    112     """"""
--> 113     super().__init__(*args, **kwargs)

File [~/anaconda3/lib/python3.8/site-packages/pydantic/v1/main.py:341](http://localhost:8888/lab/tree/ELITA/product/ENERGON/anaconda3/lib/python3.8/site-packages/pydantic/v1/main.py#line=340), in BaseModel.__init__(__pydantic_self__, **data)
    339 values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
    340 if validation_error:
--> 341     raise validation_error
    342 try:
    343     object_setattr(__pydantic_self__, '__dict__', values)

ValidationError: 4 validation errors for MilvusCollectionHybridSearchRetriever
field_embeddings -> 0
  instance of Embeddings expected (type=type_error.arbitrary_type; expected_arbitrary_type=Embeddings)
field_embeddings -> 0
  instance of BaseSparseEmbedding expected (type=type_error.arbitrary_type; expected_arbitrary_type=BaseSparseEmbedding)
field_embeddings -> 1
  instance of Embeddings expected (type=type_error.arbitrary_type; expected_arbitrary_type=Embeddings)
field_embeddings -> 1
  instance of BaseSparseEmbedding expected (type=type_error.arbitrary_type; expected_arbitrary_type=BaseSparseEmbedding)
Selection deleted

System Info

"pip freeze | grep langchain"

langchain==0.3.3
langchain-anthropic==0.2.3
langchain-community==0.3.2
langchain-core==0.3.10
langchain-huggingface==0.1.0
langchain-milvus==0.1.5
langchain-ollama==0.2.0
langchain-openai==0.2.2
langchain-text-splitters==0.3.0
langchainhub==0.1.21

Platform (Fedora/Linux)
uname -r 6.1.102-111.182.amzn2023.x86_64