How can i search with metadata in Milvus (langchain_milvus)

[sunwupark]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from pymilvus import Collection, connections

connections.connect(alias="default", host="localhost", port="19530")
collection = Collection("my-collection")

collection.query(
  expr=f"id in {list}",
  output_fields=["description"]
)

Description

I’m trying to retrieve all documents from Milvus using langchain_milvus by querying with metadata,
such as id >= 1 AND id < 50.

However, it seems there’s no straightforward way to perform metadata-based searches directly within langchain_milvus.
The only option I’ve found so far is leveraging the query method in pymilvus, which allows metadata querying.

I’d like to know if there’s a way to achieve this in langchain_milvus, or if there are any plans to introduce this functionality in the future.

System Info

langchain==0.2.16
langchain-community==0.2.16
langchain-core==0.2.38
langchain-milvus==0.1.4
langchain-openai==0.1.23
langchain-text-splitters==0.2.4

platform: mac
python version: Python 3.9.18

[dosubot[bot]]

Hey @sunwupark! I'm here to help you with bugs, questions, and becoming a contributor. Let's tackle this together! 🤖

To search with metadata in Milvus using langchain_milvus, you can perform a similarity search with a filter on the metadata. Here is an example:

results = vector_store.similarity_search(
    "LangChain provides abstractions to make working with LLMs easy",
    k=2,
    filter={"source": "tweet"},
)
for res in results:
    print(f"* {res.page_content} [{res.metadata}]")

This code snippet performs a similarity search for the query "LangChain provides abstractions to make working with LLMs easy" and filters the results to only include documents where the metadata source is "tweet".

Additionally, if you are using the MilvusCollectionHybridSearchRetriever class, you can leverage its capabilities to perform hybrid searches based on multiple fields and filter the results using boolean expressions. Here is an example of how you might set up and use this class:

from langchain_milvus.retrievers.milvus_hybrid_search import MilvusCollectionHybridSearchRetriever
from pymilvus import Collection

# Initialize the Milvus collection
collection = Collection("your_collection_name")

# Define the retriever
retriever = MilvusCollectionHybridSearchRetriever(
    collection=collection,
    rerank=None,  # Replace with your ranker if needed
    anns_fields=["vector_field_1", "vector_field_2"],
    field_embeddings=[embedding_function_1, embedding_function_2],
    field_search_params=[{"metric_type": "L2", "params": {"nprobe": 10}}],
    field_limits=[10, 10],
    field_exprs=["source == 'tweet'", None],
    top_k=2,
    text_field="text",
    output_fields=["source", "author"]
)

# Perform the search
results = retriever._get_relevant_documents("LangChain provides abstractions to make working with LLMs easy")

for res in results:
    print(f"* {res.page_content} [{res.metadata}]")

This example demonstrates how to set up a hybrid search retriever with multiple vector fields and filter the results based on metadata using boolean expressions.

For more detailed examples and tests, you can refer to the integration tests provided in the langchain_milvus repository. For instance, the test_milvus_with_metadata function shows how to perform a similarity search with metadata:

def test_milvus_with_metadata() -> None:
    """Test with metadata"""
    docsearch = _milvus_from_texts(metadatas=[{"label": "test"}] * len(fake_texts))
    output = docsearch.similarity_search("foo", k=1)
    assert_docs_equal_without_pk(
        output, [Document(page_content="foo", metadata={"label": "test"})]
    )

This test function demonstrates how to set up a Milvus instance with metadata and perform a similarity search, ensuring that the metadata is correctly associated with the search results ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[nhathoang0110]

Have you solved this problem?