How to filter documents based on a list of metadata in LangChain's Chroma VectorStore?

[abhigrazitti]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

lst = ['doc1', 'doc2', 'doc3']

base_retriever = chroma_db.as_retriever(search_kwargs={'k': 10})

Description

I'm working with LangChain's Chroma VectorStore and I'm trying to filter documents based on a list of document names.

However, I’m not sure how to modify this code to filter documents based on my list of document names. Could anyone guide me on how to achieve this? Any help would be greatly appreciated!

Thank you in advance!

System Info

System Information

OS: Windows
OS Version: 10.0.19045
Python Version: 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)]

Package Information

langchain_core: 0.2.0
langchain: 0.1.20
langchain_community: 0.0.38
langsmith: 0.1.31
langchain_anthropic: 0.1.4
langchain_experimental: 0.0.58
langchain_openai: 0.0.5
langchain_text_splitters: 0.0.1
langgraph: 0.0.53

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

To filter documents based on a list of document names in LangChain's Chroma VectorStore, you can modify your code to include a filter using the where_document parameter. Here's how you can achieve this:

1. Define the list of document names you want to filter by.
2. Modify the as_retriever method to include the filter in the search_kwargs.

Here's an example:

# Define your list of document names
lst = ['doc1', 'doc2', 'doc3']

# Create a filter dictionary to filter by document names
filter_dict = {"name": {"$in": lst}}

# Modify the as_retriever method to include the filter in search_kwargs
base_retriever = chroma_db.as_retriever(search_kwargs={'k': 10, 'filter': filter_dict})

# Now you can use the retriever to search with the filter applied
query = "your search query"
results = base_retriever.invoke(query)
for result in results:
    print(result)

Explanation:

• List of Document Names: The list lst contains the document names you want to filter by.
• Filter Dictionary: The filter_dict specifies that the document names should be in the list lst using the $in operator.
• Search with Filter: The search_kwargs parameter in the as_retriever method is updated to include the filter_dict, ensuring that the search results are filtered based on the document names.

By following this approach, you can filter documents based on a list of document names in LangChain's Chroma VectorStore.

Details

Sources

• libs/community/langchain_community/vectorstores/chroma.py
• libs/partners/chroma/langchain_chroma/vectorstores.py
• libs/community/tests/integration_tests/vectorstores/fixtures/filtering_test_cases.py
• libs/core/langchain_core/documents/base.py
• libs/core/langchain_core/vectorstores.py
• docs/docs/integrations/vectorstores/chroma.ipynb
• docs/docs/how_to/query_constructing_filters.ipynb
• libs/langchain/langchain/retrievers/document_compressors/chain_filter.py
• libs/community/tests/integration_tests/retrievers/document_compressors/test_chain_filter.py
• libs/langchain/langchain/chains/query_constructor/prompt.py
• libs/community/tests/integration_tests/vectorstores/test_chroma.py
• libs/partners/chroma/tests/integration_tests/test_vectorstores.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[abhigrazitti]

Thank You so much :)

[simpliatanu]

this is not working for me, kindly help.
filter_dict = { "keywords": {'$in': 'Movo'}}
retriever = v_store.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={"score_threshold": 0.8, "k": 3, 'filter': filter_dict})

retriever.invoke ("What is Aviva's unique proposition for Movo?")

gives error during invoke :

ValidationError: 1 validation error for QueryRequest
filter.keywords
Extra inputs are not permitted [type=extra_forbidden, input_value={'$in': 'Movo'}, input_type=dict]

[abhigrazitti]

Hi,
Can you please tell me how to filter when we have multiple list of metadata to filter
Thanks

[dosubot[bot]]

To filter documents based on multiple lists of metadata in LangChain's Chroma VectorStore, you can use the $and or $or operators to combine multiple filter conditions. Here's an example:

Example Code

from langchain_community.vectorstores.chroma import Chroma

# Define your lists of metadata
name_list = ['doc1', 'doc2', 'doc3']
author_list = ['Alice', 'Bob']
published_list = [True]

# Create filter dictionaries for each list
name_filter = {"name": {"$in": name_list}}
author_filter = {"author": {"$in": author_list}}
published_filter = {"published": {"$in": published_list}}

# Combine filters using $and or $or operators
combined_filter = {
    "$and": [
        name_filter,
        author_filter,
        published_filter
    ]
}

# Create the retriever with the combined filter
base_retriever = chroma_db.as_retriever(search_kwargs={'k': 10, 'filter': combined_filter})

# Perform the query
query = "your search query"
results = base_retriever.invoke(query)

# Print the results
for result in results:
    print(result)

Explanation

1. Define Metadata Lists: Create lists for each metadata field you want to filter by (e.g., name_list, author_list, published_list).

2. Create Filter Dictionaries: For each list, create a filter dictionary using the $in operator to match any value in the list.

3. Combine Filters: Use the $and operator to combine the individual filters into a single filter dictionary. This ensures that documents must match all the specified conditions. If you want documents that match any of the conditions, use the $or operator instead.

4. Create the Retriever: Pass the combined filter dictionary to the as_retriever method's search_kwargs parameter.

5. Perform the Query: Use the invoke method to perform the query and retrieve the filtered documents.

This approach should help you filter documents based on multiple lists of metadata effectively. If you have any further questions or need additional assistance, feel free to ask!

Details

Sources

• Multiple Filters using Chroma().as_retriever
• Filter out vectorstore by metadata
• Filtering a corpus of text on metadata, before running RetrievalQA
• ConversationalRetrievalChain Vectorstore cannot apply filter to a key with given multiple values
• similarity Search Issue
• libs/community/langchain_community/vectorstores/utils.py
• libs/partners/chroma/langchain_chroma/vectorstores.py
• libs/community/langchain_community/vectorstores/chroma.py
• libs/community/tests/unit_tests/vectorstores/test_utils.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.