Faiss Document search Score for relevant and irrelevant queries issue.

[VikRathod]

I developed a chat app using faiss document store to find similar documents for a query. It retrieves top 5 docs, with the highest as "High_score" and the lowest as "minimum score." A condition is applied, requiring values to meet specific criteria (e.g., minimum_value_1 >= 0 and < 1.04, maximum_value_1 <= 1.45).
The application uses a condition to assess query correctness and retrieve relevant documents accordingly. Unfortunately, it classifies some relevant queries, leading to a failure in providing answers for queries that should be considered relevant.

Is there any solution that you guys have, or i am missing something in the process.

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're experiencing an issue with the Faiss document store in your chat application. Specifically, the problem lies in the scoring system for retrieving similar documents based on a query. You've set a condition for the scores to fall within a certain range, but it seems that some relevant queries are being classified as irrelevant, which prevents the application from providing the correct responses.

One potential solution to this issue could be to adjust the threshold values you've set for the minimum and maximum scores. The scores returned by the similarity_search_with_score method are L2 distances, which means a lower score indicates a better match. If your threshold values are not set correctly, this could lead to relevant queries being classified as irrelevant.

You could try running a set of known relevant and irrelevant queries, observe the scores, and adjust your thresholds based on these observations. This might help you to better classify the queries.

Another approach could be to use the similarity_search_with_score method with a filter. This method allows you to filter the documents based on metadata. If you have metadata that can help distinguish between relevant and irrelevant queries, you can use it to filter the results. Here's an example of how you can do this:

results_with_scores = db.similarity_search_with_score("foo", filter=dict(page=1))
for doc, score in results_with_scores:
    print(f"Content: {doc.page_content}, Metadata: {doc.metadata}, Score: {score}")

In this example, the method is filtering the results to only include documents where the metadata field 'page' is equal to 1. You can replace 'page' with any metadata field that is relevant to your application.

Additionally, you can use a score threshold with the similarity_search_by_vector and similarity_search_with_score_by_vector methods. This allows you to only return documents with a score below a certain value. Here's an example:

query_vec = FakeEmbeddings().embed_query(text="foo")
output = docsearch.similarity_search_by_vector(query_vec, k=2, score_threshold=0.2)

In this example, the method is only returning documents with a score below 0.2. You can adjust the score threshold based on your application's needs.

Please note that these are potential solutions and might not completely resolve your issue. If you find these solutions helpful and believe they could benefit other users, we encourage you to make a pull request to update the documentation. This way, you can share your experience and solutions with the broader community.

Sources

• docs/extras/integrations/vectorstores/faiss.ipynb
• libs/langchain/tests/integration_tests/vectorstores/test_faiss.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.