IPFS_Faiss_Py

This module is meant to search through embeddings datasets, either through ones that are monolithic, or ones that are sharded, or alternatively to call qdrant/elastic search. By default when the KNN indicies are being sharded using K means clustering, the maximum shard size is set to 4096 rows or 25MB. Iterative embeddings search is done by by first matching the query vector to the centroids list, then requesting the cluster indicated by the closest centroid, then querying the items in that centroid, for the top 64 candidates, and then collecting the chunks from those candidates, and searching through those to collect the top 64, and then reranking the top 64 results to provide the final selection into the LLM context window.