How to save vector database in disk

[talhaanwarch]

Hi, How can i save milvus or any other vector database to disk so i can use it latter.

from langchain.vectorstores import Milvus

vector_db = Milvus.from_documents(
    docs,
    hfemb,
)

If i want to use vector_db again for similarity_search(query) how can i load it again.

[oddrationale]

I'm not familiar with Milvus, but you can save your vectorstore to disk on Chroma and FAISS.

[rodgermoore]

I have the same question. How can I construct a Milvus object to use for search after already having loaded the data into the Milvus database?

[rodgermoore]

I'm trying to rewrite the milvus.py VectoreStore to make this happen.

[talhaanwarch]

@rodgermoore please look into the haystack discussion, it will help. actually mivlus store meta data in sqlite, if you got the same data and you compare it with existing meta data, you just need to prevent it overwriting the existing one

[rodgermoore]

@rodgermoore please look into the haystack discussion, it will help. actually mivlus store meta data in sqlite, if you got the same data and you compare it with existing meta data, you just need to prevent it overwriting the existing one

Thanks for the tip! @talhaanwarch but I got it working using the VectorStore now. It is a quick and dirty solution though: https://github.com/rodgermoore/langchain/blob/master/langchain/vectorstores/milvus.py

[rodgermoore]

@rodgermoore please look into the haystack discussion, it will help. actually mivlus store meta data in sqlite, if you got the same data and you compare it with existing meta data, you just need to prevent it overwriting the existing one

Thanks for the tip! @talhaanwarch but I got it working using the VectorStore now. It is a quick and dirty solution though: https://github.com/rodgermoore/langchain/blob/master/langchain/vectorstores/milvus.py

Ooops... I got in contact with @filip-halt and he pointed out you can use the drop_old boolean for this! Just set it to False and you can construct the Vectorstore for search only.

[talhaanwarch]

@rodgermoore please look into the haystack discussion, it will help. actually mivlus store meta data in sqlite, if you got the same data and you compare it with existing meta data, you just need to prevent it overwriting the existing one

Thanks for the tip! @talhaanwarch but I got it working using the VectorStore now. It is a quick and dirty solution though: https://github.com/rodgermoore/langchain/blob/master/langchain/vectorstores/milvus.py

Ooops... I got in contact with @filip-halt and he pointed out you can use the drop_old boolean for this! Just set it to False and you can construct the Vectorstore for search only.

Is not drop_old set to False by default? I dont think it make it persistent. I am not sure, i have to check. Is the solution, you find is some thing different ?

[filip-halt]

Drop_old == True will overwrite the old collection stored there. Without drop old you will pull the old collection in and use it. It should be persistent.

[pachgadehardik]

If we load the Milvus collection, then we can't apply similarity_search(query) as the function require vector_store, is there any way where the collection can be typecasted to vectorstore, so that it can be used as a retriever in chain calls? (Thinking out loud here, please let me know whether my understanding is correct abt this)

[jupyterjazz]

Hi @talhaanwarch, here's how you can do it via DocArrayHnswSearch:

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import DocArrayHnswSearch

embeddings = OpenAIEmbeddings()
docs = ... # create docs

# everything will be stored in the directory you provide, hnswlib_store in this case
db = DocArrayHnswSearch.from_documents(docs, embeddings, work_dir='hnswlib_store/', n_dim=1536)

# then you can just provide the same dir and initialize again
# small trick: you can pass empty list instead of docs
saved_db = DocArrayHnswSearch.from_documents([], embeddings, work_dir='hnswlib_store/', n_dim=1536)

[talhaanwarch]

Initially i was doing the same, but latter learnt that i dont need to use from_document when using already created db

…

On Fri, 16 Jun 2023, 4:41 pm Saba Sturua, ***@***.***> wrote: Hi @talhaanwarch <https://github.com/talhaanwarch>, here's how you can do it via DocArrayHnswSearch: from langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.vectorstores import DocArrayHnswSearch embeddings = OpenAIEmbeddings()docs = ... # create docs # everything will be stored in the directory you provide, hnswlib_store in this casedb = DocArrayHnswSearch.from_documents(docs, embeddings, work_dir='hnswlib_store/', n_dim=1536) # then you can just provide the same dir and initialize again# small trick: you can pass empty list instead of docssaved_db = DocArrayHnswSearch.from_documents([], embeddings, work_dir='hnswlib_store/', n_dim=1536) — Reply to this email directly, view it on GitHub <#4077 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5FYO5FS66OMGN7FRQTTMTXLRA6TANCNFSM6AAAAAAXVEQWWM> . You are receiving this because you were mentioned.Message ID: ***@***.***>