Why RetrievalQA.from_chain_type query on my own data(on pinecone) give me response from the outside world?

[lironezra]

Hi all,

I am trying to fetch to my chatbot data from Pinecone with RetrievalQA, when I'm asking not relevant question about our documents on Pinecone I am getting response from outside world and not based on my data.
Here is my code:

` index = pinecone.Index(index_name)

pinecone_vectore_store = Pinecone(index, embeddings.embed_query, "text")
llm = ChatOpenAI(temperature=0, openai_api_key=OPENAI_API_KEY, model_name="gpt-3.5-turbo", verbose=True)

retrival_chain = RetrievalQA.from_chain_type(
   llm=llm, 
   chain_type="stuff", 
   retriever=pinecone_vectore_store.as_retriever(), 
   return_source_documents=True, - I want to see the relevant docs from pinecone
 )

  res = retrival_chain({"query": user_question})`

I asked the chat bot about Leo Messi wife, although we don't have data on Leo Messi on Pinecone index and I'm getting response anyway from outside knowledge base or something.

and this is the response I got:

You can see in the image the the source_documents that returned from Pinecone index is [] which is grate because we don't have this data on our vector store.

So why he don't understand that if we don't have relevant document, you should not search in outside sources?

Thanks guys!