Question answering over docs: How to reuse document context for multiple queries?

[clstaudt]

Using the "Getting started" documentation, I managed to successfully run this example for question answering over documents:

from langchain.chains.question_answering import load_qa_chain
chain = load_qa_chain(llm, chain_type="stuff")
chain.run(input_documents=docs, question=query)

However, resending the documents to the API for every question is expensive. From browsing the documentation it was not clear to me how to send the documents once and ask several questions.

[roeekl]

Hi, did you find any answer?

This feature, together with automatic initialization will be very useful when using small documents or when GPT-4 with 32k tokens will be available.

[clstaudt]

To my knowledge: GPT-like models have no working memory and generate based on the context provided with each request. In the case of chat models, as much of the chat history as possible is sent with each new request.

The current state of things seems to be retrieval-augmented generation, in which a search and retrieval engine is put into the system to retrieve relevant document parts and send them with the request.