Why langchain provides context for answering in the system prompt and not in the 'user' prompt?

[lfoppiano]

Hi all, I'm wondering why the implementation of the RAG in langchain add the context from the vector db in the system prompt instead of the user prompt?
I'm trying to understand. Since at each question the system prompt is re-initialised, i wonder what is the advantage of such approach. AFAIK the system prompt is supposed to be repeated with some short instructions that are common to all the prompts that will be provided, isn't it?

Thank you in advance
Luca

[dosubot[bot]]

🤖

Hi Luca,

The implementation of the RAG in LangChain adds the context from the vector db in the system prompt instead of the user prompt because the system prompt is designed to be dynamic and adaptable to the context of the current question.

In the code, the StuffDocumentsChain is used to combine the documents. This chain takes the language model chain (llm_chain), the document variable name (document_variable_name), and the document prompt (document_prompt) as arguments. The document prompt is defined as Context:\n{page_content}, where page_content is the context retrieved from the vector database.

document_prompt = PromptTemplate(
    input_variables=["page_content"], template="Context:\n{page_content}"
)
combine_documents_chain = StuffDocumentsChain(
    llm_chain=llm_chain,
    document_variable_name="context",
    document_prompt=document_prompt,
    callbacks=callbacks,
)

This design allows the system to dynamically generate the prompt based on the context retrieved from the vector database for each question. If the context was added to the user prompt, it would be static and wouldn't change based on the context of the current question. This would limit the ability of the system to provide accurate and context-aware responses.

The StuffDocumentsChain in the LangChain framework is a class that combines multiple documents into a single context and passes it to a language model for processing. It takes a list of documents and combines them into a single string. This is done by formatting each document into a string with the document_prompt and then joining them together with document_separator. The combined string is then added to the inputs with the variable name set by document_variable_name. These inputs are then passed to the llm_chain, which is a language model chain that processes the combined documents.

I hope this clarifies your question. If you have any more questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/chains/retrieval_qa/base.py
• libs/langchain/langchain/chains/combine_documents/stuff.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[lfoppiano]

This design allows the system to dynamically generate the prompt based on the context retrieved from the vector database for each question. If the context was added to the user prompt, it would be static and wouldn't change based on the context of the current question. This would limit the ability of the system to provide accurate and context-aware responses.

🤔 Are you sure about this?