How to use RunnableWithMessageHistory and ConversationalRetrievalChain together?

[adityabhattad2021]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.prompts import PromptTemplate
from langchain_google_genai import GoogleGenerativeAIEmbeddings,ChatGoogleGenerativeAI
from langchain.chains import ConversationalRetrievalChain
from langchain.vectorstores.pgvector import PGVector
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_community.chat_message_histories import RedisChatMessageHistory
import google.generativeai as genai
import os
from dotenv import find_dotenv, load_dotenv

from langchain.globals import set_debug

set_debug(True)

load_dotenv(find_dotenv())
genai.configure(api_key=os.getenv('GOOGLE_API_KEY'))

class ChatManager:
    def __init__(self):
        self.connection_string = os.getenv('CONNECTION_STRING')
        self.collection_name = os.getenv('CONNECTION_NAME')
    
    def get_custom_prompt(self):
        prompt_template = """
        You are a very helpful chat bot, that uses the provided context to answer the users questions 
        accurately.
        context: {context}
        
        Question: {question}

        """

        prompt = PromptTemplate(template=prompt_template,input_variables=['question','history'])
        return prompt
    
    def get_retriever(self):
        embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
        store = PGVector(
            collection_name=self.collection_name,
            connection_string=self.connection_string,
            embedding_function=embeddings
        )
        return store.as_retriever()
    
    def get_retrival_qa_chain(self):
        prompt = self.get_custom_prompt()
        retriever = self.get_retriever()
        llm = ChatGoogleGenerativeAI(model="gemini-pro", temperature=0.5, convert_system_message_to_human=True)
        qa_chain = ConversationalRetrievalChain.from_llm(llm=llm, verbose=True, retriever=retriever)
        return qa_chain
    
    def get_chain_with_history(self):
        chain = self.get_retrival_qa_chain()
        chain_with_history = RunnableWithMessageHistory(
            chain,
            lambda session_id: RedisChatMessageHistory(
                session_id=session_id, url="redis://redis-service:6379"
            ),
            input_messages_key="question",
            history_messages_key="chat_history"
        )
        return chain_with_history
    
    def get_response(self, query: str, session_id: str):
        config = {"configurable": {"session_id": session_id}}
        chain_with_history = self.get_chain_with_history()
        res = chain_with_history.invoke({"question": query}, config=config)
        return res["answer"]
    
def get_chat_manager():
    return ChatManager()

Description

I am attempting to integrate RunnableWithMessageHistory and ConversationalRetrievalChain, but I'm encountering issues. RunnableWithMessageHistory successfully passes the question and chat history to ConversationalRetrievalChain, which performs the following tasks:

1. It transforms the chat history and follow-up question into a consolidated follow-up question.
2. This refined question is then forwarded to a retriever, which fetches relevant documents.
3. The refined question, along with the context, is sent to the LLM to generate an output.

However, I'm facing two main problems:

1. The default prompt configuration causes the LLM to translate my question into an entirely different language, as seen here:

. I want to provide a custom prompt to the LLMChain that includes both the context and the user's question.

2. The StuffDocumentChain is generating an error message: Error in RootListenersTracer.on_chain_end callback: KeyError('output').

System Info

langchain==0.1.8
langchain-community==0.0.21
langchain-core==0.1.24
langchain-google-genai==0.0.9

platform windows ubuntu
python version 3.10.13

[dosubot[bot]]

🤖

Hey @adityabhattad2021! Great to see you diving into another challenge with LangChain. How's everything going on your end?

Based on the code you've shared and the issues you're facing, it seems like you're on the right track with integrating RunnableWithMessageHistory and ConversationalRetrievalChain. However, there are a few adjustments you might need to make to resolve the issues you're encountering.

1. Translation of the question into a different language: This could be happening due to the language model interpreting the prompt in an unexpected way. You can try to modify the prompt to make it more explicit that the response should be in the same language as the question. For example, you could add a line like "Answer in the same language as the question." to your prompt template.

2. Error in StuffDocumentChain: The error message Error in RootListenersTracer.on_chain_end callback: KeyError('output') suggests that the StuffDocumentChain is expecting an 'output' key in the result, but it's not finding it. This could be due to the way you're setting up your ConversationalRetrievalChain. Make sure that the output_key attribute of your ConversationalRetrievalChain matches the key that your StuffDocumentChain is expecting. If you're not sure what key it's expecting, you can check the source code or documentation for StuffDocumentChain.

Here's an example of how you might modify your get_retrival_qa_chain method to address these issues:

def get_retrival_qa_chain(self):
    prompt = self.get_custom_prompt()
    retriever = self.get_retriever()
    llm = ChatGoogleGenerativeAI(model="gemini-pro", temperature=0.5, convert_system_message_to_human=True)
    qa_chain = ConversationalRetrievalChain.from_llm(llm=llm, verbose=True, retriever=retriever, output_key='output')
    return qa_chain

In this example, I've added output_key='output' to the ConversationalRetrievalChain.from_llm call. This tells the ConversationalRetrievalChain to use 'output' as the key for its output, which should match what StuffDocumentChain is expecting.

I hope this helps! If you're still encountering issues, could you please provide more details about the exact error messages you're seeing and the context in which they occur? This will help me provide more targeted assistance.

Sources

• libs/langchain/langchain/chains/conversational_retrieval/base.py
• libs/core/tests/unit_tests/runnables/test_history.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[yarastouhi]

were you able to solve this issue? I am currently getting the same error "Error in RootListenersTracer.on_chain_end callback: KeyError('output')" @adityabhattad2021

[adityabhattad2021]

We solved it by not using ConversationalRetrievalChain.

[sathish-simple]

how?