ConversationalRetrievalChain.from_llm with SystemMessage

[dquevedor]

Hi guys, I'm working on a chatbot that answers based on one document, it's based on a ConversationalRetrievalChain. I wanna know if there's a way to give "identity" to the bot and some instructions with a SystemsMessage or maybe with other aproach. This is my testing code:

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.document_loaders import TextLoader
from langchain.memory import ConversationBufferMemory

loader = TextLoader("./QA.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0)
documents = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(documents, embeddings)

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever(), memory=memory)

query = "what's your name?"
result = qa({"question": query})
print(result["answer"])

[neebs12]

From what ive seen in the code, this isn't possible as there are a lot of prompts that happen in the background. What you could do is to set up a basic sequential chain that essentially rewords this current chain output to the system prompt that you feed it.

Say you have result["answer"], feed it to another chain with the system prompt that gives the answer a flavour. IE: "In the tone of... ,"

[oddrationale]

You can try overwriting the default system template. See my answer here for #3975.

Hope that helps!

[dquevedor]

Nice, thanks. I made a little changes to my code and it ended up like this:

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.document_loaders import TextLoader
from langchain.memory import ConversationBufferMemory
from langchain.prompts import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)

system_message_prompt = SystemMessagePromptTemplate.from_template(
    "Your name is Roboto, you are a nice virtual assistant. The context is:\n{context}"
)
human_message_prompt = HumanMessagePromptTemplate.from_template(
    "{question}"
)

loader = TextLoader("./QA2.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0)
documents = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(documents, embeddings)

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

chat_history = [
    ("my name is David, what's your name?", "Hello David! My name is Roboto. How can I assist you today?"),
]

# qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever(), memory=memory)
chain = ConversationalRetrievalChain.from_llm(
    llm=ChatOpenAI(temperature=0),
    retriever=vectorstore.as_retriever(),
    memory=ConversationBufferMemory(
        memory_key="chat_history",
        return_messages=True,
    ),
    combine_docs_chain_kwargs={
        "prompt": ChatPromptTemplate.from_messages([
            system_message_prompt,
            human_message_prompt,
        ]),
    },
)


query = "do you know my name?"
result = chain({"question": query, "chat_history": chat_history})
print(result["answer"])

But it appears that the chat_history don't work no (the chat_history variable brings all the past messages from a database):

I'm sorry, but I don't know your name. As a virtual assistant, I don't have access to personal information unless you provide it to me. How may I assist you today?

What could be?

[yysturdy]

您好，我也遇到这样的问题了，请问一下你是怎么解决的

[oddrationale]

So couple things here.

Since we are using the chat models, you should update the chat history like this:

memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True,
)
memory.chat_memory.add_user_message("my name is David, what is your name?")
memory.chat_memory.add_ai_message("Hello, David! My name is Roboto. How can I help you today?")

Then when you create your chain, make sure to re-use your memory variable instead of overwriting it with a new memory:

chain = ConversationalRetrievalChain.from_llm(
    ...
    memory=memory,
    ...
)

Then you can run your query with just

chain.run("do you know my name?")

Secondly, I would recommend setting verbose = True to better understand what is going on under the hood:

chain.verbose = True
chain.combine_docs_chain.verbose = True
chain.combine_docs_chain.llm_chain.verbose = True
chain.combine_docs_chain.llm_chain.llm.verbose = True
chain.question_generator.verbose = True
chain.question_generator.llm.verbose = True

There are two different LLM calls happening under the hood. First, the question_generator runs to summarize the previous chat history and new question into a stand-alone question. Then the combine_docs_chain.llm_chain runs with the stand-alone question and the context from the vectorstore retriever. The chat history is not sent to the combine_docs_chain at all, since it was summarized by the question_generator. You'll see this more clearly when you run the chain with verbose=True.

So how to solve this? Well, you might consider using a different chain or adjusting the chain so that the chat history is passed every time. Or you can consider using a conversational agent instead with a RetrievalQAChain as a Tool.

[dquevedor]

Thanks for the reply and the explanation, it's more clear for me how the , I'm trying to build and API endpoint capable of receive a question and give a response based on some .txt documents and the oldest messages from the chat (these are stored on a mongodb) so, with a conversational agent is possible to archive this kind of chatbot? I mean, based on your expertice wich aproach is the best for this kind of solution?

[oddrationale]

Best to experiment and see what gives you better results. A conversational agent will access the conversation history and only use the .txt documents when it thinks that the query is related to the Tool description.

On the other hand, if you want to respond based on the conversation history and document context simultaneously, then might want to try a custom chain and prompt.

[dquevedor]

Yeah, I need both sources to be considered on every chain call, how can customize my chain and/or prompt? Thanks in advance.

[charles-adedotun]

Yeah, I need both sources to be considered on every chain call, how can customize my chain and/or prompt? Thanks in advance.

Were you able to find a solution to this?

[yanghuijava]

This approach also has issues, as the conversation history will lead to a new question as a follow-up question.

[ChrisDelClea]

When intializing the ConversationalRetrievalChain chain there should be a param system_message similar to how it is done in ConversationalChatAgent.