No clear v2 option for ConversationChain with ConversationSummaryBufferMemory, as RunnableWithMessageHistory has critical gap

[chrislrobert]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_openai import ChatOpenAI, AzureChatOpenAI
from langchain_core.prompts import SystemMessagePromptTemplate, HumanMessagePromptTemplate, ChatPromptTemplate
from langchain.memory import ConversationSummaryBufferMemory, ChatMessageHistory
from langchain.chains import ConversationChain

summarize_llm = ChatOpenAI(
	temperature=self.temperature,
	verbose=False,
	model_name=self.summarize_model,
	openai_api_key=self.openai_api_key,
	tiktoken_model_name=self.tiktoken_model_name
)
conversation_llm = ChatOpenAI(
	verbose=False,
	temperature=self.temperature,
	model_name=self.evaluation_model,
	openai_api_key=self.openai_api_key,
	tiktoken_model_name=self.tiktoken_model_name
)

# construct starting history
history = ChatMessageHistory()
if starting_chat_history is not None:
	for (human_message, ai_message) in starting_chat_history:
		history.add_user_message(human_message)
		history.add_ai_message(ai_message)

# requirement for ConversationChain history to work the way we want
return_messages_preference = False

# construct memory
memory = ConversationSummaryBufferMemory(
	llm=summarize_llm,
	chat_memory=history,
	max_token_limit=max_history_tokens,
	output_key='answer',
	memory_key='chat_history',
	ai_prefix="Assistant",
	human_prefix="Human",
	return_messages=return_messages_preference,
	tiktoken_model_name=self.tiktoken_model_name)

# prune starting history (if any)
if starting_chat_history is not None:
	memory.prune()

# assemble prompts
system_message_prompt = SystemMessagePromptTemplate.from_template(system_prompt)
human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)
chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])

# create conversation chain
llm_chain = ConversationChain(
	llm=conversation_llm, prompt=chat_prompt, memory=memory, input_key="question", output_key="answer"
)

Description

I have an excellent series of chatbots and automated workflows built with LangChain, all of which rely on ConversationChain and ConversationSummaryBufferMemory, as in the example code I shared. I have looked over the v2 migration guidance in https://python.langchain.com/v0.2/docs/how_to/migrate_chains/ many times, and I have spent hours searching for examples and guidance. However, I cannot find any clear way to either:

1. Use ConversationSummaryBufferMemory in a Runnable (e.g., via RunnableWithMessageHistory).
2. Use some alternative that helps to similarly keep within reasonable token limits.

What I seem to be missing from all of the v2 guidance, examples, and docs is this: how can you control how much of the context window is dedicated to verbatim conversation vs. progressively-summarized earlier conversation? All of the examples I see seem to just include 100% of the verbatim conversation, as if there are no limits on context windows (or cost or performance considerations). The flexible memory approach taken in v1 was quite necessary for any application that can entail over-long conversation histories, and I'm failing to see how this is meant to be handled in v2.

Is there a way to utilize ConversationSummaryBufferMemory within a RunnableWithMessageHistory framework?

Chris

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000
Python Version: 3.10.9 (v3.10.9:1dd9be6584, Dec 6 2022, 14:37:36) [Clang 13.0.0 (clang-1300.0.29.30)]

Package Information

langchain_core: 0.1.52
langchain: 0.1.16
langchain_community: 0.0.33
langsmith: 0.1.38
langchain_openai: 0.1.6
langchain_text_splitters: 0.0.1

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[dosubot[bot]]

To utilize ConversationSummaryBufferMemory within a RunnableWithMessageHistory framework in LangChain v2, you can follow these steps to control the context window for verbatim conversation versus progressively-summarized earlier conversation:

1. Initialize ConversationSummaryBufferMemory: Create an instance of ConversationSummaryBufferMemory with the desired max_token_limit.

2. Integrate with RunnableWithMessageHistory: Use the ConversationSummaryBufferMemory instance to manage the conversation history within the RunnableWithMessageHistory framework.

3. Save and Prune Context: Use the save_context method to save the conversation context and automatically prune the buffer if it exceeds the token limit.

Here is an example code snippet demonstrating this integration:

from langchain.memory.summary_buffer import ConversationSummaryBufferMemory
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI

# Initialize ConversationSummaryBufferMemory
memory = ConversationSummaryBufferMemory(max_token_limit=2000)

# Store for session histories
store = {}

def get_session_history(session_id: str) -> InMemoryChatMessageHistory:
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

# Example function to simulate a conversation within RunnableWithMessageHistory framework
def run_conversation(runnable, inputs, session_id):
    # Load existing memory
    context = memory.load_memory_variables(inputs)
    
    # Run the conversation with the current context
    outputs = runnable.run(inputs, context)
    
    # Save the new context
    memory.save_context(inputs, outputs)
    
    return outputs

# Example usage
class ExampleRunnable:
    def run(self, inputs, context):
        # Simulate processing inputs with context
        return {"response": "This is a response based on the context."}

# Create an instance of the runnable
runnable = ExampleRunnable()

# Simulate conversation inputs
inputs = {"user_input": "Hello, how are you?"}

# Run the conversation
outputs = run_conversation(runnable, inputs, session_id="1")
print(outputs)

In this example:

• ConversationSummaryBufferMemory is initialized with a max_token_limit of 2000 tokens.
• The run_conversation function simulates running a conversation within the RunnableWithMessageHistory framework, loading the existing memory, running the conversation, and saving the new context.
• The ExampleRunnable class simulates a runnable that processes inputs with the given context.
• The get_session_history function manages session-specific histories, ensuring that each session maintains its own context.

This setup ensures that the conversation context is managed efficiently, with earlier parts of the conversation being summarized progressively while maintaining a verbatim context window for recent interactions. The ConversationSummaryMemory class from the LangChain library provides the necessary methods to handle summarization and context management, ensuring that the prompt input variables are consistent and the buffer is updated correctly ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[chrislrobert]

@dosu, that answer doesn't make any sense, sorry. But it's pretty consistent with the lack of clarity around the memory module in LangChain v2.

[dosubot[bot]]

This is a tough nut to crack, could really use your help on this, @eyurtsev.

To continue talking to Dosu, mention @dosu.

[chrislrobert]

FWIW, ConversationSummaryBufferMemory does have a BaseChatMessageHistory in chat_memory, so one might think, okay, you can just use RunnableWithMessageHistory and, in the callback, return that chat_memory. The problem is that that only includes the verbatim part of recent conversation, no earlier summary. To include the earlier summary, you can call load_memory_variables(), which will organize the summary and verbatim bits into text to use for the conversation history in the prompt. That overall structure seems at odds with RunnableWithMessageHistory, which is part of why it seems so difficult to integrate the v1 memory module with the v2 Runnable approach. But again, maybe I'm missing something obvious? Everybody in v2 land can't be living without conversation histories robust to long conversations, can they? That would seem crazy.

[chrislrobert]

Using a few examples I found here and there (seemingly removed from the LangChain cookbook/docs since no links work), I was able to create a runnable version of a memory object like so:

        runnable_memory = RunnablePassthrough.assign(
            chat_history=RunnableLambda(memory.load_memory_variables) | itemgetter("chat_history"),
        )

But then I still have to call save_context() manually after every chain invocation. #16525 seems to be trying to make it automated end-to-end the way old LangChain ConversationChains used to work, but it looks abandoned. Honestly, I'm finding this all extremely perplexing. Maybe LangChain v0.1 serves chatbots and other chains that require sensible handling of conversation history, and v0.2 is for agents with infinite context windows?

[NicolasCampana]

@chrislrobert, I was able to do something like it using the RunnableWithMessageHistory with DynamoDBChatMessageHistory with a small history_size, for example 10 or 20 messages. Something like this wouldnt it be enough?
I also noticed that DynamoDBChatMessageHistory inherits from BaseChatMessageHistory and ConversationSummaryBufferMemory from BaseChatMemory.

[chrislrobert]

No, there are two problems with that: 1. Even a small number of messages could be more than the token budget for conversation history 2. Only remembering the last x messages in a conversation can lead to poor UX, as earlier details are lost The various strategies supported within the memory module resolve combinations of both. The history module, in contrast, seems over-simplistic for production applications that must perform well in long conversations.

…

On Tue, Jul 23, 2024 at 5:52 PM Nicolas Porto Campana < ***@***.***> wrote: @chrislrobert <https://github.com/chrislrobert>, I was able to do something like it using the RunnableWithMessageHistory with DynamoDBChatMessageHistory with a small history_size, for example 10 or 20 messages. Something like this wouldnt it be enough? I also noticed that DynamoDBChatMessageHistory inherits from BaseChatMessageHistory and ConversationSummaryBufferMemory from BaseChatMemory. — Reply to this email directly, view it on GitHub <#24562 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACB3IHQKVTEN7WMWBT6XLVDZN3GCLAVCNFSM6AAAAABLKYR6CWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJTGEZDOOI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[chrislrobert]

@ccurme, thanks for that suggestion. I somehow missed that example! While that particular approach would technically work for ConversationSummaryBufferMemory, it would be very slow and expensive for long conversations because it would be starting from scratch to re-summarize from the beginning every time history is needed. In my case, I wouldn't want it to be re-summarizing the same early conversation over and over within a chain — or even within a persistent websocket connection (my chatbot uses websockets to provide a fast and efficient UX, but it does also support resuming from disk whenever connections are lost/re-established).

I'm guessing that a custom history implementation that uses the memory object behind the scenes would be the only clean and efficient approach? Not sure if somebody has already done that.

[ccurme]

Hi @chrislrobert, appreciate you highlighting this gap with RunnableWithMessageHistory.

There's a LangGraph chatbot implementation that summarizes message history demonstrated in this guide. Wanted to run this by you as another option. Interested in your feedback.

Thanks again.

[chrislrobert]

Thanks, @ccurme, for sharing that. As a LangGraph-based POC, that makes sense, though in a real implementation I would tend to focus on token counts vs. number of messages, just given massive variation in message lengths and the fact that it's context windows, cost, and performance (all of which relate to token counts) being what ultimately matters.

But look, technical feasibility isn't really in question here, as ConversationSummaryBufferMemory has demonstrated reasonable performance within the v0.1 LangChain framework. And in principle, you can build custom v0.2 chains to perform similarly. I guess I just don't understand why everybody should be re-inventing the wheel here vs. having this be core functionality within the v0.2 memory module.

At the end of the day, it's a bit a question for the LangChain community: how production-robust should the platform be for chatbot applications? In my view, how chatbots deal with conversation history is absolutely critical for production performance, but can be easily papered over in testing or POC implementations.

I've seen reviews of LangChain that suggest it's good for rapid POCs but shouldn't be used in production, and dropping stuff like ConversationSummaryBufferMemory is consistent with either not caring about production use or not caring about chatbot applications (vs., e.g., more "agentic" applications). Either way, maybe v0.2 memory support for buffered+summarized conversation history just isn't important to the maintainers, in which case, yeah, folks will just need to transition to other frameworks or build custom solutions between now and when ConversationSummaryBufferMemory is fully deprecated.

[hwchase17]

Hi @chrislrobert - thanks for your thoughtful comments here. Jumping in with some thoughts:

At a high level, one of the big benefits of LangGraph is the support for persistence, which enables more robust, production ready, and controllable memory implementations. I would argue most of LangChain's memory modules with the exception of RunnableWithMessageHistory are not really production ready, due to a combination of lack of persistence and lack of multi-tenancy support. Memory is a HUGE component of LLM apps (as you point out).

Since LangGraph has great memory support, we want to use that as the basis for chains that require memory going forward (again, with the exeption of RunnableWithMessageHistory, which is a simple yet robust/performant implementation)

For things that make sense, we're trying to migrate over to LangGraph. Agents was one example of that. Memory is another, and we want to do a big push on memory this month. Our current thought is to show how implement all of these memory modules with LangGraph. They will then be more production ready (persistant, multi-tenant) and customizable.

I'd love to do that in a way that is intuitive and an easy bridge. What would make this the best experience possible?

[dokato]

I also agree that this migration could be handled better. I used ConversationSummaryBufferMemory with RetrievalQA like so:

    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        chain_type_kwargs={
            "prompt": prompt,
            "memory": memory,
        },
    )

and none of the langgraph examples shows an easy drop in replacement.

[chrislrobert]

Thanks, @hwchase17, for helping to make the roadmap here a bit more clear.

Personally, I expected LangChain to handle the LLM interface in a first-class way, and I was perfectly happy with how I addressed persistence in my app — just as I was happy with how I handled websockets and streaming, balancing privacy and compliance needs with UX and performance concerns in how conversation histories were stored, UI concerns for streaming markdown and auto-scrolling, etc., etc. Obviously, I understand the instinct to build increasing functionality, convenience, and elegance into the core library, and I certainly appreciate the more consistent support for async, streaming, and more.

In terms of the experience as a LangChain user, I guess I'd just say that ideally, when you announce a deprecation, there'd be a bit more clarity around how to migrate while maintaining as-good-or-better functionality. This lack of a clear path for ConversationSummaryBufferMemory has cost me a bunch of time and worry, even to the point of making me question whether I should have adopted LangChain for my application. My current codebase is a messy mix of v0.1 and v0.2 code, with a lot of awkwardness and uncertainty around the bits that continue to rely on v0.1. I set a high bar for QA and production quality, and it feels harder and harder to maintain that bar as LangChain evolves.

A related issue is the web of interdependencies between the LangChain and LangGraph packages with vendor packages like OpenAI and Azure. Being forced to update one package for a vulnerability or fix, and that forcing then a large number of cascading dependency updates — some of which introduce regressions, incompatibilities, or deprecations — is nothing unique to the LangChain ecosystem. But, as you expand the footprint of core functionality, introduce additional integrations, and split functionality across, e.g., the LangChain and LangGraph packages, the challenges might grow at a nonlinear rate. As an existing LangChain user, it's creating increasing drag on both my velocity and my quality, both of which cause regular "is it time to throw in the towel?" discussions.

At the end of the day, though, I get it: it's always a challenge to balance the needs of early adopters and existing users with those of new users and growth. You guys have worked extremely hard and provided extraordinary value at an extraordinary speed. So I appreciate that, and I am looking forward to what sounds like a fuller suite of very robust and flexible v0.2 conversation-history/context approaches.

Thanks again.