prebuilt createReactAgent can not work with streamEvents

[Wulino]

Bug Report: Stream Events with ChatMessageChunk

Description

I am trying to implement a fully streaming call using this.agentExecutor.stream(xxx)(this is not fully streaming), and everything was working fine initially. However, after making the following changes:

this.agentExecutor = createReactAgent({
  llm: this.chatModel,
  tools,
  messageModifier: prompt,
});
this.agentExecutor.streamEvents(
  { messages },
  {
    version: 'v2',
  },
);

So, is there anyway to achieve Streaming LLM Tokens? I have been struggling with this issue for many days. ReactAgent without streaming is really terrible!!

[jacoblee93]

What model are you using?

[Wulino]

What model are you using?

It's trained By my company, just like ChatGPT. The llm class is based on @langchain/openai/ChatOpenAi, and I have only modified the input and output parameters of the API. So I think it's same as ChatGPT.
I believe the model itself is not the issue; it seems more like there are problems with the entire workflow's support for streamEvents, right? cause both "invoke" and "stream" are working as expected.

By the way, since I don't have a ChatGPT API key, is there anyone who can help me try using ChatGPT or other models to see if there are any issues with createReactAgent and streamEvents? This way, I can figure out exactly where the problem lies.

[jacoblee93]

So I think it's same as ChatGPT. I believe the model itself is not the issue; it seems more like there are problems with the entire workflow's support for streamEvents, right? cause both "invoke" and "stream" are working as expected.

Can you double check this by calling model.streamEvents on your implementation? As far as I know this works for OpenAI

[fiachra]

@Wulino Hey just FYI, I hit this problem using LiteLLM but when I swapped to a direct Azure connection it went away. I was using the same model in both examples just LiteLLM is the proxy in the middle in the failed example. I don't know the difference yet but it does seem to be an interface issue.