AgentFinish llm token streaming

[rere950303]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

When i create a stream endpoint through function agent and langserve, only Agent Action, Observation, and Final result are streamed as shown on https://python.langchain.com/docs/modules/agents/how_to/streaming#using-agentactionobservation . However, for Final result, I would like to stream in units of tokens that llm gives as a response.

Motivation

I thought it would be more user-friendly to stream Final result(AgentFinish) in llm token units than Agent Action and Observation for ordinary users who asked the question. Because ordinary users can recognize that receiving llm responses in token units is currently responding to my question and they do not need to know about the intermediate steps of the agent.

Proposal (If applicable)

No response

[JarandN]

I agree. Documentation states AgentFinish is not available as part of the streaming method. If this is something you’d like to be added, please start a discussion on github and explain why its needed..

I would argue that AgentFinish is the most important thing to stream.

The Action and Observation steps completes within a few seconds and that leaves the user with nothing for about 20 seconds before getting the output in one go. Streaming Agent Finish should effectively half the time for the user to to see the first final answer token.

With the current inference speed by GPT-4 Turbo (which anecdotally has become much slower the last month), the lack of streaming of output essentially makes the agents with this setup not viable for chat features

[JarandN]

After some serious digging I found that stream events for the newest agent essentially solves this.