Add actual tool output to astream_events instead of string cast output

[adreo00]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Currently tool output from the on_tool_end event, does not seem to match what the tool returns when invoked directly. In the following example, when the tool is invoked directly, we see a list[Documents] returned. When the tool is invoked via the agent_executor.astream_events, we see a string returned.

TLDR: I'm trying to get the output of tool calls, not after they've been converted to a string, when using astream_events from an agent_executor.

minimal example .ipynb

from langchain import hub
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain.tools import tool
from langchain_openai import ChatOpenAI
from langchain_core.documents.base import Document

prompt = hub.pull("hwchase17/openai-tools-agent")

@tool
def get_retriever_docs(query: str) -> list[Document]:
    """Returns a list of documents from the retriever."""
    return [Document(page_content="# test file\n\nThis is a test file with a secret code of 'blue-green-apricot-brownie-cake-mousepad'.", metadata={'source': './test.md'})]



model = ChatOpenAI(temperature=0, streaming=True)
tools = [get_retriever_docs]
agent = create_openai_tools_agent(
    model.with_config({"tags": ["agent_llm"]}), tools, prompt
)
agent_executor = AgentExecutor(agent=agent, tools=tools).with_config(
    {"run_name": "Agent"}
)
query="call get_retriever_docs and tell me a secret from the docs"

events = []
async for event in agent_executor.astream_events(
    {"input": query},
    version="v1",
):
    if event.get('name')=="get_retriever_docs" and event.get('event')=="on_tool_end":
        tool_end_output = event.get('data').get('output')
    if event.get('event')=="on_chain_stream" and event.get('name')=="Agent" and event.get('data').get('chunk',{}).get('steps'):
        on_stream_end_event = event

test_docs = get_retriever_docs.invoke({"query": "secret"}) ## returns a list[Document] as expected...
print(f"this is a list[Document]: {type(test_docs[0])}")

print(f"this is a string, but could we make this a list[Documents] as the tool actually returns?: {type(tool_end_output)}")
agent_observation = on_stream_end_event.get('data').get('chunk',{}).get('steps')[0].observation[0]
print(f"the list[Documents] type is evident by this event: {type(agent_observation)}")

Motivation

If I'm trying to reliable find the out of of the tool, from the astream_events method. Having to dig through the many on_chain_stream events doesn't seem reliable, especially if an agent has multiple tools. Can we find a way to expose the output of the tool via the astream_events, as it is returned by the tool (i.e list[Documents] NOT str type).

Proposal (If applicable)

No response

[adreo00]

I have the same issue when I test with the original tool @eyurtsev (some_tool) wrote in their original proposal. This is the on_tool_end event. Note how the type of data.output is a str, not a dict. Is this expected? If so, what's the best approach for retrieving the actual dict tool response?

{'event': 'on_tool_end', 'name': 'some_tool', 'run_id': 'da2748a4-d356-4c74-b653-419b968bf833', 'tags': [], 'metadata': {}, 'data': {'input': {'x': 5, 'y': 'hello'}, 'output': "{'x': 5, 'y': 'hello'}"}}

[adreo00]

I've tried tracing this locally to understand where we are converting the return of the tool to a string, but I'm struggling to trace through all the async/concurrent library calls. I can see when the tool is called here, it's still in the expected type (dict, or list[Documents]), but after that, when it gets put in the event.data.outputs dict, it somehow gets converted to a string.

[eyurtsev]

Potentially a bug. I'm on mobile could you post on the RFC to link to this idea

[eyurtsev]

OK not an issue with a vanilla tool.

from langchain_core.tools import tool
from langchain_core.documents import Document

@tool
def get_docs(x: int):
    """Hello"""
    return [Document(page_content='hello')]

async for event in get_docs.astream_events({"x": 5}, version='v1'):
    print(event)

{'event': 'on_tool_start', 'run_id': '95a96f6c-afa7-4cb7-b8a3-e61347929d44', 'name': 'get_docs', 'tags': [], 'metadata': {}, 'data': {'input': {'x': 5}}}
{'event': 'on_tool_stream', 'run_id': '95a96f6c-afa7-4cb7-b8a3-e61347929d44', 'tags': [], 'metadata': {}, 'name': 'get_docs', 'data': {'chunk': [Document(page_content='hello')]}}
{'event': 'on_tool_end', 'name': 'get_docs', 'run_id': '95a96f6c-afa7-4cb7-b8a3-e61347929d44', 'tags': [], 'metadata': {}, 'data': {'output': [Document(page_content='hello')]}}

[adreo00]

Confirmed. I also get the same output: 'data': {'output': [Document(page_content='hello')]}

[eyurtsev]

Potentially an issue with agents -- recreated the issue with this notebook and a modified tool that returned dicts

https://python.langchain.com/docs/modules/agents/how_to/streaming#custom-streaming-with-events

[alex-dreo-persefoni]

@eyurtsev anything I can do to help on this? I'm happy to create an issue from this discussion, now that we've both separately been able to reproduce the issue?

[eyurtsev]

This code is a bit in the core of langchain, so i'll take care of it -- won't be fast for a non core maintainer to make a correct fix

[alex-dreo-persefoni]

Awesome, thanks!

[eyurtsev]

Confirmed issue: #18760