Letting an agent view an image returned from a tool -- what format to use? #25881
-
Checked other resources
Commit to Help
Example Codefrom langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_community.tools import tool
from langchain_core.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
from PIL import Image, ImageDraw, ImageFont
import os
os.environ["OPENAI_API_KEY"] = <my-api-key-here>
#create an image for MWE
image = Image.new('RGB', (100, 50), 'white')
draw = ImageDraw.Draw(image)
draw.text((20, 10), "Hello!", fill="black", font = ImageFont.load_default(size = 24))
image.save('secret_image.png')
#define tools
@tool
def load_secret_image():
"load a secret image that I have prepared for you"
img = Image.open("secret_image.png")
return img
#build model
prompt = ChatPromptTemplate.from_messages([
('system', 'You are a helpful AI bot'),
("human", "{input}"),
MessagesPlaceholder("agent_scratchpad"),
]
)
llm = ChatOpenAI(model="gpt-4o-mini")
agent = create_openai_tools_agent(llm, tools = [load_secret_image], prompt = prompt)
executor = AgentExecutor(agent = agent, tools=[load_secret_image])
#run model
for event in executor.stream({"input": "What word is present in the secret image?"}):
print(event)
#relevant part of what is printed:
#{'actions': [ToolAgentAction(tool='load_secret_image', tool_input={}, ...
#{'steps': [AgentStep(action=ToolAgentAction(tool='load_secret_image', ..., observation=<PIL.PngImagePlugin.PngImageFile image mode=RGB size=100x50 at 0x120CF2360>)], ...
#{'output': "I can't directly analyze the contents of the image. However, if you can describe the image or provide any context, I might be able to help you with more information!" ...DescriptionI'm trying to figure out how a tool should pass back multimodal output (in particular, an image) so an agent can comprehend it and continue working with it. I want the agent to analyze the image the same as though it had been included in the original human prompt, but it seems unable to parse it. The ultimate use case will be calling a plotting tool, since I think the agent might have an easier time comprehending visually presented data. For the MWE above, though, I am just loading a small image with the text 'Hello!', and asking the model what text is present in the image. Thing's I have tested:Returning a PIL image from the toolIn the MWE above, I was simply passing the image as loaded by PIL, with no further formatting Returning a ToolMessage with the image in base64To include an image in the original human prompt, it should be passed in a specific format using its base64 representation (as seen here). I tried using this format, wrapped in a tool message import base64
from langchain_core.messages import ToolMessage
@tool
def load_secret_image():
"load a secret image that I have prepared for you as a human message"
with open('secret_image.png', 'rb') as image_file:
binary_data = image_file.read()
image_data = base64.b64encode(binary_data).decode("utf-8")
return ToolMessage(
content = [
{"type": "text", "text": ""},
{
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{image_data}"}
}],
tool_call_id = "testid"
)Upon running the model as in the MWE, the tool is called, but the agent outputs: "The secret image does not contain any visible text..." Passing the ToolMessage with the image in base64 straight to model (not through the tool)To verify that the model should be able to read the word in the image, I tried passing the ToolMessage directly to the LLM: pure_message = load_secret_image({})
llm.invoke([
("human", "What word is present in this image?"),
pure_message
])This raised an error (from OpenAI): Wrapping the base64-image in a HumanMessage insteadI modified from langchain_core.messages import HumanMessage
@tool
def load_secret_image_human():
"load a secret image that I have prepared for you as a human message"
with open('secret_image.png', 'rb') as image_file:
# Read the image in binary mode
binary_data = image_file.read()
# Encode the binary data to base64
image_data = base64.b64encode(binary_data).decode("utf-8")
return HumanMessage(
content = [
{"type": "text", "text": ""},
{
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{image_data}"}
}]
)
pure_message = load_secret_image_human({})
llm.invoke([
('system', 'You are a helpful AI bot'),
("human", "What word is present in this image?"),
pure_message
])It returned Returning a HumanMessage from the toolRunning the full agent with this HumanMessage, it does not understand the output from the tool the same way: agent = create_openai_tools_agent(llm, tools = [load_secret_image_human], prompt = prompt)
executor = AgentExecutor(agent = agent, tools=[load_secret_image_human])
for event in executor.stream({"input": "What word is present in the secret image?"}):
print(event)The agent outputs "The secret image is encoded in base64, and it seems to be a PNG image. However, I cannot directly analyze or extract text from images..." System InfoSystem Information
Package Information
Optional packages not installed
Other Dependencies
pip freeze outputaiohappyeyeballs==2.4.0 |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 14 replies
-
|
@oe-andreas Hello! To let an agent view an image returned from a tool in LangChain, you should use the following format: from langchain.agents import initialize_agent, load_tools
from skimage import io
import cv2
# Load the necessary tools and initialize the agent
tools = load_tools(["dalle-image-generator"])
agent = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)
# Run the agent to generate the image
output = agent.run("Create an image of a halloween night at a haunted museum")
# Display the image
image_url = output # Assuming the output is the URL of the generated image
try:
import google.colab
IN_COLAB = True
except ImportError:
IN_COLAB = False
if IN_COLAB:
from google.colab.patches import cv2_imshow
image = io.imread(image_url)
cv2_imshow(image)
else:
image = io.imread(image_url)
cv2.imshow("Generated Image", image)
cv2.waitKey(0) # wait for a keyboard input
cv2.destroyAllWindows()This function uses the |
Beta Was this translation helpful? Give feedback.
-
|
@dosu Yes, this is what I already tested -- but as mentioned it didn't work. The agent is not able to interpret the byte-string as an image, even though it is able to do so if the same exact Human message is passed directly (not returned from a tool) |
Beta Was this translation helpful? Give feedback.
-
|
To make an agent interpret a base64-encoded image returned from a custom tool in LangChain, you can follow the approach used in the Here is an example of how you can modify your custom tool to return a base64-encoded image and ensure the agent can interpret it:
from langchain_community.tools import tool
import base64
from PIL import Image
from io import BytesIO
@tool
def load_secret_image() -> str:
"""Load a secret image and return it as a base64-encoded string."""
with open('secret_image.png', 'rb') as image_file:
image_data = base64.b64encode(image_file.read()).decode("utf-8")
return image_data
from langchain_core.messages import HumanMessage, ToolMessage, AIMessage
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
def interpret_image_with_agent():
# Define the prompt
prompt = ChatPromptTemplate.from_messages([
('system', 'You are a helpful AI bot'),
("human", "{input}"),
MessagesPlaceholder("agent_scratchpad"),
])
# Initialize the model
llm = ChatOpenAI(model="gpt-4o-mini")
# Create the agent with the custom tool
agent = create_openai_tools_agent(llm, tools=[load_secret_image], prompt=prompt)
executor = AgentExecutor(agent=agent, tools=[load_secret_image])
# Run the model
for event in executor.stream({"input": "What word is present in the secret image?"}):
print(event)
# Example usage
interpret_image_with_agent()In this example:
Additionally, the def _format_image(image_url: str) -> Dict:
"""
Formats an image of format data:image/jpeg;base64,{b64_string}
to a dict for anthropic api
{
"type": "base64",
"media_type": "image/jpeg",
"data": "/9j/4AAQSkZJRg...",
}
And throws an error if it's not a b64 image
"""
regex = r"^data:(?P<media_type>image/.+);base64,(?P<data>.+)$"
match = re.match(regex, image_url)
if match is None:
raise ValueError(
"Anthropic only supports base64-encoded images currently."
" Example: data:image/png;base64,'/9j/4AAQSk'..."
)
return {
"type": "base64",
"media_type": match.group("media_type"),
"data": match.group("data"),
}This approach ensures that the agent can interpret a base64-encoded image returned from a custom tool in LangChain [1][2][3][4][5]. |
Beta Was this translation helpful? Give feedback.
-
|
For the record if any human is reading, this issue remains unsolved |
Beta Was this translation helpful? Give feedback.
-
|
@oe-andreas I had similar issue while using
As you can see the I fixed that by literally coping the Honestly idk why ToolNode is doing this conversion of |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the great answer/detective work! I agree it seems odd that they are converting the output to a string, but the #TODO indicates that they are aware of the issue. Do you think we should submit a bug report? In my case (using
Unfortunately, OpenAI now raises the error I mentioned above: "Image URLs are only allowed for messages with role 'user'". I found this on the OpenAI forum, which led me to try the gpt-4-turbo model instead. Indeed, the turbo model doesn't raise the error, but it is unable to correctly interpret the image (it thinks the secret word is 'SECRET'). Did you have any success actually getting the model to interpret an image? If so, what LLM model did you use? EDIT: |
Beta Was this translation helpful? Give feedback.
-
|
I'm glad it moved your investigation further. I'm using In my case, although docs say we should return images in OpenAI format (here: https://python.langchain.com/v0.2/docs/how_to/multimodal_inputs/) which for images is IIRC this: I'm returning images from my tool like that instead (Claude schema, not OpenAI): Note that I'm returning an array, not entire Here is the link where you can find mentioned Claude format: Also, my tool that returns image is a class that inherits from As per submitting bug request, I see that there are a few issues related to ToolNode in langchain repo right now, I suggest checking if someone is not already fixing this or logic that is connected to this. It looks like ToolNode is indeed a work in progress with more issues than only this, but I can't investigate this further as I'm short on time. If nobody actually raised this yet, then yea, it would be awesome if you could create an Issue for that, do an investigation of why TODO comments are there and are those comments & conversions still relevant or outdated, and finally fix this in some way. In my case I'm suspecting that TODO comment maybe refers to the fact that I needed to return the image in Claude format, but if that is not why this comment and conversion to |
Beta Was this translation helpful? Give feedback.
-
|
Also, probably important: before using react agent I was using my tool in a chain and then I also needed to return images in Claude format instead of OpenAI. |
Beta Was this translation helpful? Give feedback.
-
Hi @oe-andreas , 1. Skip the force cast of tool reponse to strcomment out the def _run_one(self, call: ToolCall, config: RunnableConfig) -> ToolMessage:
# ...
try:
input = {**call, **{"type": "tool_call"}}
tool_message: ToolMessage = self.tools_by_name[call["name"]].invoke(
input, config
)
# TODO: handle this properly in core
**# tool_message.content = str_output(tool_message.content)**
return tool_message
except Exception as e:
# ...
return ToolMessage(content, name=call["name"], tool_call_id=call["id"])
async def _arun_one(self, call: ToolCall, config: RunnableConfig) -> ToolMessage:
#...
try:
input = {**call, **{"type": "tool_call"}}
tool_message: ToolMessage = await self.tools_by_name[call["name"]].ainvoke(
input, config
)
# TODO: handle this properly in core
**# tool_message.content = str_output(tool_message.content)**
return tool_message
except Exception as e:
#...
return ToolMessage(content, name=call["name"], tool_call_id=call["id"])2. return the right format in your tool.after done that, I still got the output of my tool converted to string. return {
"type": "image_url",
"image_url": {"url": local_image_to_data_url(filename)},
}I tracing the code further more, it turns out in def _is_message_content_type(obj: Any) -> bool:
"""Check for OpenAI or Anthropic format tool message content."""
if isinstance(obj, str):
return True
elif isinstance(obj, list) and all(_is_message_content_block(e) for e in obj):
return True
else:
return False
def _is_message_content_block(obj: Any) -> bool:
"""Check for OpenAI or Anthropic format tool message content blocks."""
if isinstance(obj, str):
return True
elif isinstance(obj, dict):
return obj.get("type", None) in ("text", "image_url", "image", "json")
else:
return FalseSo, the return of you tool must be a list of message content, and the type should be one of I modify my tool return content, it worked： def take_screenshot(self, filename):
self.device.screenshot(filename)
return [{
"type": "image_url",
"image_url": {"url": local_image_to_data_url(filename)},
}] |
Beta Was this translation helpful? Give feedback.
@oe-andreas I had similar issue while using
create_react_agent()from langgraph/prebuilt.I found that in my case
create_react_agent()creates a graph with ToolNode (which is responsible for calling tools - https://langchain-ai.github.io/langgraph/reference/prebuilt/#toolnode) and that this ToolNode is turning tostrthe content of the message that it got from calling tool.langgraph/prebuilt/tool_node.pyAs you can see the
contentof theToolMessageis converted tostr. If your.contentwas adictwhich describes image then it got converted tostrwithjson.dumps()and as a result thisToolMessagewill be treated by LLM/Chat model like a text reply.I fixed that by literally coping the
t…