Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chain does not support images #238

Open
samuelint opened this issue Oct 3, 2024 · 0 comments
Open

Chain does not support images #238

samuelint opened this issue Oct 3, 2024 · 0 comments

Comments

@samuelint
Copy link
Contributor

Describe the bug
Using chain (ConversationalChainBuilder or Agent) does not support images.

prompt_args! {
            "input" => input,
        }

does not allow to pass an image url.
If the url is passed as a string the in input, an error is thrown OpenAI error: stream failed: Invalid status code: 400 Bad Request probably because the image is not under the right structure (Image content part).

Also
langchain_rust::schemas::Message::messages_to_string only serialize text content. It does not take in consideration images.

pub fn messages_to_string(messages: &[Message]) -> String {
messages
.iter()
.map(|m| format!("{:?}: {}", m.message_type, m.content))
.collect::<Vec<String>>()
.join("\n")
}

To Reproduce

let chain = ConversationalChainBuilder::new()
            .llm(llm)
            .build();

let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();
let input = format!("What is that? {}", base64_image_url);
let result = chain.invoke(prompt_args! {
    "input" => input,
});

Expected behavior
A way to pass image to a chain or at minimum an Agent.

Since images are on langchain_rust::schemas::Message structure, I would expect them to work in every aspect of the library.

Additional context
This is some kind of a violation of the Liskov Principle & Interface Segregation Principles. If image are available on langchain_rust::schemas::Message they should be supported everywhere in the library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant