Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Prompt and LLM should accept images #100

Closed
3 tasks
ludwiktrammer opened this issue Oct 14, 2024 · 0 comments · Fixed by #149
Closed
3 tasks

feat: Prompt and LLM should accept images #100

ludwiktrammer opened this issue Oct 14, 2024 · 0 comments · Fixed by #149
Assignees
Labels
core Changes to the core package feature New feature or request
Milestone

Comments

@ludwiktrammer
Copy link
Collaborator

  • The BasePrompt class should include a new method for returning a list of images (default implementation: empty list). It's important that the format in which the images is compatible with how images are stored in Document Search's ImageDocument (although Prompt shouldn't directly depend on ImageDocument)
  • The Prompt class should include its own implementation of the method, that automatically returns images from chosen fields of the Prompt's input model (which fields should probably be configured as a static field in the Prompt class)
  • The LLM and LLMClient implementations (including the built-in implementation using LiteLLM) should automatically send those images to LLM models that support it (see: the Vision section in OpenAI docs and Checking if a model supports vision in LiteLLM docs
@ludwiktrammer ludwiktrammer self-assigned this Oct 14, 2024
@ludwiktrammer ludwiktrammer converted this from a draft issue Oct 14, 2024
@ludwiktrammer ludwiktrammer added the feature New feature or request label Oct 14, 2024
@micpst micpst added the core Changes to the core package label Oct 17, 2024
@ludwiktrammer ludwiktrammer moved this from Backlog to Ready in ragbits Oct 22, 2024
@ludwiktrammer ludwiktrammer removed their assignment Oct 22, 2024
@mhordynski mhordynski added this to the Ragbits 0.3 milestone Oct 23, 2024
@kdziedzic68 kdziedzic68 self-assigned this Oct 23, 2024
@mhordynski mhordynski linked a pull request Oct 24, 2024 that will close this issue
@github-project-automation github-project-automation bot moved this from In review to Done in ragbits Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Changes to the core package feature New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

4 participants