Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): user should be able to configure default types of default LLMs #140

Closed
konrad-czarnota-ds opened this issue Oct 22, 2024 · 1 comment · Fixed by #153
Closed
Assignees
Labels
core Changes to the core package feature New feature or request
Milestone

Comments

@konrad-czarnota-ds
Copy link
Collaborator

konrad-czarnota-ds commented Oct 22, 2024

  1. User should be able to add to their pyproject.toml a dict mapping LLM types to factory functions that create LLM of that type:

    [tool.ragbits.core.default_llm_factories]
    text = "my_project.llm.small_local"
    vision = "my_project.llm.gpt4o"
    structured_output = "my_project.llm.gpt4o"
    
  2. Ragbits should define an enum with all possible LLM types. The has_default_llm and get_default_llm should be updated so that they take the value of this enum as an argument (default: "text") and returns LLM of the correct type.

  3. Using an LLM type in pyproject.toml that is not present in the enum should result in an error

  4. UnstructuredImageProvider should be modified to use configured default LLM of type vision, instead of having a hard-coded default LLM.

@ludwiktrammer
Copy link
Collaborator

ludwiktrammer commented Oct 23, 2024

Context:

PR #121 introduced a default LLM object for UnstructuredImageProvider / UnstructuredPdfProvider which is used whenever the user does not provide an LLM object explicitly.

This is very similar to what we wanted to achieve with user-configurable default_llm_factory but the configuration option is not used here - instead the default is not user configurable and is hard-coded to be a LiteLLM client, using gpt-4o-mini and assumes that the API key is present in an environment variable.

The problem with using default_llm_factory here is that default_llm_factory is not guaranteed to be vision-enabled, and UnstructuredImageProvider / UnstructuredPdfProvider require a vision-enabled model.

When we first talked about default_llm_factory there was an idea of being able to configure a map with factories for different types of LLMs. For example:

[tool.ragbits.core.default_llm_factories]
text = "my_project.llm.small_local"
vision = "my_project.llm.gpt4o"
structured_output = "my_project.llm.gpt4o"

Maybe we should revisit the idea.

@mhordynski mhordynski moved this from Backlog to In Progress in ragbits Oct 23, 2024
@mhordynski mhordynski moved this from In Progress to Backlog in ragbits Oct 23, 2024
@mhordynski mhordynski moved this from Backlog to Ready in ragbits Oct 23, 2024
@ludwiktrammer ludwiktrammer changed the title feat(core): add default visual llm feat(core): user should be able to configure default types of default LLMs Oct 23, 2024
@mhordynski mhordynski added this to the Ragbits 0.4 milestone Oct 23, 2024
@konrad-czarnota-ds konrad-czarnota-ds self-assigned this Oct 25, 2024
@konrad-czarnota-ds konrad-czarnota-ds moved this from Ready to In Progress in ragbits Oct 25, 2024
@github-project-automation github-project-automation bot moved this from In review to Done in ragbits Oct 28, 2024
@micpst micpst added feature New feature or request core Changes to the core package labels Oct 30, 2024
@mhordynski mhordynski modified the milestones: Ragbits 0.4, Ragbits 0.3 Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Changes to the core package feature New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

4 participants