Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Simplification of Document Search Evaluation interface #220

Open
azygadlo opened this issue Dec 2, 2024 · 0 comments · May be fixed by #258
Open

feat: Simplification of Document Search Evaluation interface #220

azygadlo opened this issue Dec 2, 2024 · 0 comments · May be fixed by #258
Assignees
Labels
feature New feature or request
Milestone

Comments

@azygadlo
Copy link
Collaborator

azygadlo commented Dec 2, 2024

Feature description

Currently, the entry barrier to running a retrieval evaluation example is not so low.

One needs to understand the concepts and initialize objects such as:

  • dataloader
  • metrics
  • pipeline
  • evaluator

and then run evaluator.compute() and pass the other 3 objects.

The same functionality could be achieved with a simpler interface, e.g. by initializing only a single object (let's say: EvaluationPipeline), which would handle the data loading and metrics under the hood, in addition to what the current pipeline implementation does.

Furthemore, some reasonable default values can be either set (without having to pass them via a hierarchy of YAML files) or automatically inferred (e.g. based on the existence of OPENAI_API_KEY or other environment variables).

Motivation

The entry barrier should be lower for new users only willing to run example evaluation scripts.

Additional context

initial idea from @mhordynski :

from ragbits.evaluate.recipes.document_search import DocumentSearchEval

eval_recipe = DocumentSearchEval("path/to/my/dataset") \
    .embeddings(
        LiteLLMEmbeddings("text-embeddings-large"),
        LiteLLMEmbeddings("text-embeddings-small"),
        LocalEmbeddings("some-shit-from-hf")
    ).rerankers(
        Reranker1(),
        Reranker2()
    ).rephrasers(
        Reprasher1(),
        Reprasher2()
    )

eval_recipe.evaluate(...)
eval_recipe.optimize(...)

It should wrap lower-level concepts like Evaluator or Optimizer in a higher-level API for simplicity.

@azygadlo azygadlo added the feature New feature or request label Dec 2, 2024
@mhordynski mhordynski moved this to Backlog in ragbits Dec 2, 2024
@mhordynski mhordynski moved this from Backlog to Ready in ragbits Dec 2, 2024
@mhordynski mhordynski added this to the Ragbits 0.6 milestone Dec 2, 2024
@kdziedzic68 kdziedzic68 moved this from Ready to In Progress in ragbits Dec 5, 2024
@kdziedzic68 kdziedzic68 self-assigned this Dec 5, 2024
@kdziedzic68 kdziedzic68 moved this from In Progress to In review in ragbits Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Status: In review
Development

Successfully merging a pull request may close this issue.

3 participants