Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(document-search): add evaluation pipeline for retrieval accuracy #19

Closed
mhordynski opened this issue Sep 18, 2024 · 1 comment · Fixed by #91
Closed

feat(document-search): add evaluation pipeline for retrieval accuracy #19

mhordynski opened this issue Sep 18, 2024 · 1 comment · Fixed by #91
Assignees
@micpst
Labels
document search Changes to the document search package evals Adding new evaluation pipelines or improving existing ones feature New feature or request
Milestone
Ragbits 0.2

Comments

@mhordynski
Copy link
Member

Feature description

DocumentSearch module should provide a way to evaluate given configuration on the dataset provided by the user.
Dataset ideally should follow a well-know format, for example huggingface's datasets.

For given configuration and dataset evaluation pipeline should calculate metrics such as:

  • Precision / Recall / F1
  • mAP
  • mRR
  • NDCG

Motivation

DocumentSearch package has three main feature:

  • retrieval
  • ingestion
  • evaluation

This task enables us to start evaluation bit

Additional context

Some relevant tools that may be either used as a dependency, or inspire us.

https://docs.auto-rag.com/
https://github.com/explodinggradients/ragas
@mhordynski mhordynski added the feature New feature or request label Sep 18, 2024
@mhordynski mhordynski moved this to Backlog in ragbits Sep 18, 2024
@mhordynski mhordynski added this to the Ragbits Document Search 0.1 milestone Sep 18, 2024
@micpst micpst added the document search Changes to the document search package label Sep 18, 2024
@micpst
Copy link
Collaborator

micpst commented Sep 18, 2024

Do we have any open-source dataset for testing?

@mhordynski mhordynski moved this from Backlog to Ready in ragbits Sep 18, 2024
@akotyla akotyla mentioned this issue Sep 23, 2024
Closed
@mhordynski mhordynski modified the milestones: Ragbits Document Search 0.1, Ragbits 0.1, Ragbits 0.2, Ragbits 0.3 Sep 23, 2024
@micpst micpst linked a pull request Oct 13, 2024 that will close this issue
feat(eval): add evaluation pipeline for document search #91
Merged
@micpst micpst mentioned this issue Oct 14, 2024
Closed
@micpst micpst added the evals Adding new evaluation pipelines or improving existing ones label Oct 16, 2024
@github-project-automation github-project-automation bot moved this from In review to Done in ragbits Oct 23, 2024
@mhordynski mhordynski modified the milestones: Ragbits 0.3, Ragbits 0.2 Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@micpst micpst

Labels
document search Changes to the document search package evals Adding new evaluation pipelines or improving existing ones feature New feature or request
Projects
ragbits
Status: Done
Milestone
Ragbits 0.2
Development

Successfully merging a pull request may close this issue.

feat(eval): add evaluation pipeline for document search deepsense-ai/ragbits
2 participants
@mhordynski @micpst