Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(document-search): add monitoring for retrieval evaluation #32

Closed
akotyla opened this issue Sep 23, 2024 · 1 comment
Closed

feat(document-search): add monitoring for retrieval evaluation #32

akotyla opened this issue Sep 23, 2024 · 1 comment
Assignees
@micpst
Labels
document search Changes to the document search package feature New feature or request
Milestone
Ragbits 0.2

Comments

@akotyla
Copy link
Collaborator

akotyla commented Sep 23, 2024 •
edited
Loading

Feature description

Integrate Neptune (TensorBoard is nice-to-have) to monitor and track key metrics (Precision / Recall / F1, mAP, mRR, NDCG), queries, retrieved documents, and experiment settings (model, reranker, rephraser etc.) during retrieval evaluation.

This should be added to the evaluation pipeline: #19

Motivation

This will ensure comprehensive monitoring of a retrieval system, helping to improve, debug, and optimize performance.

Additional context

No response
@akotyla akotyla added feature New feature or request document search Changes to the document search package labels Sep 23, 2024
@akotyla akotyla added this to ragbits Sep 23, 2024
@akotyla akotyla moved this to Backlog in ragbits Sep 23, 2024
@mhordynski mhordynski added this to the Ragbits 0.3 milestone Sep 24, 2024
@mhordynski mhordynski moved this from Backlog to Ready in ragbits Sep 24, 2024
@micpst
Copy link
Collaborator

micpst commented Oct 14, 2024

Neptune monitoring will be implemented in #91

@micpst micpst self-assigned this Oct 22, 2024
@micpst micpst closed this as completed Oct 23, 2024
@github-project-automation github-project-automation bot moved this from In review to Done in ragbits Oct 23, 2024
@mhordynski mhordynski modified the milestones: Ragbits 0.3, Ragbits 0.2 Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@micpst micpst

Labels
document search Changes to the document search package feature New feature or request
Projects
ragbits
Status: Done
Milestone
Ragbits 0.2
Development

No branches or pull requests
3 participants
@mhordynski @micpst @akotyla