-
Notifications
You must be signed in to change notification settings - Fork 275
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add CUREv1 retrieval dataset #1459
feat: add CUREv1 retrieval dataset #1459
Conversation
--------- Co-authored-by: nadshe <[email protected]> Co-authored-by: olivierr42 <[email protected]> Co-authored-by: Daniel Buades Marcos <[email protected]>
e3ffa05
to
6bc7abe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great PR!!
We have also computed results for these tasks across 18 open-source models. We can upload them to the results repo or somewhere else, please point us in the right direction as there seems to be lots of activity with the new leaderboard!
Please do!
46aff47
to
37ddab2
Compare
Thanks for the PR! The only thing left to point out is that it'd be great if the PR description can be updated to reflect the changes made above. Otherwise I think this is good to merge. If you end up promoting this on socials, let us know :) |
Thanks, @isaac-chung ! Sorry I missed your last comment. I’ve updated the PR description retroactively and am currently running the 18 models on all the newly added tasks in the benchmark. Once that’s done, I’ll open a PR with the results. As for promoting the work, do you have any specific ideas in mind? I was planning to post something on LinkedIn next week, which I can also share here. Additionally, we’re preparing a preprint that we’ll be uploading to arXiv soon. Maybe we could use that opportunity to co-write something for the HF blog? We can discuss the angle but I believe that it could be really interesting! |
Thanks, @dbuades! Those all sound good, and I'm happy to share/repost what you have. An HF blog would be good as well - happy to collaborate there! |
Perfect! I'll keep you posted next week. |
CUREv1
Over the past year, we’ve worked closely with medical professionals to develop this dataset, which we’re now sharing with the community to support research in point-of-care information retrieval—a critical daily task for many practitioners.
The CURE is a cross-lingual retrieval dataset organized into:
Ten splits covering different medical disciplines:
One mono-lingual setting and two cross-lingual settings:
Each split and cross-lingual setting is composed of 200 natural language queries formulated by health care professionals, capturing their information needs when consulting academic literature during their daily work.
The corpus is constructed from an index of English passages extracted from biomedical academic articles. Passages are then marked as either Highly Relevant, Partially Relevant or Not Relevant with respect to each query.
For more details, please check the Dataset Card in the Hub 🤗
A preprint detailing the curation process and providing an extended rationale will soon be published on arXiv, along with pre-embedded indexes for several of these models!
MTEB(Medical)
At the same time, we take the opportunity to introduce a specialized benchmark that groups MTEB tasks relevant to the medical domain. Initially, we’ve included the following tasks, but we welcome any suggestions for additional tasks you think may be valuable:
We have also computed results for these tasks across 18 open-source models. We can upload them to the results repo or somewhere else, please point us in the right direction as there seems to be lots of activity with the new leaderboard! 💪
Adding datasets checklist
mteb -m {model_name} -t {task_name}
command.sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
intfloat/multilingual-e5-small
self.stratified_subsampling() under dataset_transform()
make test
.make lint
.