Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add CUREv1 retrieval dataset #1459

Merged

Conversation

dbuades
Copy link
Contributor

@dbuades dbuades commented Nov 14, 2024


CUREv1

Over the past year, we’ve worked closely with medical professionals to develop this dataset, which we’re now sharing with the community to support research in point-of-care information retrieval—a critical daily task for many practitioners.

The CURE is a cross-lingual retrieval dataset organized into:

  • Ten splits covering different medical disciplines:

    • Dentistry and Oral Health
    • Dermatology
    • Gastroenterology
    • Genetics
    • Neuroscience and Neurology
    • Orthopedic Surgery
    • Otorhinolaryngology
    • Plastic Surgery
    • Psychiatry and Psychology
    • Pulmonology
  • One mono-lingual setting and two cross-lingual settings:

    • English-to-English
    • Spanish-to-English
    • French-to-English

Each split and cross-lingual setting is composed of 200 natural language queries formulated by health care professionals, capturing their information needs when consulting academic literature during their daily work.

The corpus is constructed from an index of English passages extracted from biomedical academic articles. Passages are then marked as either Highly Relevant, Partially Relevant or Not Relevant with respect to each query.

For more details, please check the Dataset Card in the Hub 🤗
A preprint detailing the curation process and providing an extended rationale will soon be published on arXiv, along with pre-embedded indexes for several of these models!


MTEB(Medical)

At the same time, we take the opportunity to introduce a specialized benchmark that groups MTEB tasks relevant to the medical domain. Initially, we’ve included the following tasks, but we welcome any suggestions for additional tasks you think may be valuable:

  • CUREv1
  • NFCorpus
  • TRECCOVID
  • TRECCOVID-PL
  • SciFact
  • SciFact-PL
  • MedicalQARetrieval
  • PublicHealthQA
  • MedrxivClusteringP2P.v2
  • MedrxivClusteringS2S.v2
  • CmedqaRetrieval
  • CMedQAv2-reranking

We have also computed results for these tasks across 18 open-source models. We can upload them to the results repo or somewhere else, please point us in the right direction as there seems to be lots of activity with the new leaderboard! 💪


Adding datasets checklist

  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
  • I have filled out the metadata object in the dataset file (find documentation on it here).
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

---------

Co-authored-by: nadshe <[email protected]>
Co-authored-by: olivierr42 <[email protected]>
Co-authored-by: Daniel Buades Marcos <[email protected]>
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR!!

We have also computed results for these tasks across 18 open-source models. We can upload them to the results repo or somewhere else, please point us in the right direction as there seems to be lots of activity with the new leaderboard!

Please do!

mteb/tasks/Retrieval/multilingual/CUREv1Retrieval.py Outdated Show resolved Hide resolved
mteb/benchmarks/benchmarks.py Outdated Show resolved Hide resolved
@isaac-chung
Copy link
Collaborator

Thanks for the PR! The only thing left to point out is that it'd be great if the PR description can be updated to reflect the changes made above. Otherwise I think this is good to merge. If you end up promoting this on socials, let us know :)

@isaac-chung isaac-chung merged commit 1cc6c9e into embeddings-benchmark:main Nov 21, 2024
10 checks passed
@dbuades
Copy link
Contributor Author

dbuades commented Nov 21, 2024

Thanks for the PR! The only thing left to point out is that it'd be great if the PR description can be updated to reflect the changes made above. Otherwise I think this is good to merge. If you end up promoting this on socials, let us know :)

Thanks, @isaac-chung ! Sorry I missed your last comment. I’ve updated the PR description retroactively and am currently running the 18 models on all the newly added tasks in the benchmark. Once that’s done, I’ll open a PR with the results.

As for promoting the work, do you have any specific ideas in mind? I was planning to post something on LinkedIn next week, which I can also share here. Additionally, we’re preparing a preprint that we’ll be uploading to arXiv soon. Maybe we could use that opportunity to co-write something for the HF blog? We can discuss the angle but I believe that it could be really interesting!

@isaac-chung
Copy link
Collaborator

Thanks, @dbuades! Those all sound good, and I'm happy to share/repost what you have. An HF blog would be good as well - happy to collaborate there!

@dbuades
Copy link
Contributor Author

dbuades commented Nov 22, 2024

Thanks, @dbuades! Those all sound good, and I'm happy to share/repost what you have. An HF blog would be good as well - happy to collaborate there!

Perfect! I'll keep you posted next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants