Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bugs] Incompatible task names between github repo and leaderboard config #23

Open
hanhainebula opened this issue Jul 29, 2024 · 2 comments

Comments

@hanhainebula
Copy link

hanhainebula commented Jul 29, 2024

Dear MTEB Team,

I noticed that there are some incompatible task names between github repo and leaderboard config:

  • C-MTEB:
    • CmedQAv1-reranking v.s. CmedQAv1
    • CmedQAv2-reranking v.s. CmedQAv2
  • MTEB-fr:
    • XQARetrieval (fra-fra) v.s. XQARetrieval (fr-fr)
  • MTEB-pl:
    • EightTagsClustering v.s. 8TagsClustering

The above bugs will cause the corresponding results in the model card generated with the latest mteb package can not be loaded into the leaderboard. Therefore, our latest model bge-multilingual-gemma2 now has some empty results in the leaderboard. I just modified the model card to adapt to the leaderboard config. Could you update the leaderboard again to load the missing results? Thank you very much!

@hanhainebula
Copy link
Author

hanhainebula commented Jul 29, 2024

Thank you for the update! But I'm so sorry for the correction:

  • MTEB-fr:
    • XQARetrieval (fra-fra) v.s. XQARetrieval (fr)

There's still one missing result in MTEB-fr. Could you update the leaderboard again? Thank you!

@hanhainebula
Copy link
Author

The metric used for BSARDRetrieval is also incompatiable:

  • github repo: recall_at_100
  • leaderboard: ndcg_at_10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant