Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BREAKING: v2.0.0 #1433

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

BREAKING: v2.0.0 #1433

wants to merge 16 commits into from

Conversation

KennethEnevoldsen
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen commented Nov 11, 2024

This is a work-in-progress branch which will be the release of MTEB v2.0.0!

Features:

@x-tabdeveloping, @orionw, @isaac-chung, @Samoed, @gowitheflow-1998 etc. please make PR to this when relevant (MIEB still goes it its own branch but will try to merge it in here)

orionw and others added 5 commits November 13, 2024 11:30
* update

* merged retrieval; working

* update tasks; working multilingual

* everything working except instructions

* working instructions; just need cleanup

* add metadata for all but MindSmall

* faster evaluation; mindsmall can compute in reasonable time

* fix bad merge of docs

* lint

* fix test

* qa

* updated mindsmall

* lint

* fix debug

* Update mteb/abstasks/dataloaders.py

Co-authored-by: Roman Solomatin <[email protected]>

* lint

---------

Co-authored-by: Roman Solomatin <[email protected]>
Samoed and others added 10 commits November 14, 2024 21:26
* fix: Count unique texts, data leaks in calculate metrics (#1438)

* add more stat

* add more stat

* update statistics

* fix: update task metadata to allow for null (#1448)

* Update tasks table

* 1.19.5

Automatically generated by python-semantic-release

* base

* sync with main

---------

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <[email protected]>
* enable codecarbon by default

* lint

* update flag

* add allow_multiple_runs param

* make lint

* add warning

* lint

* negate the flag

---------

Co-authored-by: Isaac Chung <[email protected]>
* run tasks

* remove test script

* lint

* remove cache

* fix sickbrsts

* fix tests

* add datasets
* fix test

* skip mock

* add message to assert

* fix test

* lint

* fix tests

* upd tests

* update descriptive stats files

* add stat to speed
* multilingual loader

* lint
* add citations

* fix typo
* add code for comupting number of qrels

* add stats fever hotpotqa msmarco topiocqa

* miracl mrtidy

* multilongdoc  miracl reranking

* add multi eurlex

* fix tests for descriptive stats

* fix tests

---------

Co-authored-by: Roman Solomatin <[email protected]>
* add code for comupting number of qrels

* BibleNLPBitextMining descriptive stats added

* SwissJudgementClassification descriptive stats added

* VoyageMMarcoReranking descriptive stats added

* WebLINXCandidatesReranking descriptive stats added

* MultiEURLEXMultilabelClassification descriptive stats added

* MIRACLReranking descriptive stats added

* MindSmallReranking descriptive stats added

* updated test_TaskMetadata

* fix test

---------

Co-authored-by: Imene Kerboua <[email protected]>
Co-authored-by: Imene Kerboua <[email protected]>
Co-authored-by: Roman Solomatin <[email protected]>
* fix bright loader

* lint

* fix comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants