fix: Fix test for empty descriptive tasks #1413
Conversation
|
Oh, I missed that test not passing. I'm preparing an update to descriptive stats, and after that I think I can try to calculate on some datasets |
|
Moving this to v2.0.0, but we could move it back if it makes things easier |
|
@KennethEnevoldsen Tests are fixed. I think this is ready to merge into main for now. If anyone wants to add new tasks, this test should be helpful. Future updates to descriptive stats could be planned for version 2.0.0. |
|
Shouldn't the test fail? GIven that most datasets don't have descriptive stats? |
|
I've added current tasks without metadata to exceptions like in |
|
Hmm, we might be misunderstanding each other. A task like "TbilisiCityHallBitextMining" no longer has descriptive statistics, but did have it beforehand. |
Samoed
force-pushed
the
fix_test_descriptive_stat
branch
from
dd0f5f0
to
ddd04ab
Compare
|
I updated the test. Now, if a task is in the exceptions list (tasks currently without descriptive stats) and has metadata, it will raise an error. Otherwise, descriptive stats are required. |
# Conflicts: # mteb/descriptive_stats/InstructionReranking/Core17InstructionRetrieval.json # tests/test_tasks/test_metadata.py
|
@KennethEnevoldsen I've updated test. Can you review, please? |
KennethEnevoldsen
merged commit 0a5bedb
into
embeddings-benchmark:v2.0.0
Checklist
make test.make lint.It seems that when creating the test, I missed recreating the array in the test case, which caused the test to not work correctly. Fixes #1412