Need help with integrating the model jxm/cde-small-v1

[YashDThapliyal]

I’ve been working on integrating the model jxm/cde-small-v1 using the SentenceTransformer library as per the instructions from the mteb/docs/adding_a_model.md, but I’m encountering a persistent issue that I haven’t been able to resolve.

Specifically, I copied the example code from the instructions and replaced the model name as required. However, I’m seeing the following error:

ModuleNotFoundError: No module named 'sentence_transformers_impl'

Here’s a summary of what I’ve tried so far to troubleshoot:

1. Set up a fresh virtual environment to isolate dependencies.
2. Updated all relevant libraries, including pip and sentence-transformers, to their latest versions.
3. Verified my environment by successfully loading other models like all-MiniLM-L6-v2 using SentenceTransformer.
4. Cleared the Hugging Face cache and re-downloaded the model to rule out corruption.
5. Inspected the error trace and configuration files, but it seems the model references a non-existent or custom module named sentence_transformers_impl, which isn’t part of the sentence-transformers library.

At this point, I’m unclear if the issue is with how I’ve implemented the model, a missing dependency, or a problem with the model itself. I would appreciete any guidance on how I can tackle this issue.

Thank you!

I have attached a picture of the code that I have written:

[Samoed]

You should use it like this

model = mteb.get_model(
    "jxm/cde-small-v1",
    trust_remote_code=True,
    model_prompts={
        "query": "search_query: ",
        "passage": "search_document: "
    }
)

[YashDThapliyal]

Current issue: Process Hanging and Warnings During Task Execution in MTEB

Hi @Samoed,

Thank you very much for providing the snippet earlier! I used it, and the code is now running, successfully listing tasks across categories like Classification, Clustering, and others.

However, I'm encountering two issues:

1. Warnings: I'm seeing multiple warnings, particularly about:

   • Repo card metadata:
     "Repo card metadata block was not found. Setting CardData to empty."
   • HuggingFace tokenizers Parallelism:
     "The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks..."

   While I understand these warnings might not directly indicate errors, they seem concerning given the next issue.

2. Hanging Process: The code appears to stall indefinitely and does not complete processing. Even after letting it run for a while, it does not proceed further. Disabling the warnings might not help because the code seems stuck, not just verbose.

---

Expected Behavior

The code should complete processing and proceed to the next steps without stalling.

Actual Behavior

The process hangs after listing the tasks, and no further steps are executed.

---

Additional Thoughts

I believe the parallelism warnings might be related to the stalling, but I'm unsure how to handle this correctly. Disabling the warning via TOKENIZERS_PARALLELISM might suppress the message but may not solve the underlying issue.

Do you have any suggestions for resolving this? Any guidance would be appreciated!

Thank you again for your time and support.
Yash

[Samoed]

I ran it using this snippet, and everything works fine.

import mteb
from sentence_transformers import SentenceTransformer
from transformers import AutoTokenizer, AutoModel

model = mteb.get_model(
    "jxm/cde-small-v1",
    trust_remote_code=True,
    model_prompts={
        "query": "search_query: ",
        "passage": "search_document: "
    }
)
tasks = mteb.get_tasks(
    tasks=[
        # classification
        "AmazonCounterfactualClassification", 
        # clustering
        "RedditClustering",
        # pair classification
        "TwitterSemEval2015",
        # reranking
        "AskUbuntuDupQuestions",
        # retrieval
        "SCIDOCS", 
#         # sts
        "STS22",
#         # summarization
        "SummEval",
        ]
)
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(
    model,
    output_folder="results",
    extra_kwargs={"batch_size": 8},
    overwrite_results=True
)