Inconsistency between the instruction template suggested vs that in the training data

[debraj135]

I noticed that the instructions in the training data end with ; and no whitespace after that.

For example 'Represent the Science sentence;' instead of 'Represent the Science sentence: '

Whereas in the readme, the proposed format seems to be 'Represent the Science sentence: ' sometimes and 'Represent the Science sentence:' in other places.

All of these three seem to be resulting in different embeddings and hence different similarity numbers. Can you please let us know what is the right instruction template?

[debraj135]

Wondering if I'm missing a detail. Did anyone else also come across this?

[hongjin-su]

Thanks a lot for your interest in the INSTRUCTOR!

Like other LLMs, the INSTRUCTOR is sensitive to the instructions, which may be worsened by its small size. I would say all of your proposed instructions follow the basic templates, while we may need more trials or heuristics to figure out the best instruction.

[debraj135]

Thank you. I had a few follow up questions

1. Did the instruction that was provided as an input to the model during training have a semicolon ; or a colon : at the end of the instruction?
2. Which of the above three templates that I previously mentioned were used in performing the evaluation on the MTEB leaderboard?

[debraj135]

Following back on this.

[hongjin-su]

Sorry for the late reply!

In our training and evaluation, we may not be very strict on punctuation. We are glad to make it more consistent in our future versions!