embeddings: add support for prefixes in embeddings

[eliransin]

Some of the embedding models (e.g nomic-embed-text) are trained to embed for several purposes, this on the flip side, means that if not given the purpose for the embedding the retrieval results might and probably will be suboptimal. This change aims to deal with it by adding a configuration option for chunk and query prefixes with the option to support more embedding purposes in the future (i.e clustering).

Some refferences to justify this change:

1. https://huggingface.co/nomic-ai/nomic-embed-text-v1.5#task-instruction-prefixes The nomic-embed-text prefixes section
2. https://www.youtube.com/watch?v=76EIC_RaDNw (Don’t Embed Wrong! by Matt Williams)

Description

[ What changed? Feel free to be brief. ]

Checklist

• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Screenshots

[ For visual changes, include screenshots. ]

Testing instructions

Nothing special just adding some prefixes and making sure that a different index is being created. Some better retrievals might be observed but I guess it depend on the size of the index (the smaller the index the less the effect I assume).

[netlify]

✅ Deploy Preview for continuedev canceled.

Name	Link
🔨 Latest commit	51090ef
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67de531fdd868e0008e9dee1

[sestinj]

@eliransin thanks for the great work here! Everything written so far looks good to me. There's one thing we should add, which is support for the config.yaml format. This would be added in the config-yaml/src/schemas/models.ts embedOptionsSchema, and then should be passed through here: https://github.com/continuedev/continue/blob/c1699c975fc252d2ed14b8c4a9770d802852cd3b/core/config/yaml/models.ts

[eliransin]

Changes in this version:

1. rebased on main and resolved conflicts
2. added zod definitions as instructed in cr comments
   🙂

[eliransin]

@sestinj I have a bit of a noob qusetion/s, I noticed that config-yaml/src/schemas/models.ts embedOptionsSchema sits in its own package.
should I bump the version of this package (and probably also bump up the coresponding entry in the core package.json) ? If I do, is the dependency installed from local source or should I first open a PR to update this package and have the new version uploaded and registered in package registy and only then do a second PR that uses this package in core?
Tx

[eliransin]

Summary:
Figured out I should do the following, but unsure if I need to do it in two separate steps:

1. Make changes to packages/config-yaml including bumping up the version
2. Update the dependencies of core to the new @continuedev/config-yml version
3. Add the extra assignment from the newly added option in core/config/yaml/models.ts
4. All of the current already made changes

My question is:
Should 1 be in a separate PR?

I am going to go ahead and submit one unified PR and will assume that if this is the wrong thing to do, checks will fail 🙂

Probably will only do it in a day or so, so maybe I'll get an answer by then...

[eliransin]

@sestinj I digged a bit in the previous commits, it seams like I need to first have the package in registry so probably it should occure in 2 separate stages. I will break this PR into two.

[eliransin]

@sestinj converted this to draft until the config-yaml package is published and proper adjutments are made as a result.