[WIP] Support for Capella-hosted embedding models #68
Conversation
…be pushed. TODO: [] - tests for different sentence transformers models [] - tests for embedding model use from OpenAI endpoint
| ``AGENT_CATALOG_EMBEDDING_MODEL_AUTH`` | ||
| The field used in the authorization header of all OpenAI-standard client embedding requests. | ||
| For embedding models hosted by OpenAI, this field refers to the API key. | ||
| For embedding models hosted by Capella, this field refers to the Base64-encoded value of |
There was a problem hiding this comment.
users provide the database conn uname and password
this field would not be a requirement and instead the request can contain the header with uname and pwd
i assume we dont need the user to get this encoded string
There was a problem hiding this comment.
If the users provide this field, we will use this (as we want to support not just Capella, but all OpenAI-standard endpoints) -- but yes, we can use the AGENT_CATALOG_USERNAME and AGENT_CATALOG_PASSWORD variables to create this encoding string if AGENT_CATALOG_EMBEDDING_MODEL_AUTH is not set.
| "of each catalog entry.", | ||
| examples=["sentence-transformers/all-MiniLM-L12-v2"], | ||
| embedding_model: EmbeddingModel = pydantic.Field( | ||
| description="Embedding model used for all descriptions in the catalog.", |
There was a problem hiding this comment.
| description="Embedding model used for all descriptions in the catalog.", | |
| description="Embedding model used to generate embedding for tool/prompt descriptions to store in the catalogs.", |
something like this would be more suitable
|
|
||
| # Grab our local tool embedding model... | ||
| local_tool_catalog_path = self.catalog_path / DEFAULT_TOOL_CATALOG_NAME | ||
| if local_tool_catalog_path.exists(): | ||
| with local_tool_catalog_path.open("r") as fp: | ||
| local_tool_catalog = CatalogDescriptor.model_validate_json(fp.read()) | ||
| collected_embedding_model_names.add(local_tool_catalog.embedding_model) | ||
| collected_embedding_models.add(local_tool_catalog.embedding_model) | ||
|
|
||
| # ...and now our local prompt embedding model. |
There was a problem hiding this comment.
do users have the provision to specify different models for tool and prompt each?
if im not wrong, AGENT_CATALOG_EMBEDDING_MODEL defines the embedding model used and its for both prompts and tools
There was a problem hiding this comment.
they shouldn't, but this is the constraint we have now (because right now we have two metadata collections). in the future, we want to remove this
| The embedding model that Agent Catalog will use when indexing and querying tools and prompts. | ||
| This *must* be a valid embedding model that is supported by the :python:`sentence_transformers.SentenceTransformer` | ||
| class. | ||
| class *or* the name of a model that can be used from the endpoint specified in the environment variable | ||
| ``AGENT_CATALOG_EMBEDDING_MODEL_URL``. |
There was a problem hiding this comment.
can we specify the allowed models for openai in the docs
what can the value for this and url be to hit the /endpoint route
There was a problem hiding this comment.
there aren't really allowed models, its whatever the model service provides (given that the model supports function calling, etc...)
| ) | ||
| base_url: typing.Optional[str] = pydantic.Field( | ||
| description="The base URL of the embedding model." | ||
| "This field must be specified is using a non-SentenceTransformers-based model.", |
There was a problem hiding this comment.
| "This field must be specified is using a non-SentenceTransformers-based model.", | |
| "This field must be specified if using a non-SentenceTransformers-based model.", |
|
|
||
|
|
||
| class EmbeddingModel(pydantic.BaseModel): | ||
| kind: typing.Literal["sentence-transformers", "openai"] = pydantic.Field( |
There was a problem hiding this comment.
for caepella models, what is the kind?
There was a problem hiding this comment.
openai (because Capella is supposed to be OpenAI-compliant)
| ) | ||
| name: str = pydantic.Field( | ||
| description="The name of the embedding model being used.", | ||
| examples=["all-MiniLM-L12-v2", "https://12fs345d.apps.cloud.couchbase.com"], |
There was a problem hiding this comment.
name of model if its on capella is intfloat/e5-mistral-7b-instruct
last i checked this is the only embedding model they have hosted
There was a problem hiding this comment.
https://12fs345d.apps.cloud.couchbase.com is more of the endpoint fqdn url if im not wrong
There was a problem hiding this comment.
yep, this name example is wrong 😅
it would be better if you merge this to dev branch instead of master |
Putting a hold on this -- going to wait for the "init" command PR to be pushed.
TODO: