Skip to content

Commit

Permalink
Added embedding dimensions by model (#307)
Browse files Browse the repository at this point in the history
  • Loading branch information
Paul-Cornell authored Oct 30, 2024
1 parent b893eb7 commit 4025330
Show file tree
Hide file tree
Showing 3 changed files with 34 additions and 13 deletions.
13 changes: 10 additions & 3 deletions platform/embedding.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,16 @@ on Hugging Face:

## Generate embeddings

To generate embeddings, choose one of the following embedding providers in the **Providers** section of an **Embedder** node in a workflow:
To generate embeddings, choose one of the following embedding providers and models in the **Providers** section of an **Embedder** node in a workflow:

<Note>You can change a workflow's predefined provider only through [Custom](/platform/workflows#create-a-custom-workflow) workflow settings.</Note>

- **OpenAI**: Use [OpenAI](https://openai.com) to generate embeddings.
- **Vertex AI**: Use [Vertex AI](https://cloud.google.com/vertex-ai) to generate embeddings.
- **OpenAI**: Use [OpenAI](https://openai.com) to generate embeddings. Also, choose the model to use:

- **text-embedding-3-small**, with 1536 dimensions.
- **text-embedding-3-large**, with 3072 dimensions.
- **Ada 002 (Text)**, with 1536 dimensions.

[Learn more](https://platform.openai.com/docs/guides/embeddings).

- **Vertex AI**: Use [Vertex AI](https://cloud.google.com/vertex-ai) to generate embeddings by using the [textembedding-gecko@001](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings) model, with 768 dimensions.
22 changes: 18 additions & 4 deletions platform/workflows.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -171,8 +171,15 @@ There are two ways to create a custom workflow:
16. In the **Embed** area, for **Provider**, choose one of the following:

- **None**: Do not generate embeddings.
- **OpenAI**: Use OpenAI to generate embeddings.
- **Vertex AI**: Use Vertex AI to generate embeddings.
- **OpenAI**: Use OpenAI to generate embeddings. Also, choose the model to use:

- **text-embedding-3-small**, with 1536 dimensions.
- **text-embedding-3-large**, with 3072 dimensions.
- **Ada 002 (Text)**, with 1536 dimensions.

[Learn more](https://platform.openai.com/docs/guides/embeddings).

- **Vertex AI**: Use Vertex AI to generate embeddings by using the `textembedding-gecko@001` model, with 768 dimensions. [Learn more](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings).

Learn more:

Expand Down Expand Up @@ -299,8 +306,15 @@ There are two ways to create a custom workflow:
<Accordion title="Embedder node">
For **Providers**, select one of the following:

- **OpenAI**: Use OpenAI to generate embeddings.
- **Vertex AI**: Use Vertex AI to generate embeddings.
- **OpenAI**: Use OpenAI to generate embeddings. Also, choose the model to use:

- **text-embedding-3-small**, with 1536 dimensions.
- **text-embedding-3-large**, with 3072 dimensions.
- **Ada 002 (Text)**, with 1536 dimensions.

[Learn more](https://platform.openai.com/docs/guides/embeddings).

- **Vertex AI**: Use Vertex AI to generate embeddings by using the `textembedding-gecko@001` model, with 768 dimensions. [Learn more](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings).

Learn more:

Expand Down
12 changes: 6 additions & 6 deletions snippets/ingest-configuration-shared/embedding-configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,16 @@ A common embedding configuration is a critical component that allows for dynamic

* `aws-bedrock`: None

* `huggingface`: `sentence-transformers/all-MiniLM-L6-v2`
* `huggingface`: `sentence-transformers/all-MiniLM-L6-v2`, with 384 dimensions

* `mixedbread-ai`: `mixedbread-ai/mxbai-embed-large-v1`
* `mixedbread-ai`: `mixedbread-ai/mxbai-embed-large-v1`, with 1024 dimensions

* `octoai`: `thenlper/gte-large`
* `octoai`: `thenlper/gte-large`, with 1024 dimensions

* `openai`: `text-embedding-ada-002`
* `openai`: `text-embedding-ada-002`, with 1536 dimensions

* `togetherai`: `togethercomputer/m2-bert-80M-8k-retrieval`
* `togetherai`: `togethercomputer/m2-bert-80M-8k-retrieval`, with 768 dimensions

* `vertexai`: `textembedding-gecko@001`
* `vertexai`: `textembedding-gecko@001`, with 768 dimensions

* `voyageai`: None

0 comments on commit 4025330

Please sign in to comment.