From 660960a68ec805d088224993ce0704dea282663a Mon Sep 17 00:00:00 2001 From: bracesproul Date: Fri, 2 Aug 2024 13:23:05 -0700 Subject: [PATCH] additional cells and drop old doc --- .../integrations/text_embedding/openai.ipynb | 114 ++++++++++++++++++ .../integrations/text_embedding/openai.mdx | 77 ------------ 2 files changed, 114 insertions(+), 77 deletions(-) delete mode 100644 docs/core_docs/docs/integrations/text_embedding/openai.mdx diff --git a/docs/core_docs/docs/integrations/text_embedding/openai.ipynb b/docs/core_docs/docs/integrations/text_embedding/openai.ipynb index 3450afb7bbc1..3c820eec3d55 100644 --- a/docs/core_docs/docs/integrations/text_embedding/openai.ipynb +++ b/docs/core_docs/docs/integrations/text_embedding/openai.ipynb @@ -91,6 +91,15 @@ "});" ] }, + { + "cell_type": "markdown", + "id": "fb4153d3", + "metadata": {}, + "source": [ + "If you're part of an organization, you can set `process.env.OPENAI_ORGANIZATION` to your OpenAI organization id, or pass it in as `organization` when\n", + "initializing the model." + ] + }, { "cell_type": "markdown", "id": "77d271b6", @@ -270,6 +279,111 @@ "console.log(vectors[1].slice(0, 100));" ] }, + { + "cell_type": "markdown", + "id": "2b1a3527", + "metadata": {}, + "source": [ + "## Specifying dimensions\n", + "\n", + "With the `text-embedding-3` class of models, you can specify the size of the embeddings you want returned. For example by default `text-embedding-3-large` returns embeddings of dimension 3072:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "a611fe1a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3072\n" + ] + } + ], + "source": [ + "import { OpenAIEmbeddings } from \"@langchain/openai\";\n", + "\n", + "const embeddingsDefaultDimensions = new OpenAIEmbeddings({\n", + " model: \"text-embedding-3-large\",\n", + "});\n", + "\n", + "const vectorsDefaultDimensions = await embeddingsDefaultDimensions.embedDocuments([\"some text\"]);\n", + "console.log(vectorsDefaultDimensions[0].length);" + ] + }, + { + "cell_type": "markdown", + "id": "08efe771", + "metadata": {}, + "source": [ + "But by passing in `dimensions: 1024` we can reduce the size of our embeddings to 1024:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "19667fdb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1024\n" + ] + } + ], + "source": [ + "import { OpenAIEmbeddings } from \"@langchain/openai\";\n", + "\n", + "const embeddings1024 = new OpenAIEmbeddings({\n", + " model: \"text-embedding-3-large\",\n", + " dimensions: 1024,\n", + "});\n", + "\n", + "const vectors1024 = await embeddings1024.embedDocuments([\"some text\"]);\n", + "console.log(vectors1024[0].length);" + ] + }, + { + "cell_type": "markdown", + "id": "6b84c0df", + "metadata": {}, + "source": [ + "## Custom URLs\n", + "\n", + "You can customize the base URL the SDK sends requests to by passing a `configuration` parameter like this:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3bfa20a6", + "metadata": {}, + "outputs": [], + "source": [ + "import { OpenAIEmbeddings } from \"@langchain/openai\";\n", + "\n", + "const model = new OpenAIEmbeddings({\n", + " configuration: {\n", + " baseURL: \"https://your_custom_url.com\",\n", + " },\n", + "});" + ] + }, + { + "cell_type": "markdown", + "id": "ac3cac9b", + "metadata": {}, + "source": [ + "You can also pass other `ClientOptions` parameters accepted by the official SDK.\n", + "\n", + "If you are hosting on Azure OpenAI, see the [dedicated page instead](/docs/integrations/text_embedding/azure_openai)." + ] + }, { "cell_type": "markdown", "id": "8938e581", diff --git a/docs/core_docs/docs/integrations/text_embedding/openai.mdx b/docs/core_docs/docs/integrations/text_embedding/openai.mdx deleted file mode 100644 index bba94b0777ee..000000000000 --- a/docs/core_docs/docs/integrations/text_embedding/openai.mdx +++ /dev/null @@ -1,77 +0,0 @@ ---- -keywords: [openaiembeddings] ---- - -# OpenAI - -The `OpenAIEmbeddings` class uses the OpenAI API to generate embeddings for a given text. By default it strips new line characters from the text, as recommended by OpenAI, but you can disable this by passing `stripNewLines: false` to the constructor. - -import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; - - - -```bash npm2yarn -npm install @langchain/openai -``` - -```typescript -import { OpenAIEmbeddings } from "@langchain/openai"; - -const embeddings = new OpenAIEmbeddings({ - apiKey: "YOUR-API-KEY", // In Node.js defaults to process.env.OPENAI_API_KEY - batchSize: 512, // Default value if omitted is 512. Max is 2048 - model: "text-embedding-3-large", -}); -``` - -If you're part of an organization, you can set `process.env.OPENAI_ORGANIZATION` to your OpenAI organization id, or pass it in as `organization` when -initializing the model. - -## Specifying dimensions - -With the `text-embedding-3` class of models, you can specify the size of the embeddings you want returned. For example by default `text-embedding-3-large` returns embeddings of dimension 3072: - -```typescript -const embeddings = new OpenAIEmbeddings({ - model: "text-embedding-3-large", -}); - -const vectors = await embeddings.embedDocuments(["some text"]); -console.log(vectors[0].length); -``` - -``` -3072 -``` - -But by passing in `dimensions: 1024` we can reduce the size of our embeddings to 1024: - -```typescript -const embeddings1024 = new OpenAIEmbeddings({ - model: "text-embedding-3-large", - dimensions: 1024, -}); - -const vectors2 = await embeddings1024.embedDocuments(["some text"]); -console.log(vectors2[0].length); -``` - -``` -1024 -``` - -## Custom URLs - -You can customize the base URL the SDK sends requests to by passing a `configuration` parameter like this: - -```typescript -const model = new OpenAIEmbeddings({ - configuration: { - baseURL: "https://your_custom_url.com", - }, -}); -``` - -You can also pass other `ClientOptions` parameters accepted by the official SDK. - -If you are hosting on Azure OpenAI, see the [dedicated page instead](/docs/integrations/text_embedding/azure_openai).