diff --git a/docs/core_docs/docs/integrations/chat/anthropic.ipynb b/docs/core_docs/docs/integrations/chat/anthropic.ipynb new file mode 100644 index 000000000000..3aea237d0a3a --- /dev/null +++ b/docs/core_docs/docs/integrations/chat/anthropic.ipynb @@ -0,0 +1,1011 @@ +{ + "cells": [ + { + "cell_type": "raw", + "id": "afaf8039", + "metadata": { + "vscode": { + "languageId": "raw" + } + }, + "source": [ + "---\n", + "sidebar_label: Anthropic\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "e49f1e0d", + "metadata": {}, + "source": [ + "# ChatAnthropic\n", + "\n", + "This will help you getting started with ChatAnthropic [chat models](/docs/concepts/#chat-models). For detailed documentation of all ChatAnthropic features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_anthropic.ChatAnthropic.html).\n", + "\n", + "## Overview\n", + "### Integration details\n", + "\n", + "| Class | Package | Local | Serializable | [PY support](https://python.langchain.com/docs/integrations/chat/anthropic/) | Package downloads | Package latest |\n", + "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", + "| [ChatAnthropic](https://api.js.langchain.com/classes/langchain_anthropic.ChatAnthropic.html) | [@langchain/anthropic](https://api.js.langchain.com/modules/langchain_anthropic.html) | ❌ | ✅ | ✅ | ![NPM - Downloads](https://img.shields.io/npm/dm/@langchain/anthropic?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/@langchain/anthropic?style=flat-square&label=%20&) |\n", + "\n", + "### Model features\n", + "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n", + "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n", + "| ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | \n", + "\n", + "## Setup\n", + "\n", + "You'll need to sign up and obtain an [Anthropic API key](https://www.anthropic.com/), and install the `@langchain/anthropic` integration package.\n", + "\n", + "### Credentials\n", + "\n", + "Head to [Anthropic's website](https://www.anthropic.com/) to sign up to Anthropic and generate an API key. Once you've done this set the `ANTHROPIC_API_KEY` environment variable:\n", + "\n", + "```bash\n", + "export ANTHROPIC_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n", + "\n", + "```bash\n", + "# export LANGCHAIN_TRACING_V2=\"true\"\n", + "# export LANGCHAIN_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "### Installation\n", + "\n", + "The LangChain ChatAnthropic integration lives in the `@langchain/anthropic` package:\n", + "\n", + "```{=mdx}\n", + "\n", + "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", + "import Npm2Yarn from \"@theme/Npm2Yarn\";\n", + "\n", + "\n", + "\n", + "\n", + " @langchain/anthropic\n", + "\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "a38cde65-254d-4219-a441-068766c0d4b5", + "metadata": {}, + "source": [ + "## Instantiation\n", + "\n", + "Now we can instantiate our model object and generate chat completions:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae", + "metadata": {}, + "outputs": [], + "source": [ + "import { ChatAnthropic } from \"@langchain/anthropic\" \n", + "\n", + "const llm = new ChatAnthropic({\n", + " model: \"claude-3-haiku-20240307\",\n", + " temperature: 0,\n", + " maxTokens: undefined,\n", + " maxRetries: 2,\n", + " // other params...\n", + "})" + ] + }, + { + "cell_type": "markdown", + "id": "2b4f3e15", + "metadata": {}, + "source": [ + "## Invocation" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "62e0dbc3", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"msg_01M9yt3aSqKJKM1RnZF4f44Q\",\n", + " \"content\": \"Voici la traduction en français :\\n\\nJ'adore la programmation.\",\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_01M9yt3aSqKJKM1RnZF4f44Q\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 29,\n", + " \"output_tokens\": 20\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_01M9yt3aSqKJKM1RnZF4f44Q\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 29,\n", + " \"output_tokens\": 20\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 29,\n", + " \"output_tokens\": 20,\n", + " \"total_tokens\": 49\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "const aiMsg = await llm.invoke([\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n", + " ],\n", + " [\"human\", \"I love programming.\"],\n", + "])\n", + "aiMsg" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Voici la traduction en français :\n", + "\n", + "J'adore la programmation.\n" + ] + } + ], + "source": [ + "console.log(aiMsg.content)" + ] + }, + { + "cell_type": "markdown", + "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8", + "metadata": {}, + "source": [ + "## Chaining\n", + "\n", + "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"msg_012gUKUG65teaois31W3bfGF\",\n", + " \"content\": \"Ich liebe das Programmieren.\",\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_012gUKUG65teaois31W3bfGF\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 23,\n", + " \"output_tokens\": 11\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_012gUKUG65teaois31W3bfGF\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 23,\n", + " \"output_tokens\": 11\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 23,\n", + " \"output_tokens\": 11,\n", + " \"total_tokens\": 34\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatPromptTemplate } from \"@langchain/core/prompts\"\n", + "\n", + "const prompt = ChatPromptTemplate.fromMessages(\n", + " [\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + " ]\n", + ")\n", + "\n", + "const chain = prompt.pipe(llm);\n", + "await chain.invoke(\n", + " {\n", + " input_language: \"English\",\n", + " output_language: \"German\",\n", + " input: \"I love programming.\",\n", + " }\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "d1ee55bc-ffc8-4cfa-801c-993953a08cfd", + "metadata": {}, + "source": [ + "## Multimodal inputs\n", + "\n", + "Claude-3 models support image multimodal inputs. The passed input must be a base64 encoded image with the\n", + "filetype as a prefix (e.g. `data:image/png;base64,{YOUR_BASE64_ENCODED_DATA}`).\n", + "Here's an example:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "1cb65e95", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"msg_01AuGpm6xbacTwoUFdNiCnzu\",\n", + " \"content\": \"The image shows a hot dog. It consists of a cylindrical bread roll or bun that has been sliced lengthwise, revealing the bright red hot dog sausage filling inside. The hot dog sausage appears to be made from seasoned and smoked meat. This classic fast food item is a popular snack or meal, commonly enjoyed at sporting events, cookouts, and casual eateries.\",\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_01AuGpm6xbacTwoUFdNiCnzu\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-sonnet-20240229\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 276,\n", + " \"output_tokens\": 88\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_01AuGpm6xbacTwoUFdNiCnzu\",\n", + " \"model\": \"claude-3-sonnet-20240229\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 276,\n", + " \"output_tokens\": 88\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 276,\n", + " \"output_tokens\": 88,\n", + " \"total_tokens\": 364\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import fs from \"fs/promises\";\n", + "\n", + "import { ChatAnthropic } from \"@langchain/anthropic\";\n", + "import { HumanMessage } from \"@langchain/core/messages\";\n", + "\n", + "const imageData2 = await fs.readFile(\"../../../../../examples/hotdog.jpg\");\n", + "const llm2 = new ChatAnthropic({\n", + " model: \"claude-3-sonnet-20240229\",\n", + "});\n", + "const message2 = new HumanMessage({\n", + " content: [\n", + " {\n", + " type: \"text\",\n", + " text: \"What's in this image?\",\n", + " },\n", + " {\n", + " type: \"image_url\",\n", + " image_url: {\n", + " url: `data:image/jpeg;base64,${imageData2.toString(\"base64\")}`,\n", + " },\n", + " },\n", + " ],\n", + "});\n", + "\n", + "await llm2.invoke([message2]);" + ] + }, + { + "cell_type": "markdown", + "id": "5c14fbc0", + "metadata": {}, + "source": [ + "See [the official docs](https://docs.anthropic.com/claude/docs/vision#what-image-file-types-does-claude-support)\n", + "for a complete list of supported file types." + ] + }, + { + "cell_type": "markdown", + "id": "9bce78a1", + "metadata": {}, + "source": [ + "## Agents\n", + "\n", + "Anthropic models that support tool calling can be used in the Tool Calling agent. Here's an example:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "0648b504", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " {\n", + " index: 0,\n", + " type: 'text',\n", + " text: '\\n\\nThe current weather in San Francisco, CA is 28°C.'\n", + " }\n", + "]\n" + ] + } + ], + "source": [ + "import { z } from \"zod\";\n", + "\n", + "import { ChatAnthropic } from \"@langchain/anthropic\";\n", + "import { tool } from \"@langchain/core/tools\";\n", + "import { AgentExecutor, createToolCallingAgent } from \"langchain/agents\";\n", + "\n", + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "\n", + "const llm3 = new ChatAnthropic({\n", + " model: \"claude-3-sonnet-20240229\",\n", + " temperature: 0,\n", + "});\n", + "\n", + "// Prompt template must have \"input\" and \"agent_scratchpad input variables\"\n", + "const prompt3 = ChatPromptTemplate.fromMessages([\n", + " [\"system\", \"You are a helpful assistant\"],\n", + " [\"placeholder\", \"{chat_history}\"],\n", + " [\"human\", \"{input}\"],\n", + " [\"placeholder\", \"{agent_scratchpad}\"],\n", + "]);\n", + "\n", + "const currentWeatherTool3 = tool(async () => \"28 °C\", {\n", + " name: \"get_current_weather\",\n", + " description: \"Get the current weather in a given location\",\n", + " schema: z.object({\n", + " location: z.string().describe(\"The city and state, e.g. San Francisco, CA\"),\n", + " }),\n", + "});\n", + "\n", + "const agent3 = createToolCallingAgent({\n", + " llm: llm3,\n", + " tools: [currentWeatherTool3],\n", + " prompt: prompt3,\n", + "});\n", + "\n", + "const agentExecutor3 = new AgentExecutor({\n", + " agent: agent3,\n", + " tools: [currentWeatherTool3],\n", + "});\n", + "\n", + "const input3 = \"What's the weather like in SF?\";\n", + "const result3 = await agentExecutor3.invoke({ input: input3 });\n", + "\n", + "console.log(result3.output);" + ] + }, + { + "cell_type": "markdown", + "id": "d452d4b6", + "metadata": {}, + "source": [ + "## Custom headers\n", + "\n", + "You can pass custom headers in your requests like this:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "41943f0a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"msg_013Ft3kN62gNtiMWRqg6xxt8\",\n", + " \"content\": \"The sky appears blue due to a phenomenon called Rayleigh scattering. Here's a brief explanation:\\n\\n1) Sunlight is made up of different wavelengths of light, including the visible spectrum that we see as colors.\\n\\n2) As sunlight passes through the Earth's atmosphere, the different wavelengths of light interact with the gas molecules in the air.\\n\\n3) The shorter wavelengths of light, such as the blue and violet colors, get scattered more easily by the tiny gas molecules. This is because the wavelengths are similar in size to the molecules.\\n\\n4) The longer wavelengths of light, such as red and orange, get scattered much less by the gas molecules and travel more directly through the atmosphere.\\n\\n5) The blue wavelengths that are scattered in different directions become scattered across the entire sky, making the sky appear blue to our eyes.\\n\\n6) During sunrise and sunset, the sun's rays travel through more atmosphere before reaching our eyes, causing the blue light to get scattered away and allowing more of the red/orange wavelengths to pass through, giving those colors in the sky.\\n\\nSo in essence, the abundant scattering of blue light by the gas molecules in the atmosphere is what causes the sky to appear blue during the daytime.\",\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_013Ft3kN62gNtiMWRqg6xxt8\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-sonnet-20240229\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 13,\n", + " \"output_tokens\": 272\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_013Ft3kN62gNtiMWRqg6xxt8\",\n", + " \"model\": \"claude-3-sonnet-20240229\",\n", + " \"stop_reason\": \"end_turn\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 13,\n", + " \"output_tokens\": 272\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 13,\n", + " \"output_tokens\": 272,\n", + " \"total_tokens\": 285\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatAnthropic } from \"@langchain/anthropic\";\n", + "\n", + "const llm4 = new ChatAnthropic({\n", + " model: \"claude-3-sonnet-20240229\",\n", + " maxTokens: 1024,\n", + " clientOptions: {\n", + " defaultHeaders: {\n", + " \"X-Api-Key\": process.env.ANTHROPIC_API_KEY,\n", + " },\n", + " },\n", + "});\n", + "\n", + "const res4 = await llm4.invoke(\"Why is the sky blue?\");\n", + "\n", + "console.log(res4);" + ] + }, + { + "cell_type": "markdown", + "id": "985c4b4b", + "metadata": {}, + "source": [ + "## Tools\n", + "\n", + "The Anthropic API supports tool calling, along with multi-tool calling. The following examples demonstrate how to call tools:\n", + "\n", + "### Single Tool" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "2ce56548", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"msg_01XPUHrR4sNCqPr1i9zcsAsg\",\n", + " \"content\": [\n", + " {\n", + " \"type\": \"text\",\n", + " \"text\": \"Okay, let me use the calculator tool to find the answer:\"\n", + " },\n", + " {\n", + " \"type\": \"tool_use\",\n", + " \"id\": \"toolu_01MhUVuUedc1drBKLarhedFZ\",\n", + " \"name\": \"calculator\",\n", + " \"input\": {\n", + " \"number1\": 2,\n", + " \"number2\": 2,\n", + " \"operation\": \"add\"\n", + " }\n", + " }\n", + " ],\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_01XPUHrR4sNCqPr1i9zcsAsg\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"tool_use\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 449,\n", + " \"output_tokens\": 101\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_01XPUHrR4sNCqPr1i9zcsAsg\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"tool_use\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 449,\n", + " \"output_tokens\": 101\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [\n", + " {\n", + " \"name\": \"calculator\",\n", + " \"args\": {\n", + " \"number1\": 2,\n", + " \"number2\": 2,\n", + " \"operation\": \"add\"\n", + " },\n", + " \"id\": \"toolu_01MhUVuUedc1drBKLarhedFZ\",\n", + " \"type\": \"tool_call\"\n", + " }\n", + " ],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 449,\n", + " \"output_tokens\": 101,\n", + " \"total_tokens\": 550\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatAnthropic } from \"@langchain/anthropic\";\n", + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "import { z } from \"zod\";\n", + "import { zodToJsonSchema } from \"zod-to-json-schema\";\n", + "\n", + "const calculatorSchema5 = z.object({\n", + " operation: z\n", + " .enum([\"add\", \"subtract\", \"multiply\", \"divide\"])\n", + " .describe(\"The type of operation to execute.\"),\n", + " number1: z.number().describe(\"The first number to operate on.\"),\n", + " number2: z.number().describe(\"The second number to operate on.\"),\n", + "});\n", + "\n", + "const tool5 = {\n", + " name: \"calculator\",\n", + " description: \"A simple calculator tool\",\n", + " input_schema: zodToJsonSchema(calculatorSchema5),\n", + "};\n", + "\n", + "const llm5 = new ChatAnthropic({\n", + " apiKey: process.env.ANTHROPIC_API_KEY,\n", + " model: \"claude-3-haiku-20240307\",\n", + "}).bind({\n", + " tools: [tool5],\n", + "});\n", + "\n", + "const prompt5 = ChatPromptTemplate.fromMessages([\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant who always needs to use a calculator.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + "]);\n", + "\n", + "// Chain your prompt and model together\n", + "const chain5 = prompt5.pipe(llm5);\n", + "\n", + "const response5 = await chain5.invoke({\n", + " input: \"What is 2 + 2?\",\n", + "});\n", + "console.log(response5);" + ] + }, + { + "cell_type": "markdown", + "id": "6e91f97b", + "metadata": {}, + "source": [ + "### Forced tool calling\n", + "\n", + "In this example we'll provide the model with two tools:\n", + "\n", + "- `calculator`\n", + "- `get_weather`\n", + "\n", + "Then, when we call `bindTools`, we'll force the model to use the `get_weather` tool by passing the `tool_choice` arg like this:\n", + "\n", + "```typescript\n", + ".bindTools({\n", + " tools,\n", + " tool_choice: {\n", + " type: \"tool\",\n", + " name: \"get_weather\",\n", + " }\n", + "});\n", + "```\n", + "\n", + "Finally, we'll invoke the model, but instead of asking about the weather, we'll ask it to do some math.\n", + "Since we explicitly forced the model to use the `get_weather` tool, it will ignore the input and return the weather information (in this case it returned ``, which is expected.)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "8d6e4828", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"msg_018G4mEZu8KNKtaQxZQ3o8YB\",\n", + " \"content\": [\n", + " {\n", + " \"type\": \"tool_use\",\n", + " \"id\": \"toolu_01DS9RwsFKdhHNYmhwPJHdHa\",\n", + " \"name\": \"get_weather\",\n", + " \"input\": {\n", + " \"city\": \"\",\n", + " \"state\": \"\"\n", + " }\n", + " }\n", + " ],\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_018G4mEZu8KNKtaQxZQ3o8YB\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"tool_use\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 672,\n", + " \"output_tokens\": 51\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_018G4mEZu8KNKtaQxZQ3o8YB\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"tool_use\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 672,\n", + " \"output_tokens\": 51\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [\n", + " {\n", + " \"name\": \"get_weather\",\n", + " \"args\": {\n", + " \"city\": \"\",\n", + " \"state\": \"\"\n", + " },\n", + " \"id\": \"toolu_01DS9RwsFKdhHNYmhwPJHdHa\",\n", + " \"type\": \"tool_call\"\n", + " }\n", + " ],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 672,\n", + " \"output_tokens\": 51,\n", + " \"total_tokens\": 723\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatAnthropic } from \"@langchain/anthropic\";\n", + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "import { z } from \"zod\";\n", + "import { zodToJsonSchema } from \"zod-to-json-schema\";\n", + "\n", + "const calculatorSchema6 = z.object({\n", + " operation: z\n", + " .enum([\"add\", \"subtract\", \"multiply\", \"divide\"])\n", + " .describe(\"The type of operation to execute.\"),\n", + " number1: z.number().describe(\"The first number to operate on.\"),\n", + " number2: z.number().describe(\"The second number to operate on.\"),\n", + "});\n", + "\n", + "const weatherSchema6 = z.object({\n", + " city: z.string().describe(\"The city to get the weather from\"),\n", + " state: z.string().optional().describe(\"The state to get the weather from\"),\n", + "});\n", + "\n", + "const tools6 = [\n", + " {\n", + " name: \"calculator\",\n", + " description: \"A simple calculator tool\",\n", + " input_schema: zodToJsonSchema(calculatorSchema6),\n", + " },\n", + " {\n", + " name: \"get_weather\",\n", + " description:\n", + " \"Get the weather of a specific location and return the temperature in Celsius.\",\n", + " input_schema: zodToJsonSchema(weatherSchema6),\n", + " },\n", + "];\n", + "\n", + "const llm6 = new ChatAnthropic({\n", + " apiKey: process.env.ANTHROPIC_API_KEY,\n", + " model: \"claude-3-haiku-20240307\",\n", + "}).bind({\n", + " tools: tools6,\n", + " tool_choice: {\n", + " type: \"tool\",\n", + " name: \"get_weather\",\n", + " },\n", + "});\n", + "\n", + "const prompt6 = ChatPromptTemplate.fromMessages([\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant who always needs to use a calculator.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + "]);\n", + "\n", + "// Chain your prompt and model together\n", + "const chain6 = prompt6.pipe(llm6);\n", + "\n", + "const response6 = await chain6.invoke({\n", + " input: \"What is the sum of 2725 and 273639\",\n", + "});\n", + "\n", + "console.log(response6);" + ] + }, + { + "cell_type": "markdown", + "id": "1aa777bc", + "metadata": {}, + "source": [ + "The `tool_choice` argument has three possible values:\n", + "\n", + "- `{ type: \"tool\", name: \"tool_name\" }` | `string` - Forces the model to use the specified tool. If passing a single string, it will be treated as the tool name.\n", + "- `\"any\"` - Allows the model to choose the tool, but still forcing it to choose at least one.\n", + "- `\"auto\"` - The default value. Allows the model to select any tool, or none." + ] + }, + { + "cell_type": "markdown", + "id": "15253085", + "metadata": {}, + "source": [ + "### `withStructuredOutput`" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "5e466d35", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{ operation: 'add', number1: 2, number2: 2 }\n" + ] + } + ], + "source": [ + "import { ChatAnthropic } from \"@langchain/anthropic\";\n", + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "import { z } from \"zod\";\n", + "\n", + "const calculatorSchema7 = z\n", + " .object({\n", + " operation: z\n", + " .enum([\"add\", \"subtract\", \"multiply\", \"divide\"])\n", + " .describe(\"The type of operation to execute.\"),\n", + " number1: z.number().describe(\"The first number to operate on.\"),\n", + " number2: z.number().describe(\"The second number to operate on.\"),\n", + " })\n", + " .describe(\"A simple calculator tool\");\n", + "\n", + "const llm7 = new ChatAnthropic({\n", + " apiKey: process.env.ANTHROPIC_API_KEY,\n", + " model: \"claude-3-haiku-20240307\",\n", + "});\n", + "\n", + "// Pass the schema and tool name to the withStructuredOutput method\n", + "const modelWithTool7 = llm7.withStructuredOutput(calculatorSchema7);\n", + "\n", + "const prompt7 = ChatPromptTemplate.fromMessages([\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant who always needs to use a calculator.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + "]);\n", + "\n", + "// Chain your prompt and model together\n", + "const chain7 = prompt7.pipe(modelWithTool7);\n", + "\n", + "const response7 = await chain7.invoke({\n", + " input: \"What is 2 + 2?\",\n", + "});\n", + "console.log(response7);" + ] + }, + { + "cell_type": "markdown", + "id": "4973b265", + "metadata": {}, + "source": [ + "You can supply a \"name\" field to give the LLM additional context around what you are trying to generate. You can also pass `includeRaw` to get the raw message back from the model too." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "951c5352", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " raw: AIMessage {\n", + " \"id\": \"msg_01TrkHbEkioCYNHQhqxw5unu\",\n", + " \"content\": [\n", + " {\n", + " \"type\": \"tool_use\",\n", + " \"id\": \"toolu_01XMrGHXeSVTfSw1oKFZokzG\",\n", + " \"name\": \"calculator\",\n", + " \"input\": {\n", + " \"number1\": 2,\n", + " \"number2\": 2,\n", + " \"operation\": \"add\"\n", + " }\n", + " }\n", + " ],\n", + " \"additional_kwargs\": {\n", + " \"id\": \"msg_01TrkHbEkioCYNHQhqxw5unu\",\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"tool_use\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 552,\n", + " \"output_tokens\": 69\n", + " }\n", + " },\n", + " \"response_metadata\": {\n", + " \"id\": \"msg_01TrkHbEkioCYNHQhqxw5unu\",\n", + " \"model\": \"claude-3-haiku-20240307\",\n", + " \"stop_reason\": \"tool_use\",\n", + " \"stop_sequence\": null,\n", + " \"usage\": {\n", + " \"input_tokens\": 552,\n", + " \"output_tokens\": 69\n", + " },\n", + " \"type\": \"message\",\n", + " \"role\": \"assistant\"\n", + " },\n", + " \"tool_calls\": [\n", + " {\n", + " \"name\": \"calculator\",\n", + " \"args\": {\n", + " \"number1\": 2,\n", + " \"number2\": 2,\n", + " \"operation\": \"add\"\n", + " },\n", + " \"id\": \"toolu_01XMrGHXeSVTfSw1oKFZokzG\",\n", + " \"type\": \"tool_call\"\n", + " }\n", + " ],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 552,\n", + " \"output_tokens\": 69,\n", + " \"total_tokens\": 621\n", + " }\n", + " },\n", + " parsed: { operation: 'add', number1: 2, number2: 2 }\n", + "}\n" + ] + } + ], + "source": [ + "const includeRawModel7 = llm7.withStructuredOutput(calculatorSchema7, {\n", + " name: \"calculator\",\n", + " includeRaw: true,\n", + "});\n", + "const includeRawChain7 = prompt7.pipe(includeRawModel7);\n", + "\n", + "const includeRawResponse7 = await includeRawChain7.invoke({\n", + " input: \"What is 2 + 2?\",\n", + "});\n", + "\n", + "console.log(includeRawResponse7);" + ] + }, + { + "cell_type": "markdown", + "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3", + "metadata": {}, + "source": [ + "## API reference\n", + "\n", + "For detailed documentation of all ChatAnthropic features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_anthropic.ChatAnthropic.html" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "TypeScript", + "language": "typescript", + "name": "tslab" + }, + "language_info": { + "codemirror_mode": { + "mode": "typescript", + "name": "javascript", + "typescript": true + }, + "file_extension": ".ts", + "mimetype": "text/typescript", + "name": "typescript", + "version": "3.7.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/core_docs/docs/integrations/chat/anthropic.mdx b/docs/core_docs/docs/integrations/chat/anthropic.mdx deleted file mode 100644 index 416a1ac16132..000000000000 --- a/docs/core_docs/docs/integrations/chat/anthropic.mdx +++ /dev/null @@ -1,126 +0,0 @@ ---- -sidebar_label: Anthropic ---- - -# ChatAnthropic - -LangChain supports Anthropic's Claude family of chat models. - -You'll first need to install the [`@langchain/anthropic`](https://www.npmjs.com/package/@langchain/anthropic) package: - -import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; - - - -```bash npm2yarn -npm install @langchain/anthropic -``` - -You'll also need to sign up and obtain an [Anthropic API key](https://www.anthropic.com/). -Set it as an environment variable named `ANTHROPIC_API_KEY`, or pass it into the constructor as shown below. - -## Usage - -import UnifiedModelParamsTooltip from "@mdx_components/unified_model_params_tooltip.mdx"; - - - -You can initialize an instance like this: - -import CodeBlock from "@theme/CodeBlock"; -import Anthropic from "@examples/models/chat/integration_anthropic.ts"; - -{Anthropic} - -## Multimodal inputs - -Claude-3 models support image multimodal inputs. The passed input must be a base64 encoded image with the -filetype as a prefix (e.g. `data:image/png;base64,{YOUR_BASE64_ENCODED_DATA}`). -Here's an example: - -import AnthropicMultimodal from "@examples/models/chat/integration_anthropic_multimodal.ts"; - -{AnthropicMultimodal} - -See [the official docs](https://docs.anthropic.com/claude/docs/vision#what-image-file-types-does-claude-support) -for a complete list of supported file types. - -## Agents - -Anthropic models that support tool calling can be used in the Tool Calling agent. Here's an example: - -import AnthropicToolCallingAgent from "@examples/models/chat/integration_anthropic_tool_calling_agent.ts"; - -{AnthropicToolCallingAgent} - -:::tip -See the LangSmith trace [here](https://smith.langchain.com/public/e93ff7f6-03f7-4eb1-96c8-09a17dee1462/r) -::: - -## Custom headers - -You can pass custom headers in your requests like this: - -import AnthropicCustomHeaders from "@examples/models/chat/integration_anthropic_custom_headers.ts"; - -{AnthropicCustomHeaders} - -## Tools - -The Anthropic API supports tool calling, along with multi-tool calling. The following examples demonstrate how to call tools: - -### Single Tool - -import AnthropicSingleTool from "@examples/models/chat/integration_anthropic_single_tool.ts"; - -{AnthropicSingleTool} - -:::tip -See the LangSmith trace [here](https://smith.langchain.com/public/90c03ed0-154b-4a50-afbf-83dcbf302647/r) -::: - -### Forced tool calling - -import AnthropicForcedTool from "@examples/models/chat/integration_anthropic_forced_tool.ts"; - -In this example we'll provide the model with two tools: - -- `calculator` -- `get_weather` - -Then, when we call `bindTools`, we'll force the model to use the `get_weather` tool by passing the `tool_choice` arg like this: - -```typescript -.bindTools({ - tools, - tool_choice: { - type: "tool", - name: "get_weather", - } -}); -``` - -Finally, we'll invoke the model, but instead of asking about the weather, we'll ask it to do some math. -Since we explicitly forced the model to use the `get_weather` tool, it will ignore the input and return the weather information (in this case it returned ``, which is expected.) - -{AnthropicForcedTool} - -The `bind_tools` argument has three possible values: - -- `{ type: "tool", name: "tool_name" }` - Forces the model to use the specified tool. -- `"any"` - Allows the model to choose the tool, but still forcing it to choose at least one. -- `"auto"` - The default value. Allows the model to select any tool, or none. - -:::tip -See the LangSmith trace [here](https://smith.langchain.com/public/c5cc8fe7-5e76-4607-8c43-1e0b30e4f5ca/r) -::: - -### `withStructuredOutput` - -import AnthropicWSA from "@examples/models/chat/integration_anthropic_wsa.ts"; - -{AnthropicWSA} - -:::tip -See the LangSmith trace [here](https://smith.langchain.com/public/efbd11c5-886e-4e07-be1a-951690fa8a27/r) -::: diff --git a/docs/core_docs/docs/integrations/chat/openai.ipynb b/docs/core_docs/docs/integrations/chat/openai.ipynb new file mode 100644 index 000000000000..d63d918e2da9 --- /dev/null +++ b/docs/core_docs/docs/integrations/chat/openai.ipynb @@ -0,0 +1,1440 @@ +{ + "cells": [ + { + "cell_type": "raw", + "id": "afaf8039", + "metadata": { + "vscode": { + "languageId": "raw" + } + }, + "source": [ + "---\n", + "sidebar_label: OpenAI\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "e49f1e0d", + "metadata": {}, + "source": [ + "# ChatOpenAI\n", + "\n", + "This will help you getting started with ChatOpenAI [chat models](/docs/concepts/#chat-models). For detailed documentation of all ChatOpenAI features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html).\n", + "\n", + "## Overview\n", + "### Integration details\n", + "\n", + "| Class | Package | Local | Serializable | [PY support](https://python.langchain.com/docs/integrations/chat/openai) | Package downloads | Package latest |\n", + "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", + "| [ChatOpenAI](https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) | [@langchain/openai](https://api.js.langchain.com/modules/langchain_openai.html) | ❌ | ✅ | ✅ | ![NPM - Downloads](https://img.shields.io/npm/dm/@langchain/openai?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/@langchain/openai?style=flat-square&label=%20&) |\n", + "\n", + "### Model features\n", + "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n", + "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n", + "| ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | \n", + "\n", + "## Setup\n", + "\n", + "To access ChatOpenAI models you'll need to create a ChatOpenAI account, get an API key, and install the `@langchain/openai` integration package.\n", + "\n", + "### Credentials\n", + "\n", + "Head to [OpenAI's website](https://platform.openai.com/) to sign up to ChatOpenAI and generate an API key. Once you've done this set the `OPENAI_API_KEY` environment variable:\n", + "\n", + "```bash\n", + "export OPENAI_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n", + "\n", + "```bash\n", + "# export LANGCHAIN_TRACING_V2=\"true\"\n", + "# export LANGCHAIN_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "### Installation\n", + "\n", + "The LangChain ChatOpenAI integration lives in the `@langchain/openai` package:\n", + "\n", + "```{=mdx}\n", + "\n", + "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", + "import Npm2Yarn from \"@theme/Npm2Yarn\";\n", + "\n", + "\n", + "\n", + "\n", + " @langchain/openai\n", + "\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "a38cde65-254d-4219-a441-068766c0d4b5", + "metadata": {}, + "source": [ + "## Instantiation\n", + "\n", + "Now we can instantiate our model object and generate chat completions:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae", + "metadata": {}, + "outputs": [], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\" \n", + "\n", + "const llm = new ChatOpenAI({\n", + " model: \"gpt-4o\",\n", + " temperature: 0,\n", + " maxTokens: undefined,\n", + " timeout: undefined,\n", + " maxRetries: 2,\n", + " // other params...\n", + "})" + ] + }, + { + "cell_type": "markdown", + "id": "2b4f3e15", + "metadata": {}, + "source": [ + "## Invocation" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "62e0dbc3", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"chatcmpl-9rB4GvhlRb0x3hxupLBQYOKKmTxvV\",\n", + " \"content\": \"J'adore la programmation.\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 8,\n", + " \"promptTokens\": 31,\n", + " \"totalTokens\": 39\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 31,\n", + " \"output_tokens\": 8,\n", + " \"total_tokens\": 39\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "const aiMsg = await llm.invoke([\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n", + " ],\n", + " [\"human\", \"I love programming.\"],\n", + "])\n", + "aiMsg" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "J'adore la programmation.\n" + ] + } + ], + "source": [ + "console.log(aiMsg.content)" + ] + }, + { + "cell_type": "markdown", + "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8", + "metadata": {}, + "source": [ + "## Chaining\n", + "\n", + "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"chatcmpl-9rB4JD9rVBLzTuMee9AabulowEH0d\",\n", + " \"content\": \"Ich liebe das Programmieren.\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 6,\n", + " \"promptTokens\": 26,\n", + " \"totalTokens\": 32\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 26,\n", + " \"output_tokens\": 6,\n", + " \"total_tokens\": 32\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatPromptTemplate } from \"@langchain/core/prompts\"\n", + "\n", + "const prompt = ChatPromptTemplate.fromMessages(\n", + " [\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + " ]\n", + ")\n", + "\n", + "const chain = prompt.pipe(llm);\n", + "await chain.invoke(\n", + " {\n", + " input_language: \"English\",\n", + " output_language: \"German\",\n", + " input: \"I love programming.\",\n", + " }\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "d1ee55bc-ffc8-4cfa-801c-993953a08cfd", + "metadata": {}, + "source": [ + "## Multimodal messages\n", + "\n", + "```{=mdx}\n", + "\n", + ":::info\n", + "This feature is currently in preview. The message schema may change in future releases.\n", + ":::\n", + "\n", + "```\n", + "\n", + "OpenAI supports interleaving images with text in input messages with their `gpt-4-vision-preview`. Here's an example of how this looks:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "fd55c000", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"chatcmpl-9rB59AKTPDrSHuTv0y7BNUcM0QDV2\",\n", + " \"content\": \"The image shows a classic hot dog, consisting of a grilled or steamed sausage served in the slit of a partially sliced bun. The sausage appears to have grill marks, indicating it may have been cooked on a grill. This is a typical and popular snack or fast food item often enjoyed at sporting events, barbecues, and fairs.\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 69,\n", + " \"promptTokens\": 438,\n", + " \"totalTokens\": 507\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 438,\n", + " \"output_tokens\": 69,\n", + " \"total_tokens\": 507\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import * as fs from \"node:fs/promises\";\n", + "\n", + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "import { HumanMessage } from \"@langchain/core/messages\";\n", + "\n", + "const imageData2 = await fs.readFile(\"../../../../../examples/hotdog.jpg\");\n", + "const llm2 = new ChatOpenAI({\n", + " model: \"gpt-4-vision-preview\",\n", + " maxTokens: 1024,\n", + " apiKey: process.env.OPENAI_API_KEY,\n", + "});\n", + "const message2 = new HumanMessage({\n", + " content: [\n", + " {\n", + " type: \"text\",\n", + " text: \"What's in this image?\",\n", + " },\n", + " {\n", + " type: \"image_url\",\n", + " image_url: {\n", + " url: `data:image/jpeg;base64,${imageData2.toString(\"base64\")}`,\n", + " },\n", + " },\n", + " ],\n", + "});\n", + "\n", + "const res2 = await llm2.invoke([message2]);\n", + "console.log(res2);\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "eafbba15", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"chatcmpl-9rB5EWz5AyOHg6UiFkt4HC8H4UZJu\",\n", + " \"content\": \"The image contains text that reads \\\"LangChain\\\". Additionally, there is an illustration of a parrot on the left side and two interlinked rings on the right.\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 33,\n", + " \"promptTokens\": 778,\n", + " \"totalTokens\": 811\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 778,\n", + " \"output_tokens\": 33,\n", + " \"total_tokens\": 811\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "const hostedImageMessage3 = new HumanMessage({\n", + " content: [\n", + " {\n", + " type: \"text\",\n", + " text: \"What does this image say?\",\n", + " },\n", + " {\n", + " type: \"image_url\",\n", + " image_url:\n", + " \"https://www.freecodecamp.org/news/content/images/2023/05/Screenshot-2023-05-29-at-5.40.38-PM.png\",\n", + " },\n", + " ],\n", + "});\n", + "const res3 = await llm2.invoke([hostedImageMessage3]);\n", + "console.log(res3);" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "a3832fc3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"chatcmpl-9rB5IUbzvMo5nsOGYW3jvrQjaCiCg\",\n", + " \"content\": \"The image shows a user interface of a digital service or platform called \\\"WebLangChain\\\" which appears to be powered by \\\"Tailify.\\\" There is a prompt that encourages users to \\\"Ask me anything about anything!\\\" Alongside this, there is a text input field labeled \\\"Ask anything...\\\" which also features some example questions or search queries such as \\\"what is langchain?\\\", \\\"history of mesopotamia\\\", \\\"how to build a discord bot\\\", \\\"leonardo dicaprio girlfriend\\\", \\\"fun gift ideas for software engineers\\\", \\\"how does a prism separate light\\\", and \\\"what bear is best\\\". The overall design is clean, with a dark background and a send button represented by a blue icon with a paper airplane, which typically symbolizes sending a message or submitting a query.\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 158,\n", + " \"promptTokens\": 101,\n", + " \"totalTokens\": 259\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 101,\n", + " \"output_tokens\": 158,\n", + " \"total_tokens\": 259\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "const lowDetailImage4 = new HumanMessage({\n", + " content: [\n", + " {\n", + " type: \"text\",\n", + " text: \"Summarize the contents of this image.\",\n", + " },\n", + " {\n", + " type: \"image_url\",\n", + " image_url: {\n", + " url: \"https://blog.langchain.dev/content/images/size/w1248/format/webp/2023/10/Screenshot-2023-10-03-at-4.55.29-PM.png\",\n", + " detail: \"low\",\n", + " },\n", + " },\n", + " ],\n", + "});\n", + "const res4 = await llm2.invoke([lowDetailImage4]);\n", + "console.log(res4);" + ] + }, + { + "cell_type": "markdown", + "id": "1a39ecb3", + "metadata": {}, + "source": [ + "## Tool calling\n", + "\n", + "OpenAI chat models support calling multiple functions to get all required data to answer a question.\n", + "Here's an example how a conversation turn with this functionality might look:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "c65f489f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " {\n", + " name: 'get_current_weather',\n", + " args: { location: 'San Francisco', unit: 'celsius' },\n", + " type: 'tool_call',\n", + " id: 'call_2ytmjITA18j3kLOzzjF5QSC4'\n", + " },\n", + " {\n", + " name: 'get_current_weather',\n", + " args: { location: 'Tokyo', unit: 'celsius' },\n", + " type: 'tool_call',\n", + " id: 'call_3sU2dCNZ8e8A8wrYlYa7Xq0G'\n", + " },\n", + " {\n", + " name: 'get_current_weather',\n", + " args: { location: 'Paris', unit: 'celsius' },\n", + " type: 'tool_call',\n", + " id: 'call_Crmc0QG4x1VHRUyiwPsqzmQS'\n", + " }\n", + "]\n" + ] + } + ], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "\n", + "// Bind function to the model as a tool\n", + "const llm5 = new ChatOpenAI({\n", + " model: \"gpt-3.5-turbo-1106\",\n", + " maxTokens: 128,\n", + "}).bind({\n", + " tools: [\n", + " {\n", + " type: \"function\",\n", + " function: {\n", + " name: \"get_current_weather\",\n", + " description: \"Get the current weather in a given location\",\n", + " parameters: {\n", + " type: \"object\",\n", + " properties: {\n", + " location: {\n", + " type: \"string\",\n", + " description: \"The city and state, e.g. San Francisco, CA\",\n", + " },\n", + " unit: { type: \"string\", enum: [\"celsius\", \"fahrenheit\"] },\n", + " },\n", + " required: [\"location\"],\n", + " },\n", + " },\n", + " },\n", + " ],\n", + " tool_choice: \"auto\",\n", + "});\n", + "\n", + "// Ask initial question that requires multiple tool calls\n", + "const res5 = await llm5.invoke([\n", + " [\"human\", \"What's the weather like in San Francisco, Tokyo, and Paris?\"],\n", + "]);\n", + "console.log(res5.tool_calls);" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "c0d3a6a1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"id\": \"chatcmpl-9rB5Sc3ERHpRymmAAsGS67zczVhAl\",\n", + " \"content\": \"The current weather in:\\n- San Francisco is 72°F\\n- Tokyo is 10°C\\n- Paris is 22°C\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 27,\n", + " \"promptTokens\": 236,\n", + " \"totalTokens\": 263\n", + " },\n", + " \"finish_reason\": \"stop\",\n", + " \"system_fingerprint\": \"fp_adbef9f124\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 236,\n", + " \"output_tokens\": 27,\n", + " \"total_tokens\": 263\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ToolMessage } from \"@langchain/core/messages\";\n", + "\n", + "// Mocked out function, could be a database/API call in production\n", + "function getCurrentWeather(location: string, _unit?: string) {\n", + " if (location.toLowerCase().includes(\"tokyo\")) {\n", + " return JSON.stringify({ location, temperature: \"10\", unit: \"celsius\" });\n", + " } else if (location.toLowerCase().includes(\"san francisco\")) {\n", + " return JSON.stringify({\n", + " location,\n", + " temperature: \"72\",\n", + " unit: \"fahrenheit\",\n", + " });\n", + " } else {\n", + " return JSON.stringify({ location, temperature: \"22\", unit: \"celsius\" });\n", + " }\n", + "}\n", + "\n", + "// Format the results from calling the tool calls back to OpenAI as ToolMessages\n", + "const toolMessages5 = res5.additional_kwargs.tool_calls?.map((toolCall) => {\n", + " const toolCallResult5 = getCurrentWeather(\n", + " JSON.parse(toolCall.function.arguments).location\n", + " );\n", + " return new ToolMessage({\n", + " tool_call_id: toolCall.id,\n", + " name: toolCall.function.name,\n", + " content: toolCallResult5,\n", + " });\n", + "});\n", + "\n", + "// Send the results back as the next step in the conversation\n", + "const finalResponse5 = await llm5.invoke([\n", + " [\"human\", \"What's the weather like in San Francisco, Tokyo, and Paris?\"],\n", + " res5,\n", + " ...(toolMessages5 ?? []),\n", + "]);\n", + "\n", + "console.log(finalResponse5);" + ] + }, + { + "cell_type": "markdown", + "id": "067715fe", + "metadata": {}, + "source": [ + "### `.withStructuredOutput({ ... })`\n", + "\n", + "You can also use the `.withStructuredOutput({ ... })` method to coerce `ChatOpenAI` into returning a structured output.\n", + "\n", + "The method allows for passing in either a Zod object, or a valid JSON schema (like what is returned from [`zodToJsonSchema`](https://www.npmjs.com/package/zod-to-json-schema)).\n", + "\n", + "Using the method is simple. Just define your LLM and call `.withStructuredOutput({ ... })` on it, passing the desired schema.\n", + "\n", + "Here is an example using a Zod schema and the `functionCalling` mode (default mode):" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "94bab2ee", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{ operation: 'add', number1: 2, number2: 2 }\n" + ] + } + ], + "source": [ + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "import { z } from \"zod\";\n", + "\n", + "const llm6 = new ChatOpenAI({\n", + " temperature: 0,\n", + " model: \"gpt-4-turbo-preview\",\n", + "});\n", + "\n", + "const calculatorSchema6 = z.object({\n", + " operation: z.enum([\"add\", \"subtract\", \"multiply\", \"divide\"]),\n", + " number1: z.number(),\n", + " number2: z.number(),\n", + "});\n", + "\n", + "const modelWithStructuredOutput6 = llm6.withStructuredOutput(calculatorSchema6);\n", + "\n", + "const prompt6 = ChatPromptTemplate.fromMessages([\n", + " [\"system\", \"You are VERY bad at math and must always use a calculator.\"],\n", + " [\"human\", \"Please help me!! What is 2 + 2?\"],\n", + "]);\n", + "const chain6 = prompt6.pipe(modelWithStructuredOutput6);\n", + "const result6 = await chain6.invoke({});\n", + "console.log(result6);\n" + ] + }, + { + "cell_type": "markdown", + "id": "b6e97547", + "metadata": {}, + "source": [ + "You can also specify `includeRaw` to return the parsed and raw output in the result." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "640acaf4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " raw: AIMessage {\n", + " \"id\": \"chatcmpl-9rB5emIYRslBFrUIsC2368dXltljw\",\n", + " \"content\": \"\",\n", + " \"additional_kwargs\": {\n", + " \"tool_calls\": [\n", + " {\n", + " \"id\": \"call_JaH5OB3KYvKF76TUOt6Lp8mu\",\n", + " \"type\": \"function\",\n", + " \"function\": \"[Object]\"\n", + " }\n", + " ]\n", + " },\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 15,\n", + " \"promptTokens\": 93,\n", + " \"totalTokens\": 108\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [\n", + " {\n", + " \"name\": \"calculator\",\n", + " \"args\": {\n", + " \"number1\": 2,\n", + " \"number2\": 2,\n", + " \"operation\": \"add\"\n", + " },\n", + " \"type\": \"tool_call\",\n", + " \"id\": \"call_JaH5OB3KYvKF76TUOt6Lp8mu\"\n", + " }\n", + " ],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 93,\n", + " \"output_tokens\": 15,\n", + " \"total_tokens\": 108\n", + " }\n", + " },\n", + " parsed: { operation: 'add', number1: 2, number2: 2 }\n", + "}\n" + ] + } + ], + "source": [ + "const includeRawModel6 = llm6.withStructuredOutput(calculatorSchema6, {\n", + " name: \"calculator\",\n", + " includeRaw: true,\n", + "});\n", + "\n", + "const includeRawChain6 = prompt6.pipe(includeRawModel6);\n", + "const includeRawResult6 = await includeRawChain6.invoke({});\n", + "console.log(includeRawResult6);" + ] + }, + { + "cell_type": "markdown", + "id": "f92f236c", + "metadata": {}, + "source": [ + "Additionally, you can pass in an OpenAI function definition or JSON schema directly:\n", + "\n", + "```{=mdx}\n", + "\n", + ":::info\n", + "If using `jsonMode` as the `method` you must include context in your prompt about the structured output you want. This _must_ include the keyword: `JSON`.\n", + ":::\n", + "\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "02e01d32", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{ number1: 2, number2: 2, operation: 'add' }\n" + ] + } + ], + "source": [ + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "\n", + "const llm7 = new ChatOpenAI({\n", + " temperature: 0,\n", + " model: \"gpt-4-turbo-preview\",\n", + "});\n", + "\n", + "const calculatorSchema7 = {\n", + " type: \"object\",\n", + " properties: {\n", + " operation: {\n", + " type: \"string\",\n", + " enum: [\"add\", \"subtract\", \"multiply\", \"divide\"],\n", + " },\n", + " number1: { type: \"number\" },\n", + " number2: { type: \"number\" },\n", + " },\n", + " required: [\"operation\", \"number1\", \"number2\"],\n", + "};\n", + "\n", + "// Default mode is \"functionCalling\"\n", + "const modelWithStructuredOutput7 = llm7.withStructuredOutput(calculatorSchema7);\n", + "\n", + "const prompt7 = ChatPromptTemplate.fromMessages([\n", + " [\n", + " \"system\",\n", + " `You are VERY bad at math and must always use a calculator.\n", + "Respond with a JSON object containing three keys:\n", + "'operation': the type of operation to execute, either 'add', 'subtract', 'multiply' or 'divide',\n", + "'number1': the first number to operate on,\n", + "'number2': the second number to operate on.\n", + "`,\n", + " ],\n", + " [\"human\", \"Please help me!! What is 2 + 2?\"],\n", + "]);\n", + "const chain7 = prompt7.pipe(modelWithStructuredOutput7);\n", + "const result7 = await chain7.invoke({});\n", + "console.log(result7);" + ] + }, + { + "cell_type": "markdown", + "id": "ae798f49", + "metadata": {}, + "source": [ + "You can also specify 'includeRaw' to return the parsed and raw output in the result, as well as a \"name\" field to give the LLM additional context as to what you are generating." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "9a5579e4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " raw: AIMessage {\n", + " \"id\": \"chatcmpl-9rB5lkylQMLSP9CQ4SaQB9zGw1rP1\",\n", + " \"content\": \"{\\n \\\"operation\\\": \\\"add\\\",\\n \\\"number1\\\": 2,\\n \\\"number2\\\": 2\\n}\",\n", + " \"additional_kwargs\": {},\n", + " \"response_metadata\": {\n", + " \"tokenUsage\": {\n", + " \"completionTokens\": 25,\n", + " \"promptTokens\": 91,\n", + " \"totalTokens\": 116\n", + " },\n", + " \"finish_reason\": \"stop\"\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 91,\n", + " \"output_tokens\": 25,\n", + " \"total_tokens\": 116\n", + " }\n", + " },\n", + " parsed: { operation: 'add', number1: 2, number2: 2 }\n", + "}\n" + ] + } + ], + "source": [ + "const includeRawModel7 = llm7.withStructuredOutput(calculatorSchema7, {\n", + " name: \"calculator\",\n", + " includeRaw: true,\n", + " method: \"jsonMode\",\n", + "});\n", + "\n", + "const includeRawChain7 = prompt7.pipe(includeRawModel7);\n", + "const includeRawResult7 = await includeRawChain7.invoke({});\n", + "console.log(includeRawResult7);" + ] + }, + { + "cell_type": "markdown", + "id": "bf343e65", + "metadata": {}, + "source": [ + "### Disabling parallel tool calls\n", + "\n", + "If you have multiple tools bound to the model, but you'd only like for a single tool to be called at a time, you can pass the `parallel_tool_calls` call option to enable/disable this behavior.\n", + "By default, `parallel_tool_calls` is set to `true`." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "5cb759f2", + "metadata": {}, + "outputs": [], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "import { z } from \"zod\";\n", + "import { zodToJsonSchema } from \"zod-to-json-schema\";\n", + "\n", + "const llm8 = new ChatOpenAI({\n", + " temperature: 0,\n", + " model: \"gpt-4o\",\n", + "});\n", + "\n", + "// Define your tools\n", + "const calculatorSchema8 = z\n", + " .object({\n", + " operation: z.enum([\"add\", \"subtract\", \"multiply\", \"divide\"]),\n", + " number1: z.number(),\n", + " number2: z.number(),\n", + " })\n", + " .describe(\"A tool to perform basic arithmetic operations\");\n", + "const weatherSchema8 = z\n", + " .object({\n", + " city: z.string(),\n", + " })\n", + " .describe(\"A tool to get the weather in a city\");\n", + "\n", + "// Bind tools to the model\n", + "const modelWithTools8 = llm8.bindTools([\n", + " {\n", + " type: \"function\",\n", + " function: {\n", + " name: \"calculator\",\n", + " description: calculatorSchema8.description,\n", + " parameters: zodToJsonSchema(calculatorSchema8),\n", + " },\n", + " },\n", + " {\n", + " type: \"function\",\n", + " function: {\n", + " name: \"weather\",\n", + " description: weatherSchema8.description,\n", + " parameters: zodToJsonSchema(weatherSchema8),\n", + " },\n", + " },\n", + "]);\n", + "\n", + "// Invoke the model with `parallel_tool_calls` set to `true`\n", + "const response8 = await modelWithTools8.invoke(\n", + " [\"What is the weather in san francisco and what is 23716 times 27342?\"],\n", + " {\n", + " parallel_tool_calls: true,\n", + " }\n", + ");" + ] + }, + { + "cell_type": "markdown", + "id": "42a69645", + "metadata": {}, + "source": [ + "We can see it called two tools:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "95db614b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " {\n", + " name: 'weather',\n", + " args: { city: 'san francisco' },\n", + " type: 'tool_call',\n", + " id: 'call_FyxazII0M0OgKMnk2UuXDhjv'\n", + " },\n", + " {\n", + " name: 'calculator',\n", + " args: { operation: 'multiply', number1: 23716, number2: 27342 },\n", + " type: 'tool_call',\n", + " id: 'call_raQz2ABUtVpbkruA2K6vBNYd'\n", + " }\n", + "]\n" + ] + } + ], + "source": [ + "console.log(response8.tool_calls);" + ] + }, + { + "cell_type": "markdown", + "id": "6a46a7bb", + "metadata": {}, + "source": [ + "Invoke the model with `parallel_tool_calls` set to `false`" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "6a2bac84", + "metadata": {}, + "outputs": [], + "source": [ + "const response9 = await modelWithTools8.invoke(\n", + " [\"What is the weather in san francisco and what is 23716 times 27342?\"],\n", + " {\n", + " parallel_tool_calls: false,\n", + " }\n", + ");" + ] + }, + { + "cell_type": "markdown", + "id": "17d91e9f", + "metadata": {}, + "source": [ + "We can see it called one tool" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "5731d51d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " {\n", + " name: 'weather',\n", + " args: { city: 'san francisco' },\n", + " type: 'tool_call',\n", + " id: 'call_xFbViRUVYj8BFnJIVedU7GVn'\n", + " }\n", + "]\n" + ] + } + ], + "source": [ + "console.log(response9.tool_calls);" + ] + }, + { + "cell_type": "markdown", + "id": "06ffc86b", + "metadata": {}, + "source": [ + "## Custom URLs\n", + "\n", + "You can customize the base URL the SDK sends requests to by passing a `configuration` parameter like this:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "19a092b9", + "metadata": {}, + "outputs": [], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "\n", + "const llm10 = new ChatOpenAI({\n", + " temperature: 0.9,\n", + " configuration: {\n", + " baseURL: \"https://your_custom_url.com\",\n", + " },\n", + "});\n", + "\n", + "const message10 = await llm10.invoke(\"Hi there!\");" + ] + }, + { + "cell_type": "markdown", + "id": "013b6300", + "metadata": {}, + "source": [ + "You can also pass other `ClientOptions` parameters accepted by the official SDK.\n", + "\n", + "If you are hosting on Azure OpenAI, see the [dedicated page instead](/docs/integrations/chat/azure).\n", + "\n", + "## Calling fine-tuned models\n", + "\n", + "You can call fine-tuned OpenAI models by passing in your corresponding `modelName` parameter.\n", + "\n", + "This generally takes the form of `ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}`. For example:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7448f6a9", + "metadata": {}, + "outputs": [], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "\n", + "const llm11 = new ChatOpenAI({\n", + " temperature: 0.9,\n", + " model: \"ft:gpt-3.5-turbo-0613:{ORG_NAME}::{MODEL_ID}\",\n", + "});\n", + "\n", + "const message11 = await llm11.invoke(\"Hi there!\");" + ] + }, + { + "cell_type": "markdown", + "id": "a2270901", + "metadata": {}, + "source": [ + "## Generation metadata\n", + "\n", + "If you need additional information like logprobs or token usage, these will be returned directly in the `.invoke` response.\n", + "\n", + "```{=mdx}\n", + "\n", + ":::tip\n", + "Requires `@langchain/core` version >=0.1.48.\n", + ":::\n", + "\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "2b675330", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " content: [\n", + " {\n", + " token: 'Hello',\n", + " logprob: -0.0004585519,\n", + " bytes: [ 72, 101, 108, 108, 111 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: '!',\n", + " logprob: -0.000049305523,\n", + " bytes: [ 33 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' How',\n", + " logprob: -0.000029517714,\n", + " bytes: [ 32, 72, 111, 119 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' can',\n", + " logprob: -0.00073185476,\n", + " bytes: [ 32, 99, 97, 110 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' I',\n", + " logprob: -9.0883464e-7,\n", + " bytes: [ 32, 73 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' assist',\n", + " logprob: -0.104538105,\n", + " bytes: [\n", + " 32, 97, 115,\n", + " 115, 105, 115,\n", + " 116\n", + " ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' you',\n", + " logprob: -6.704273e-7,\n", + " bytes: [ 32, 121, 111, 117 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' today',\n", + " logprob: -0.000052643223,\n", + " bytes: [ 32, 116, 111, 100, 97, 121 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: '?',\n", + " logprob: -0.00001247159,\n", + " bytes: [ 63 ],\n", + " top_logprobs: []\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "\n", + "// See https://cookbook.openai.com/examples/using_logprobs for details\n", + "const llm12 = new ChatOpenAI({\n", + " logprobs: true,\n", + " // topLogprobs: 5,\n", + "});\n", + "\n", + "const responseMessage12 = await llm12.invoke(\"Hi there!\");\n", + "console.dir(responseMessage12.response_metadata.logprobs, { depth: null });" + ] + }, + { + "cell_type": "markdown", + "id": "899c364f", + "metadata": {}, + "source": [ + "### With callbacks\n", + "\n", + "You can also use the callbacks system:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "01e74121", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " content: [\n", + " {\n", + " token: 'Hello',\n", + " logprob: -0.0005182436,\n", + " bytes: [ 72, 101, 108, 108, 111 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: '!',\n", + " logprob: -0.000040246043,\n", + " bytes: [ 33 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' How',\n", + " logprob: -0.000035716304,\n", + " bytes: [ 32, 72, 111, 119 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' can',\n", + " logprob: -0.0006764544,\n", + " bytes: [ 32, 99, 97, 110 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' I',\n", + " logprob: -0.0000010280384,\n", + " bytes: [ 32, 73 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' assist',\n", + " logprob: -0.12827769,\n", + " bytes: [\n", + " 32, 97, 115,\n", + " 115, 105, 115,\n", + " 116\n", + " ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' you',\n", + " logprob: -4.3202e-7,\n", + " bytes: [ 32, 121, 111, 117 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' today',\n", + " logprob: -0.000059914648,\n", + " bytes: [ 32, 116, 111, 100, 97, 121 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: '?',\n", + " logprob: -0.000012352386,\n", + " bytes: [ 63 ],\n", + " top_logprobs: []\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "\n", + "// See https://cookbook.openai.com/examples/using_logprobs for details\n", + "const llm13 = new ChatOpenAI({\n", + " logprobs: true,\n", + " // topLogprobs: 5,\n", + "});\n", + "\n", + "const result13 = await llm13.invoke(\"Hi there!\", {\n", + " callbacks: [\n", + " {\n", + " handleLLMEnd(output) {\n", + " console.dir(output.generations[0][0].generationInfo.logprobs, { depth: null });\n", + " },\n", + " },\n", + " ],\n", + "});" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "7f9f01aa", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " content: [\n", + " {\n", + " token: 'Hello',\n", + " logprob: -0.0005182436,\n", + " bytes: [ 72, 101, 108, 108, 111 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: '!',\n", + " logprob: -0.000040246043,\n", + " bytes: [ 33 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' How',\n", + " logprob: -0.000035716304,\n", + " bytes: [ 32, 72, 111, 119 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' can',\n", + " logprob: -0.0006764544,\n", + " bytes: [ 32, 99, 97, 110 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' I',\n", + " logprob: -0.0000010280384,\n", + " bytes: [ 32, 73 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' assist',\n", + " logprob: -0.12827769,\n", + " bytes: [\n", + " 32, 97, 115,\n", + " 115, 105, 115,\n", + " 116\n", + " ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' you',\n", + " logprob: -4.3202e-7,\n", + " bytes: [ 32, 121, 111, 117 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: ' today',\n", + " logprob: -0.000059914648,\n", + " bytes: [ 32, 116, 111, 100, 97, 121 ],\n", + " top_logprobs: []\n", + " },\n", + " {\n", + " token: '?',\n", + " logprob: -0.000012352386,\n", + " bytes: [ 63 ],\n", + " top_logprobs: []\n", + " }\n", + " ]\n", + "}\n" + ] + } + ], + "source": [ + "console.dir(result13.response_metadata.logprobs, { depth: null });" + ] + }, + { + "cell_type": "markdown", + "id": "5194627d", + "metadata": {}, + "source": [ + "## Streaming tokens\n", + "\n", + "OpenAI supports streaming token counts via an opt-in call option. This can be set by passing `{ stream_options: { include_usage: true } }`.\n", + "Setting this call option will cause the model to return an additional chunk at the end of the stream, containing the token usage." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "f6efaebb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{ input_tokens: 13, output_tokens: 33, total_tokens: 46 }\n" + ] + } + ], + "source": [ + "import type { AIMessageChunk } from \"@langchain/core/messages\";\n", + "import { ChatOpenAI } from \"@langchain/openai\";\n", + "import { concat } from \"@langchain/core/utils/stream\";\n", + "\n", + "// Instantiate the model\n", + "const llm14 = new ChatOpenAI();\n", + "\n", + "const response14 = await llm14.stream(\"Hello, how are you?\", {\n", + " // Pass the stream options\n", + " stream_options: {\n", + " include_usage: true,\n", + " },\n", + "});\n", + "\n", + "// Iterate over the response, only saving the last chunk\n", + "let finalResult14: AIMessageChunk | undefined;\n", + "for await (const chunk14 of response14) {\n", + " finalResult14 = !finalResult14 ? chunk14 : concat(finalResult14, chunk14);\n", + "}\n", + "\n", + "console.log(finalResult14?.usage_metadata);" + ] + }, + { + "cell_type": "markdown", + "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3", + "metadata": {}, + "source": [ + "## API reference\n", + "\n", + "For detailed documentation of all ChatOpenAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "TypeScript", + "language": "typescript", + "name": "tslab" + }, + "language_info": { + "codemirror_mode": { + "mode": "typescript", + "name": "javascript", + "typescript": true + }, + "file_extension": ".ts", + "mimetype": "text/typescript", + "name": "typescript", + "version": "3.7.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/core_docs/docs/integrations/chat/openai.mdx b/docs/core_docs/docs/integrations/chat/openai.mdx deleted file mode 100644 index 4ef51a28c603..000000000000 --- a/docs/core_docs/docs/integrations/chat/openai.mdx +++ /dev/null @@ -1,153 +0,0 @@ ---- -sidebar_label: OpenAI ---- - -import CodeBlock from "@theme/CodeBlock"; - -# ChatOpenAI - -You can use OpenAI's chat models as follows: - -import OpenAI from "@examples/models/chat/integration_openai.ts"; - -import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; - - - -```bash npm2yarn -npm install @langchain/openai -``` - -import UnifiedModelParamsTooltip from "@mdx_components/unified_model_params_tooltip.mdx"; - - - -{OpenAI} - -If you're part of an organization, you can set `process.env.OPENAI_ORGANIZATION` with your OpenAI organization id, or pass it in as `organization` when -initializing the model. - -## Multimodal messages - -:::info -This feature is currently in preview. The message schema may change in future releases. -::: - -OpenAI supports interleaving images with text in input messages with their `gpt-4-vision-preview`. Here's an example of how this looks: - -import OpenAIVision from "@examples/models/chat/integration_openai_vision.ts"; - -{OpenAIVision} - -## Tool calling - -:::info -This feature is currently only available for `gpt-3.5-turbo-1106` and `gpt-4-1106-preview` models. -::: - -More recent OpenAI chat models support calling multiple functions to get all required data to answer a question. -Here's an example how a conversation turn with this functionality might look: - -import OpenAITools from "@examples/models/chat/integration_openai_tool_calls.ts"; - -{OpenAITools} - -### `.withStructuredOutput({ ... })` - -:::info -The `.withStructuredOutput` method is in beta. It is actively being worked on, so the API may change. -::: - -You can also use the `.withStructuredOutput({ ... })` method to coerce `ChatOpenAI` into returning a structured output. - -The method allows for passing in either a Zod object, or a valid JSON schema (like what is returned from [`zodToJsonSchema`](https://www.npmjs.com/package/zod-to-json-schema)). - -Using the method is simple. Just define your LLM and call `.withStructuredOutput({ ... })` on it, passing the desired schema. - -Here is an example using a Zod schema and the `functionCalling` mode (default mode): - -import WSAZodExample from "@examples/models/chat/integration_openai_wsa_zod.ts"; - -{WSAZodExample} - -Additionally, you can pass in an OpenAI function definition or JSON schema directly: - -:::info -If using `jsonMode` as the `method` you must include context in your prompt about the structured output you want. This _must_ include the keyword: `JSON`. -::: - -import WSAJSONSchemaExample from "@examples/models/chat/integration_openai_wsa_json_schema.ts"; - -{WSAJSONSchemaExample} - -## Custom URLs - -You can customize the base URL the SDK sends requests to by passing a `configuration` parameter like this: - -import OpenAICustomBase from "@examples/models/chat/integration_openai_custom_base.ts"; - -{OpenAICustomBase} - -You can also pass other `ClientOptions` parameters accepted by the official SDK. - -If you are hosting on Azure OpenAI, see the [dedicated page instead](/docs/integrations/chat/azure). - -## Calling fine-tuned models - -You can call fine-tuned OpenAI models by passing in your corresponding `modelName` parameter. - -This generally takes the form of `ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}`. For example: - -import OpenAIFineTuned from "@examples/models/chat/integration_openai_fine_tune.ts"; - -{OpenAIFineTuned} - -## Generation metadata - -If you need additional information like logprobs or token usage, these will be returned directly in the `.invoke` response. - -:::tip -Requires `@langchain/core` version >=0.1.48. -::: - -import OpenAIInvokeInfo from "@examples/models/chat/integration_openai_invoke_info.ts"; -import OpenAIGenerationInfo from "@examples/models/chat/integration_openai_generation_info.ts"; -import OpenAICallbacks from "@examples/models/chat/integration_openai_callbacks.ts"; - -{OpenAIInvokeInfo} - -### With callbacks - -You can also use the callbacks system: - -{OpenAICallbacks} - -### With `.generate()` - -{OpenAIGenerationInfo} - -### Streaming tokens - -OpenAI supports streaming token counts via an opt-in call option. This can be set by passing `{ stream_options: { include_usage: true } }`. -Setting this call option will cause the model to return an additional chunk at the end of the stream, containing the token usage. - -import OpenAIStreamTokens from "@examples/models/chat/integration_openai_stream_tokens.ts"; - -{OpenAIStreamTokens} - -:::tip -See the LangSmith trace [here](https://smith.langchain.com/public/66bf7377-cc69-4676-91b6-25929a05e8b7/r) -::: - -### Disabling parallel tool calls - -If you have multiple tools bound to the model, but you'd only like for a single tool to be called at a time, you can pass the `parallel_tool_calls` call option to enable/disable this behavior. -By default, `parallel_tool_calls` is set to `true`. - -import OpenAIParallelToolCallsTokens from "@examples/models/chat/integration_openai_parallel_tool_calls.ts"; - -{OpenAIParallelToolCallsTokens} - -:::tip -See the LangSmith trace for the first invocation [here](https://smith.langchain.com/public/68f2ff13-6331-47d8-a8c0-d1745788e84e/r) and the second invocation [here](https://smith.langchain.com/public/6c2fff29-9470-486a-8715-805fda631024/r) -:::