From 58bd811d008d541d018c83b41f691a5f29655c15 Mon Sep 17 00:00:00 2001 From: bracesproul Date: Thu, 1 Aug 2024 11:41:39 -0700 Subject: [PATCH] docs[minor]: Update Google GenAI doc --- .../chat/google_generativeai.ipynb | 618 ++++++++++++++++++ .../integrations/chat/google_generativeai.mdx | 101 --- 2 files changed, 618 insertions(+), 101 deletions(-) create mode 100644 docs/core_docs/docs/integrations/chat/google_generativeai.ipynb delete mode 100644 docs/core_docs/docs/integrations/chat/google_generativeai.mdx diff --git a/docs/core_docs/docs/integrations/chat/google_generativeai.ipynb b/docs/core_docs/docs/integrations/chat/google_generativeai.ipynb new file mode 100644 index 000000000000..af47303b2917 --- /dev/null +++ b/docs/core_docs/docs/integrations/chat/google_generativeai.ipynb @@ -0,0 +1,618 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "46f7ac07", + "metadata": {}, + "source": [ + "---\n", + "sidebar_label: Google GenAI\n", + "keywords: [gemini, gemini-pro, ChatGoogleGenerativeAI]\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "e49f1e0d", + "metadata": {}, + "source": [ + "# ChatGoogleGenerativeAI\n", + "\n", + "This will help you getting started with `ChatGoogleGenerativeAI` [chat models](/docs/concepts/#chat-models). For detailed documentation of all `ChatGoogleGenerativeAI` features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_google_genai.ChatGoogleGenerativeAI.html).\n", + "\n", + "## Overview\n", + "### Integration details\n", + "\n", + "| Class | Package | Local | Serializable | [PY support](https://python.langchain.com/docs/integrations/chat/google_generative_ai) | Package downloads | Package latest |\n", + "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", + "| [ChatGoogleGenerativeAI](https://api.js.langchain.com/classes/langchain_google_genai.ChatGoogleGenerativeAI.html) | [@langchain/google-genai](https://api.js.langchain.com/modules/langchain_google_genai.html) | ❌ | ✅ | ✅ | ![NPM - Downloads](https://img.shields.io/npm/dm/@langchain/google-genai?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/@langchain/google-genai?style=flat-square&label=%20&) |\n", + "\n", + "### Model features\n", + "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n", + "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n", + "| ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | \n", + "\n", + "## Setup\n", + "\n", + "You can access Google's `gemini` and `gemini-vision` models, as well as other\n", + "generative models in LangChain through `ChatGoogleGenerativeAI` class in the\n", + "`@langchain/google-genai` integration package.\n", + "\n", + "```{=mdx}\n", + "\n", + ":::tip\n", + "You can also access Google's `gemini` family of models via the LangChain VertexAI and VertexAI-web integrations.\n", + "\n", + "Click [here](/docs/integrations/chat/google_vertex_ai) to read the docs.\n", + ":::\n", + "\n", + "```\n", + "\n", + "### Credentials\n", + "\n", + "Get an API key here: [ai.google.dev/tutorials/setup](https://ai.google.dev/tutorials/setup)\n", + "\n", + "Then set the `GOOGLE_API_KEY` environment variable:\n", + "\n", + "```bash\n", + "export GOOGLE_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n", + "\n", + "```bash\n", + "# export LANGCHAIN_TRACING_V2=\"true\"\n", + "# export LANGCHAIN_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "### Installation\n", + "\n", + "The LangChain ChatGoogleGenerativeAI integration lives in the `@langchain/google-genai` package:\n", + "\n", + "```{=mdx}\n", + "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", + "import Npm2Yarn from \"@theme/Npm2Yarn\";\n", + "\n", + "\n", + "\n", + "\n", + " @langchain/google-genai\n", + "\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "a38cde65-254d-4219-a441-068766c0d4b5", + "metadata": {}, + "source": [ + "## Instantiation\n", + "\n", + "Now we can instantiate our model object and generate chat completions:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae", + "metadata": {}, + "outputs": [], + "source": [ + "import { ChatGoogleGenerativeAI } from \"@langchain/google-genai\"\n", + "\n", + "const llm = new ChatGoogleGenerativeAI({\n", + " model: \"gemini-1.5-pro\",\n", + " temperature: 0,\n", + " maxRetries: 2,\n", + " // other params...\n", + "})" + ] + }, + { + "cell_type": "markdown", + "id": "2b4f3e15", + "metadata": {}, + "source": [ + "## Invocation" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "62e0dbc3", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"content\": \"J'adore programmer. \\n\",\n", + " \"additional_kwargs\": {\n", + " \"finishReason\": \"STOP\",\n", + " \"index\": 0,\n", + " \"safetyRatings\": [\n", + " {\n", + " \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HARASSMENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " }\n", + " ]\n", + " },\n", + " \"response_metadata\": {\n", + " \"finishReason\": \"STOP\",\n", + " \"index\": 0,\n", + " \"safetyRatings\": [\n", + " {\n", + " \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HARASSMENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " }\n", + " ]\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 21,\n", + " \"output_tokens\": 5,\n", + " \"total_tokens\": 26\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "const aiMsg = await llm.invoke([\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n", + " ],\n", + " [\"human\", \"I love programming.\"],\n", + "])\n", + "aiMsg" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "J'adore programmer. \n", + "\n" + ] + } + ], + "source": [ + "console.log(aiMsg.content)" + ] + }, + { + "cell_type": "markdown", + "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8", + "metadata": {}, + "source": [ + "## Chaining\n", + "\n", + "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"content\": \"Ich liebe das Programmieren. \\n\",\n", + " \"additional_kwargs\": {\n", + " \"finishReason\": \"STOP\",\n", + " \"index\": 0,\n", + " \"safetyRatings\": [\n", + " {\n", + " \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HARASSMENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " }\n", + " ]\n", + " },\n", + " \"response_metadata\": {\n", + " \"finishReason\": \"STOP\",\n", + " \"index\": 0,\n", + " \"safetyRatings\": [\n", + " {\n", + " \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HARASSMENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " }\n", + " ]\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 16,\n", + " \"output_tokens\": 7,\n", + " \"total_tokens\": 23\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatPromptTemplate } from \"@langchain/core/prompts\"\n", + "\n", + "const prompt = ChatPromptTemplate.fromMessages(\n", + " [\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + " ]\n", + ")\n", + "\n", + "const chain = prompt.pipe(llm);\n", + "await chain.invoke(\n", + " {\n", + " input_language: \"English\",\n", + " output_language: \"German\",\n", + " input: \"I love programming.\",\n", + " }\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "d1ee55bc-ffc8-4cfa-801c-993953a08cfd", + "metadata": {}, + "source": [ + "## Tool calling\n", + "\n", + "```{=mdx}\n", + "\n", + ":::caution\n", + "The Google GenerativeAI API does not allow tool schemas to contain an object with unknown properties.\n", + "\n", + "For example, the following Zod schema will throw an error:\n", + "\n", + "```typescript\n", + "const schema = z.object({\n", + " properties: z.record(z.unknown()), // Not allowed\n", + "});\n", + "```\n", + "\n", + "or\n", + "\n", + "```typescript\n", + "const schema = z.record(z.unknown()); // Not allowed\n", + "```\n", + "\n", + "Instead, you should explicitly define the properties of the object field.\n", + ":::\n", + "\n", + "```\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "d6805c40", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " {\n", + " name: 'browser_tool',\n", + " args: {\n", + " url: 'https://www.weather.com',\n", + " query: 'weather tonight in new york'\n", + " },\n", + " type: 'tool_call'\n", + " }\n", + "]\n" + ] + } + ], + "source": [ + "import { tool } from \"@langchain/core/tools\";\n", + "import { ChatGoogleGenerativeAI } from \"@langchain/google-genai\";\n", + "import { z } from \"zod\";\n", + "\n", + "// Define your tool\n", + "const fakeBrowserTool = tool((_) => {\n", + " return \"The search result is xyz...\"\n", + "}, {\n", + " name: \"browser_tool\",\n", + " description: \"Useful for when you need to find something on the web or summarize a webpage.\",\n", + " schema: z.object({\n", + " url: z.string().describe(\"The URL of the webpage to search.\"),\n", + " query: z.string().optional().describe(\"An optional search query to use.\"),\n", + " }),\n", + "})\n", + "\n", + "const llmWithTool = new ChatGoogleGenerativeAI({\n", + " model: \"gemini-pro\",\n", + "}).bindTools([fakeBrowserTool]) // Bind your tools to the model\n", + "\n", + "const toolRes = await llmWithTool.invoke([\n", + " [\n", + " \"human\",\n", + " \"Search the web and tell me what the weather will be like tonight in new york. use a popular weather website\",\n", + " ],\n", + "]);\n", + "\n", + "console.log(toolRes.tool_calls);" + ] + }, + { + "cell_type": "markdown", + "id": "83061805", + "metadata": {}, + "source": [ + "### `.withStructuredOutput`" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "ef24448c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " url: 'https://www.accuweather.com/en/us/new-york-ny/10007/current-weather/349333',\n", + " query: 'weather tonight'\n", + "}\n" + ] + } + ], + "source": [ + "import { ChatGoogleGenerativeAI } from \"@langchain/google-genai\";\n", + "import { z } from \"zod\";\n", + "\n", + "// Define your model\n", + "const llmForWSO = new ChatGoogleGenerativeAI({\n", + " model: \"gemini-pro\",\n", + "});\n", + "\n", + "const browserSchema = z.object({\n", + " url: z.string().describe(\"The URL of the webpage to search.\"),\n", + " query: z.string().optional().describe(\"An optional search query to use.\"),\n", + "});\n", + "\n", + "const llmWithStructuredOutput = llmForWSO.withStructuredOutput(browserSchema, {\n", + " name: \"browser_tool\"\n", + "})\n", + "\n", + "const structuredOutputRes = await llmWithStructuredOutput.invoke([\n", + " [\n", + " \"human\",\n", + " \"Search the web and tell me what the weather will be like tonight in new york. use a popular weather website\",\n", + " ],\n", + "]);\n", + "\n", + "console.log(structuredOutputRes);" + ] + }, + { + "cell_type": "markdown", + "id": "3987f0cb", + "metadata": {}, + "source": [ + "## Multimodal support\n", + "\n", + "To provide an image, pass a human message with a `content` field set to an array of content objects. Each content object\n", + "where each dict contains either an image value (type of image_url) or a text (type of text) value. The value of image_url must be a base64\n", + "encoded image (e.g., data:image/png;base64,abcd124):" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "0b60fc5d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "AIMessage {\n", + " \"content\": \"The image shows a hot dog in a bun, isolated against a white background. The hot dog is grilled and has a slightly crispy texture. The bun is soft and fluffy, and it appears to be lightly toasted. The hot dog is positioned horizontally, with the bun covering most of the sausage. The image captures the classic American snack food, highlighting its simplicity and appeal.\",\n", + " \"additional_kwargs\": {\n", + " \"finishReason\": \"STOP\",\n", + " \"index\": 0,\n", + " \"safetyRatings\": [\n", + " {\n", + " \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HARASSMENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " }\n", + " ]\n", + " },\n", + " \"response_metadata\": {\n", + " \"finishReason\": \"STOP\",\n", + " \"index\": 0,\n", + " \"safetyRatings\": [\n", + " {\n", + " \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_HARASSMENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " },\n", + " {\n", + " \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n", + " \"probability\": \"NEGLIGIBLE\"\n", + " }\n", + " ]\n", + " },\n", + " \"tool_calls\": [],\n", + " \"invalid_tool_calls\": [],\n", + " \"usage_metadata\": {\n", + " \"input_tokens\": 264,\n", + " \"output_tokens\": 74,\n", + " \"total_tokens\": 338\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import fs from \"fs\";\n", + "import { ChatGoogleGenerativeAI } from \"@langchain/google-genai\";\n", + "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", + "\n", + "// Multi-modal\n", + "const llmWithVisionModel = new ChatGoogleGenerativeAI({\n", + " model: \"gemini-1.5-flash\",\n", + " maxOutputTokens: 2048,\n", + " maxRetries: 1,\n", + "});\n", + "const image = fs.readFileSync(\"../../../../../examples/hotdog.jpg\").toString(\"base64\");\n", + "const visionPrompt = ChatPromptTemplate.fromMessages([\n", + " [\n", + " \"human\", \n", + " [\n", + " {\n", + " type: \"text\",\n", + " text: \"Describe the following image.\",\n", + " },\n", + " {\n", + " type: \"image_url\",\n", + " image_url: \"data:image/png;base64,{image}\",\n", + " },\n", + " ]\n", + " ]\n", + "])\n", + "\n", + "const visionRes = await visionPrompt.pipe(llmWithVisionModel).invoke({\n", + " image,\n", + "});\n", + "\n", + "console.log(visionRes);" + ] + }, + { + "cell_type": "markdown", + "id": "0c6a950f", + "metadata": {}, + "source": [ + "## Gemini Prompting FAQs\n", + "\n", + "As of the time this doc was written (2023/12/12), Gemini has some restrictions on the types and structure of prompts it accepts. Specifically:\n", + "\n", + "1. When providing multimodal (image) inputs, you are restricted to at most 1 message of \"human\" (user) type. You cannot pass multiple messages (though the single human message may have multiple content entries)\n", + "2. System messages are not natively supported, and will be merged with the first human message if present.\n", + "3. For regular chat conversations, messages must follow the human/ai/human/ai alternating pattern. You may not provide 2 AI or human messages in sequence.\n", + "4. Message may be blocked if they violate the safety checks of the LLM. In this case, the model will return an empty response.\n" + ] + }, + { + "cell_type": "markdown", + "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3", + "metadata": {}, + "source": [ + "## API reference\n", + "\n", + "For detailed documentation of all ChatGoogleGenerativeAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_google_genai.ChatGoogleGenerativeAI.html" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "TypeScript", + "language": "typescript", + "name": "tslab" + }, + "language_info": { + "codemirror_mode": { + "mode": "typescript", + "name": "javascript", + "typescript": true + }, + "file_extension": ".ts", + "mimetype": "text/typescript", + "name": "typescript", + "version": "3.7.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/core_docs/docs/integrations/chat/google_generativeai.mdx b/docs/core_docs/docs/integrations/chat/google_generativeai.mdx deleted file mode 100644 index a97b768eae1d..000000000000 --- a/docs/core_docs/docs/integrations/chat/google_generativeai.mdx +++ /dev/null @@ -1,101 +0,0 @@ ---- -sidebar_label: Google GenAI -keywords: [gemini, gemini-pro, ChatGoogleGenerativeAI] ---- - -import CodeBlock from "@theme/CodeBlock"; - -# ChatGoogleGenerativeAI - -You can access Google's `gemini` and `gemini-vision` models, as well as other -generative models in LangChain through `ChatGoogleGenerativeAI` class in the -`@langchain/google-genai` integration package. - -:::tip -You can also access Google's `gemini` family of models via the LangChain VertexAI and VertexAI-web integrations. - -Click [here](/docs/integrations/chat/google_vertex_ai) to read the docs. -::: - -Get an API key here: https://ai.google.dev/tutorials/setup - -You'll first need to install the `@langchain/google-genai` package: - -import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; - - - -```bash npm2yarn -npm install @langchain/google-genai -``` - -## Usage - -import UnifiedModelParamsTooltip from "@mdx_components/unified_model_params_tooltip.mdx"; - - - -import GoogleGenerativeAI from "@examples/models/chat/googlegenerativeai.ts"; - -{GoogleGenerativeAI} - -## Tool calling - -:::caution -The Google GenerativeAI package as of version `0.0.23` does not allow tool schemas to contain an object with unknown properties. -The Google VertexAI package (as of version `0.0.20`) does support this pattern. -[Click here for the Google VertexAI package documentation](/docs/integrations/chat/google_vertex_ai). - -For example, the following Zod schema will throw an error: - -```typescript -const schema = z.object({ - properties: z.record(z.unknown()), // Not allowed -}); -``` - -or - -```typescript -const schema = z.record(z.unknown()); // Not allowed -``` - -Instead, you should explicitly define the properties of the object field, or use the Google VertexAI package. -::: - -import GoogleGenerativeAIToolCalling from "@examples/models/chat/googlegenerativeai_tools.ts"; - -{GoogleGenerativeAIToolCalling} - -:::tip -See the above run's LangSmith trace [here](https://smith.langchain.com/public/31faf31b-dbd0-436c-a425-b9eb1bccf8b7/r) -::: - -## `.withStructuredOutput` - -import GoogleGenerativeAIWSO from "@examples/models/chat/googlegenerativeai_wso.ts"; - -{GoogleGenerativeAIWSO} - -:::tip -See the above run's LangSmith trace [here](https://smith.langchain.com/public/4506314e-21ea-43a9-9718-22cad0bbbb38/r) -::: - -## Multimodal support - -To provide an image, pass a human message with a `content` field set to an array of content objects. Each content object -where each dict contains either an image value (type of image_url) or a text (type of text) value. The value of image_url must be a base64 -encoded image (e.g., data:image/png;base64,abcd124): - -import GoogleGenerativeAIMultimodal from "@examples/models/chat/googlegenerativeai_multimodal.ts"; - -{GoogleGenerativeAIMultimodal} - -## Gemini Prompting FAQs - -As of the time this doc was written (2023/12/12), Gemini has some restrictions on the types and structure of prompts it accepts. Specifically: - -1. When providing multimodal (image) inputs, you are restricted to at most 1 message of "human" (user) type. You cannot pass multiple messages (though the single human message may have multiple content entries) -2. System messages are not natively supported, and will be merged with the first human message if present. -3. For regular chat conversations, messages must follow the human/ai/human/ai alternating pattern. You may not provide 2 AI or human messages in sequence. -4. Message may be blocked if they violate the safety checks of the LLM. In this case, the model will return an empty response.