Feature/azure face (#20)

* Create an Azure face tool and unit tests * Update test file and create docs * Update cell order * Lint files * Correct spelling and types * Add suggested review changes
Sheepsta300 · Oct 19, 2024 · 5ebf320 · 5ebf320
1 parent bc1fd7b
commit 5ebf320
Show file tree

Hide file tree

Showing 7 changed files with 704 additions and 0 deletions.
diff --git a/docs/docs/example_data/face.jpg b/docs/docs/example_data/face.jpg
diff --git a/docs/docs/integrations/tools/azure_ai_face.ipynb b/docs/docs/integrations/tools/azure_ai_face.ipynb
@@ -0,0 +1,389 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# `AzureAIFaceAnalysisTool`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Overview"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Integration details"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    ">The `AzureAIFaceAnalysisTool` acts as a wrapper around the Azure AI Face Service/API. The Tool will detect faces is in an image \n",
+    ">and can supply the position of faces, as well as important features like eyes, pupils, head pose etc."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Tool features"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This integration allows for the detection of faces using Azure's Face API. It supports various different models, but defaults to the latest. Furthermore, the user can configure which information they would like to be returned such as position of the faces and image quality of the faces."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "This section provides details about how the Azure AI Face integration works, including setup."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "from langchain import hub"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We will use a prompt to instruct the model. LangChain prompts can be configured, but for simplicity, we will use a premade prompt from LangSmith. This requires an API key, which can be set up here after registration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "LANGSMITH_KEY = os.environ[\"LANGSMITH_KEY\"]\n",
+    "prompt = hub.pull(\"hwchase17/structured-chat-agent\", api_key=LANGSMITH_KEY)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we can use the `AzureAIFaceAnalysisTool` combined with a model, using `create_structured_chat_agent`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import AgentExecutor, create_structured_chat_agent\n",
+    "from langchain_community.tools.azure_ai_services.face_analysis import (\n",
+    "    AzureAIFaceAnalysisTool,\n",
+    ")\n",
+    "from langchain_openai import AzureChatOpenAI"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Credentials"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Credentials can be set by being passed as parameters and should be stored locally as environment variables."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "face_endpoint = os.environ[\"AZURE_AI_FACE_ENDPOINT\"]\n",
+    "face_key = os.environ[\"AZURE_AI_FACE_KEY\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Instantiation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Credentials can be passed directly, but they can also be retrieved automatically by the constructor if environment variables named `AZURE_AI_FACE_ENDPOINT` and `AZURE_AI_FACE_KEY` are set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "face = AzureAIFaceAnalysisTool(\n",
+    "    azure_ai_face_key=face_key,\n",
+    "    azure_ai_face_endpoint=face_endpoint,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tools = [face]\n",
+    "\n",
+    "model = AzureChatOpenAI(\n",
+    "    openai_api_version=os.environ[\"OPENAI_API_VERSION\"],\n",
+    "    azure_deployment=os.environ[\"COMPLETIONS_MODEL\"],\n",
+    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
+    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Creating an `AgentExecutor` chain allows a model to use tools to assist in it's response."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = create_structured_chat_agent(model, tools, prompt)\n",
+    "\n",
+    "agent_executor = AgentExecutor(\n",
+    "    agent=agent, tools=tools, verbose=True, handle_parsing_errors=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Invocation\n",
+    "\n",
+    "Get either a local file path or remote image url as input"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "image_path = \"../../example_data/face.jpg\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input = image_path"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### [Invoke directly with args](/docs/concepts/#invoke-with-just-the-arguments)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Firstly, the tool can be used by directly passing input as an argument."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"FACE: 1\\nfaceId: 9b31b927-24a5-405c-95a9-d76f457e7160\\nfaceRectangle: {'top': 760, 'left': 1159, 'width': 853, 'height': 1337}\\nfaceLandmarks: {'pupilLeft': {'x': 1293.2, 'y': 1319.9}, 'pupilRight': {'x': 1671.1, 'y': 1306.7}, 'noseTip': {'x': 1383.7, 'y': 1593.6}, 'mouthLeft': {'x': 1333.1, 'y': 1821.3}, 'mouthRight': {'x': 1614.4, 'y': 1823.6}, 'eyebrowLeftOuter': {'x': 1167.6, 'y': 1199.4}, 'eyebrowLeftInner': {'x': 1348.4, 'y': 1214.3}, 'eyeLeftOuter': {'x': 1237.2, 'y': 1321.2}, 'eyeLeftTop': {'x': 1294.7, 'y': 1286.6}, 'eyeLeftBottom': {'x': 1289.7, 'y': 1347.3}, 'eyeLeftInner': {'x': 1351.2, 'y': 1324.3}, 'eyebrowRightInner': {'x': 1522.6, 'y': 1201.8}, 'eyebrowRightOuter': {'x': 1826.9, 'y': 1196.0}, 'eyeRightInner': {'x': 1605.4, 'y': 1317.3}, 'eyeRightTop': {'x': 1662.8, 'y': 1268.2}, 'eyeRightBottom': {'x': 1672.7, 'y': 1335.6}, 'eyeRightOuter': {'x': 1743.3, 'y': 1305.7}, 'noseRootLeft': {'x': 1396.7, 'y': 1342.7}, 'noseRootRight': {'x': 1499.7, 'y': 1337.8}, 'noseLeftAlarTop': {'x': 1350.5, 'y': 1516.6}, 'noseRightAlarTop': {'x': 1522.3, 'y': 1518.2}, 'noseLeftAlarOutTip': {'x': 1321.3, 'y': 1611.3}, 'noseRightAlarOutTip': {'x': 1561.3, 'y': 1623.0}, 'upperLipTop': {'x': 1426.4, 'y': 1773.3}, 'upperLipBottom': {'x': 1428.9, 'y': 1811.0}, 'underLipTop': {'x': 1428.9, 'y': 1836.5}, 'underLipBottom': {'x': 1430.2, 'y': 1898.5}}\\nfaceAttributes: {'headPose': {'pitch': -12.9, 'roll': 1.3, 'yaw': -22.0}, 'qualityForRecognition': 'medium'}\\nrecognitionModel: recognition_04\\n\""
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "face.invoke({\"query\": input})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### [Invoke with ToolCall](/docs/concepts/#invoke-with-toolcall)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By using `.invoke`, the model can be told what to do and assess if using the tools it was given would assist in it's response. Here we will use a remote url as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "image_path = \"https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Guy_Sebastian_2014.jpg/330px-Guy_Sebastian_2014.jpg\"\n",
+    "input = image_path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m{\n",
+      "  \"action\": \"azure_ai_face_analysis\",\n",
+      "  \"action_input\": {\n",
+      "    \"query\": \"https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Guy_Sebastian_2014.jpg/330px-Guy_Sebastian_2014.jpg\"\n",
+      "  }\n",
+      "}\u001b[0m\u001b[36;1m\u001b[1;3mFACE: 1\n",
+      "faceId: ab842360-a7fc-4570-8082-8f0928064141\n",
+      "faceRectangle: {'top': 69, 'left': 47, 'width': 209, 'height': 266}\n",
+      "faceLandmarks: {'pupilLeft': {'x': 101.5, 'y': 192.0}, 'pupilRight': {'x': 194.8, 'y': 183.8}, 'noseTip': {'x': 148.4, 'y': 237.3}, 'mouthLeft': {'x': 109.1, 'y': 265.7}, 'mouthRight': {'x': 199.8, 'y': 257.2}, 'eyebrowLeftOuter': {'x': 70.6, 'y': 174.6}, 'eyebrowLeftInner': {'x': 120.7, 'y': 175.5}, 'eyeLeftOuter': {'x': 85.3, 'y': 192.7}, 'eyeLeftTop': {'x': 102.7, 'y': 188.3}, 'eyeLeftBottom': {'x': 100.1, 'y': 195.2}, 'eyeLeftInner': {'x': 118.0, 'y': 191.7}, 'eyebrowRightInner': {'x': 168.4, 'y': 172.0}, 'eyebrowRightOuter': {'x': 225.9, 'y': 164.1}, 'eyeRightInner': {'x': 178.8, 'y': 186.5}, 'eyeRightTop': {'x': 193.0, 'y': 179.8}, 'eyeRightBottom': {'x': 195.7, 'y': 187.1}, 'eyeRightOuter': {'x': 211.6, 'y': 181.8}, 'noseRootLeft': {'x': 132.9, 'y': 196.0}, 'noseRootRight': {'x': 159.7, 'y': 194.0}, 'noseLeftAlarTop': {'x': 128.6, 'y': 222.3}, 'noseRightAlarTop': {'x': 168.8, 'y': 219.0}, 'noseLeftAlarOutTip': {'x': 120.7, 'y': 236.2}, 'noseRightAlarOutTip': {'x': 179.7, 'y': 231.9}, 'upperLipTop': {'x': 154.7, 'y': 261.4}, 'upperLipBottom': {'x': 153.4, 'y': 267.4}, 'underLipTop': {'x': 154.5, 'y': 281.0}, 'underLipBottom': {'x': 155.5, 'y': 291.9}}\n",
+      "faceAttributes: {'headPose': {'pitch': -12.3, 'roll': -5.0, 'yaw': -1.5}, 'qualityForRecognition': 'high'}\n",
+      "recognitionModel: recognition_04\n",
+      "\u001b[0m\u001b[32;1m\u001b[1;3mAction:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"Final Answer\",\n",
+      "  \"action_input\": \"The image contains one face. The faceId is ab842360-a7fc-4570-8082-8f0928064141, and the face is located at the rectangle with top: 69, left: 47, width: 209, and height: 266. The face has various landmarks and attributes, including a high quality for recognition.\"\n",
+      "}\n",
+      "```\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "{'input': 'Can you check the following image for faces : https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Guy_Sebastian_2014.jpg/330px-Guy_Sebastian_2014.jpg',\n",
+       " 'output': 'The image contains one face. The faceId is ab842360-a7fc-4570-8082-8f0928064141, and the face is located at the rectangle with top: 69, left: 47, width: 209, and height: 266. The face has various landmarks and attributes, including a high quality for recognition.'}"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_executor.invoke(\n",
+    "    {\"input\": f\"Can you check the following image for faces : {input}\"}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Chaining"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When creating an `AgentExecutor` as described earlier, an execution chain is formed. The sequence of events will be printed using the given prompt, and actions will occur in a chain-like manner."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = create_structured_chat_agent(model, tools, prompt)\n",
+    "\n",
+    "agent_executor = AgentExecutor(\n",
+    "    agent=agent, tools=tools, verbose=True, handle_parsing_errors=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## API reference"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[Azure AI Face python SDK](https://learn.microsoft.com/python/api/overview/azure/ai-vision-face-readme?view=azure-python-preview) | [Azure AI Face Overview](https://learn.microsoft.com/azure/ai-services/computer-vision/overview-identity)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/libs/community/extended_testing_deps.txt b/libs/community/extended_testing_deps.txt
@@ -6,6 +6,7 @@ assemblyai>=0.17.0,<0.18
 atlassian-python-api>=3.36.0,<4
 azure-ai-contentsafety>=1.0.0
 azure-ai-documentintelligence>=1.0.0b1,<2
+azure.ai.vision.face>=1.0.0b1
 azure-identity>=1.15.0,<2
 azure-search-documents==11.4.0
 azure.ai.vision.imageanalysis>=1.0.0b3