scripts/menu_ingestion_search_json.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure OpenAI and AI Search Pipeline for Menu Ingestion\n",
    "\n",
    "## 1: Notebook Introduction\n",
    "This notebook demonstrates how to:\n",
    "1. Configure Azure OpenAI and Azure AI Search services.\n",
    "2. Prepare the JSON data for ingestion into Azure AI Search.\n",
    "3. Upload the prepared data to Azure AI Search for hybrid semantic search capabilities.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Install Required Packages\n",
    "\n",
    "### Description\n",
    "This cell installs all the necessary packages required for the notebook. \n",
    "It ensures that all dependencies are met before proceeding with the rest of the notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Collecting azure-core\n",
      "  Downloading azure_core-1.32.0-py3-none-any.whl.metadata (39 kB)\n",
      "Collecting azure-search-documents\n",
      "  Downloading azure_search_documents-11.5.2-py3-none-any.whl.metadata (23 kB)\n",
      "Collecting python-dotenv\n",
      "  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)\n",
      "Collecting langchain-openai\n",
      "  Downloading langchain_openai-0.2.14-py3-none-any.whl.metadata (2.7 kB)\n",
      "Collecting langchain-community\n",
      "  Downloading langchain_community-0.3.14-py3-none-any.whl.metadata (2.9 kB)\n",
      "Collecting openai\n",
      "  Downloading openai-1.59.3-py3-none-any.whl.metadata (27 kB)\n",
      "Collecting pydantic\n",
      "  Downloading pydantic-2.10.4-py3-none-any.whl.metadata (29 kB)\n",
      "Collecting tenacity\n",
      "  Downloading tenacity-9.0.0-py3-none-any.whl.metadata (1.2 kB)\n",
      "Collecting pdf2image\n",
      "  Downloading pdf2image-1.17.0-py3-none-any.whl.metadata (6.2 kB)\n",
      "Collecting pytesseract\n",
      "  Downloading pytesseract-0.3.13-py3-none-any.whl.metadata (11 kB)\n",
      "Collecting requests>=2.21.0 (from azure-core)\n",
      "  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)\n",
      "Requirement already satisfied: six>=1.11.0 in /home/vscode/.local/lib/python3.12/site-packages (from azure-core) (1.17.0)\n",
      "Collecting typing-extensions>=4.6.0 (from azure-core)\n",
      "  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)\n",
      "Collecting azure-common>=1.1 (from azure-search-documents)\n",
      "  Downloading azure_common-1.1.28-py2.py3-none-any.whl.metadata (5.0 kB)\n",
      "Collecting isodate>=0.6.0 (from azure-search-documents)\n",
      "  Downloading isodate-0.7.2-py3-none-any.whl.metadata (11 kB)\n",
      "Collecting langchain-core<0.4.0,>=0.3.27 (from langchain-openai)\n",
      "  Downloading langchain_core-0.3.29-py3-none-any.whl.metadata (6.3 kB)\n",
      "Collecting tiktoken<1,>=0.7 (from langchain-openai)\n",
      "  Downloading tiktoken-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (6.6 kB)\n",
      "Collecting PyYAML>=5.3 (from langchain-community)\n",
      "  Downloading PyYAML-6.0.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.1 kB)\n",
      "Collecting SQLAlchemy<3,>=1.4 (from langchain-community)\n",
      "  Downloading SQLAlchemy-2.0.36-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (9.7 kB)\n",
      "Collecting aiohttp<4.0.0,>=3.8.3 (from langchain-community)\n",
      "  Downloading aiohttp-3.11.11-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (7.7 kB)\n",
      "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain-community)\n",
      "  Downloading dataclasses_json-0.6.7-py3-none-any.whl.metadata (25 kB)\n",
      "Collecting httpx-sse<0.5.0,>=0.4.0 (from langchain-community)\n",
      "  Downloading httpx_sse-0.4.0-py3-none-any.whl.metadata (9.0 kB)\n",
      "Collecting langchain<0.4.0,>=0.3.14 (from langchain-community)\n",
      "  Downloading langchain-0.3.14-py3-none-any.whl.metadata (7.1 kB)\n",
      "Collecting langsmith<0.3,>=0.1.125 (from langchain-community)\n",
      "  Downloading langsmith-0.2.10-py3-none-any.whl.metadata (14 kB)\n",
      "Collecting numpy<3,>=1.26.2 (from langchain-community)\n",
      "  Downloading numpy-2.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (63 kB)\n",
      "Collecting pydantic-settings<3.0.0,>=2.4.0 (from langchain-community)\n",
      "  Downloading pydantic_settings-2.7.1-py3-none-any.whl.metadata (3.5 kB)\n",
      "Collecting anyio<5,>=3.5.0 (from openai)\n",
      "  Downloading anyio-4.7.0-py3-none-any.whl.metadata (4.7 kB)\n",
      "Collecting distro<2,>=1.7.0 (from openai)\n",
      "  Downloading distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)\n",
      "Collecting httpx<1,>=0.23.0 (from openai)\n",
      "  Downloading httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)\n",
      "Collecting jiter<1,>=0.4.0 (from openai)\n",
      "  Downloading jiter-0.8.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (5.2 kB)\n",
      "Collecting sniffio (from openai)\n",
      "  Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)\n",
      "Collecting tqdm>4 (from openai)\n",
      "  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)\n",
      "Collecting annotated-types>=0.6.0 (from pydantic)\n",
      "  Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)\n",
      "Collecting pydantic-core==2.27.2 (from pydantic)\n",
      "  Downloading pydantic_core-2.27.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (6.6 kB)\n",
      "Collecting pillow (from pdf2image)\n",
      "  Downloading pillow-11.1.0-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (9.1 kB)\n",
      "Requirement already satisfied: packaging>=21.3 in /home/vscode/.local/lib/python3.12/site-packages (from pytesseract) (24.2)\n",
      "Collecting aiohappyeyeballs>=2.3.0 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading aiohappyeyeballs-2.4.4-py3-none-any.whl.metadata (6.1 kB)\n",
      "Collecting aiosignal>=1.1.2 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading aiosignal-1.3.2-py2.py3-none-any.whl.metadata (3.8 kB)\n",
      "Collecting attrs>=17.3.0 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading attrs-24.3.0-py3-none-any.whl.metadata (11 kB)\n",
      "Collecting frozenlist>=1.1.1 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading frozenlist-1.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (13 kB)\n",
      "Collecting multidict<7.0,>=4.5 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading multidict-6.1.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (5.0 kB)\n",
      "Collecting propcache>=0.2.0 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading propcache-0.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (9.2 kB)\n",
      "Collecting yarl<2.0,>=1.17.0 (from aiohttp<4.0.0,>=3.8.3->langchain-community)\n",
      "  Downloading yarl-1.18.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (69 kB)\n",
      "Collecting idna>=2.8 (from anyio<5,>=3.5.0->openai)\n",
      "  Downloading idna-3.10-py3-none-any.whl.metadata (10 kB)\n",
      "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community)\n",
      "  Downloading marshmallow-3.23.3-py3-none-any.whl.metadata (7.1 kB)\n",
      "Collecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community)\n",
      "  Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)\n",
      "Collecting certifi (from httpx<1,>=0.23.0->openai)\n",
      "  Downloading certifi-2024.12.14-py3-none-any.whl.metadata (2.3 kB)\n",
      "Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai)\n",
      "  Downloading httpcore-1.0.7-py3-none-any.whl.metadata (21 kB)\n",
      "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai)\n",
      "  Downloading h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)\n",
      "Collecting langchain-text-splitters<0.4.0,>=0.3.3 (from langchain<0.4.0,>=0.3.14->langchain-community)\n",
      "  Downloading langchain_text_splitters-0.3.4-py3-none-any.whl.metadata (2.3 kB)\n",
      "Collecting jsonpatch<2.0,>=1.33 (from langchain-core<0.4.0,>=0.3.27->langchain-openai)\n",
      "  Downloading jsonpatch-1.33-py2.py3-none-any.whl.metadata (3.0 kB)\n",
      "Collecting orjson<4.0.0,>=3.9.14 (from langsmith<0.3,>=0.1.125->langchain-community)\n",
      "  Downloading orjson-3.10.13-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (41 kB)\n",
      "Collecting requests-toolbelt<2.0.0,>=1.0.0 (from langsmith<0.3,>=0.1.125->langchain-community)\n",
      "  Downloading requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB)\n",
      "Collecting charset-normalizer<4,>=2 (from requests>=2.21.0->azure-core)\n",
      "  Downloading charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (35 kB)\n",
      "Collecting urllib3<3,>=1.21.1 (from requests>=2.21.0->azure-core)\n",
      "  Downloading urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)\n",
      "Collecting greenlet!=0.4.17 (from SQLAlchemy<3,>=1.4->langchain-community)\n",
      "  Downloading greenlet-3.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.8 kB)\n",
      "Collecting regex>=2022.1.18 (from tiktoken<1,>=0.7->langchain-openai)\n",
      "  Downloading regex-2024.11.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (40 kB)\n",
      "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.27->langchain-openai)\n",
      "  Downloading jsonpointer-3.0.0-py2.py3-none-any.whl.metadata (2.3 kB)\n",
      "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community)\n",
      "  Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB)\n",
      "Downloading azure_core-1.32.0-py3-none-any.whl (198 kB)\n",
      "Downloading azure_search_documents-11.5.2-py3-none-any.whl (298 kB)\n",
      "Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
      "Downloading langchain_openai-0.2.14-py3-none-any.whl (50 kB)\n",
      "Downloading langchain_community-0.3.14-py3-none-any.whl (2.5 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m19.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading openai-1.59.3-py3-none-any.whl (454 kB)\n",
      "Downloading pydantic-2.10.4-py3-none-any.whl (431 kB)\n",
      "Downloading pydantic_core-2.27.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m24.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading tenacity-9.0.0-py3-none-any.whl (28 kB)\n",
      "Downloading pdf2image-1.17.0-py3-none-any.whl (11 kB)\n",
      "Downloading pytesseract-0.3.13-py3-none-any.whl (14 kB)\n",
      "Downloading aiohttp-3.11.11-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.7 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m22.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading annotated_types-0.7.0-py3-none-any.whl (13 kB)\n",
      "Downloading anyio-4.7.0-py3-none-any.whl (93 kB)\n",
      "Downloading azure_common-1.1.28-py2.py3-none-any.whl (14 kB)\n",
      "Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB)\n",
      "Downloading distro-1.9.0-py3-none-any.whl (20 kB)\n",
      "Downloading httpx-0.28.1-py3-none-any.whl (73 kB)\n",
      "Downloading httpcore-1.0.7-py3-none-any.whl (78 kB)\n",
      "Downloading httpx_sse-0.4.0-py3-none-any.whl (7.8 kB)\n",
      "Downloading isodate-0.7.2-py3-none-any.whl (22 kB)\n",
      "Downloading jiter-0.8.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (334 kB)\n",
      "Downloading langchain-0.3.14-py3-none-any.whl (1.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m18.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading langchain_core-0.3.29-py3-none-any.whl (411 kB)\n",
      "Downloading langsmith-0.2.10-py3-none-any.whl (326 kB)\n",
      "Downloading numpy-2.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.0/14.0 MB\u001b[0m \u001b[31m22.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hDownloading pillow-11.1.0-cp312-cp312-manylinux_2_28_aarch64.whl (4.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.4/4.4 MB\u001b[0m \u001b[31m25.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading pydantic_settings-2.7.1-py3-none-any.whl (29 kB)\n",
      "Downloading PyYAML-6.0.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (739 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m739.2/739.2 kB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading requests-2.32.3-py3-none-any.whl (64 kB)\n",
      "Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)\n",
      "Downloading SQLAlchemy-2.0.36-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.2/3.2 MB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading tiktoken-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
      "Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)\n",
      "Downloading aiohappyeyeballs-2.4.4-py3-none-any.whl (14 kB)\n",
      "Downloading aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)\n",
      "Downloading attrs-24.3.0-py3-none-any.whl (63 kB)\n",
      "Downloading certifi-2024.12.14-py3-none-any.whl (164 kB)\n",
      "Downloading charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (140 kB)\n",
      "Downloading frozenlist-1.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (282 kB)\n",
      "Downloading greenlet-3.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (649 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m649.1/649.1 kB\u001b[0m \u001b[31m20.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading idna-3.10-py3-none-any.whl (70 kB)\n",
      "Downloading jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
      "Downloading langchain_text_splitters-0.3.4-py3-none-any.whl (27 kB)\n",
      "Downloading marshmallow-3.23.3-py3-none-any.whl (48 kB)\n",
      "Downloading multidict-6.1.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (130 kB)\n",
      "Downloading orjson-3.10.13-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (136 kB)\n",
      "Downloading propcache-0.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (241 kB)\n",
      "Downloading regex-2024.11.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (794 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m795.0/795.0 kB\u001b[0m \u001b[31m21.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading requests_toolbelt-1.0.0-py2.py3-none-any.whl (54 kB)\n",
      "Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
      "Downloading urllib3-2.3.0-py3-none-any.whl (128 kB)\n",
      "Downloading yarl-1.18.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (332 kB)\n",
      "Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
      "Downloading jsonpointer-3.0.0-py2.py3-none-any.whl (7.6 kB)\n",
      "Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
      "Installing collected packages: azure-common, urllib3, typing-extensions, tqdm, tenacity, sniffio, regex, PyYAML, python-dotenv, propcache, pillow, orjson, numpy, mypy-extensions, multidict, marshmallow, jsonpointer, jiter, isodate, idna, httpx-sse, h11, greenlet, frozenlist, distro, charset-normalizer, certifi, attrs, annotated-types, aiohappyeyeballs, yarl, typing-inspect, SQLAlchemy, requests, pytesseract, pydantic-core, pdf2image, jsonpatch, httpcore, anyio, aiosignal, tiktoken, requests-toolbelt, pydantic, httpx, dataclasses-json, azure-core, aiohttp, pydantic-settings, openai, langsmith, azure-search-documents, langchain-core, langchain-text-splitters, langchain-openai, langchain, langchain-community\n",
      "Successfully installed PyYAML-6.0.2 SQLAlchemy-2.0.36 aiohappyeyeballs-2.4.4 aiohttp-3.11.11 aiosignal-1.3.2 annotated-types-0.7.0 anyio-4.7.0 attrs-24.3.0 azure-common-1.1.28 azure-core-1.32.0 azure-search-documents-11.5.2 certifi-2024.12.14 charset-normalizer-3.4.1 dataclasses-json-0.6.7 distro-1.9.0 frozenlist-1.5.0 greenlet-3.1.1 h11-0.14.0 httpcore-1.0.7 httpx-0.28.1 httpx-sse-0.4.0 idna-3.10 isodate-0.7.2 jiter-0.8.2 jsonpatch-1.33 jsonpointer-3.0.0 langchain-0.3.14 langchain-community-0.3.14 langchain-core-0.3.29 langchain-openai-0.2.14 langchain-text-splitters-0.3.4 langsmith-0.2.10 marshmallow-3.23.3 multidict-6.1.0 mypy-extensions-1.0.0 numpy-2.2.1 openai-1.59.3 orjson-3.10.13 pdf2image-1.17.0 pillow-11.1.0 propcache-0.2.1 pydantic-2.10.4 pydantic-core-2.27.2 pydantic-settings-2.7.1 pytesseract-0.3.13 python-dotenv-1.0.1 regex-2024.11.6 requests-2.32.3 requests-toolbelt-1.0.0 sniffio-1.3.1 tenacity-9.0.0 tiktoken-0.8.0 tqdm-4.67.1 typing-extensions-4.12.2 typing-inspect-0.9.0 urllib3-2.3.0 yarl-1.18.3\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install azure-core azure-search-documents python-dotenv langchain-openai langchain-community openai pydantic tenacity pdf2image pytesseract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Imports and Environment Setup\n",
    "\n",
    "### Description\n",
    "This cell imports necessary libraries and loads environment variables using `dotenv`. \n",
    "Ensure your `.env` file is properly set up with the required Azure API keys and endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import required libraries\n",
    "from azure.core.credentials import AzureKeyCredential\n",
    "from azure.core.exceptions import HttpResponseError\n",
    "from azure.search.documents import SearchClient\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "from azure.search.documents.indexes.models import (\n",
    "    SearchIndex,\n",
    "    SimpleField,\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    VectorSearch,\n",
    "    HnswAlgorithmConfiguration,\n",
    "    VectorSearchProfile,\n",
    "    AzureOpenAIVectorizer,\n",
    "    AzureOpenAIVectorizerParameters,\n",
    "    SemanticConfiguration,\n",
    "    SemanticPrioritizedFields,\n",
    "    SemanticField,\n",
    "    SemanticSearch\n",
    ")\n",
    "from dotenv import load_dotenv\n",
    "from langchain_community.vectorstores.azuresearch import AzureSearch\n",
    "from langchain_openai import AzureOpenAIEmbeddings\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "from openai import AzureOpenAI\n",
    "from pydantic import BaseModel\n",
    "from tenacity import retry, stop_after_attempt, wait_exponential\n",
    "from typing import List\n",
    "\n",
    "import base64\n",
    "import json\n",
    "import os\n",
    "import openai\n",
    "import re\n",
    "\n",
    "# Load environment variables\n",
    "load_dotenv()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: Azure OpenAI and Azure AI Search Configuration\n",
    "\n",
    "### Description\n",
    "This cell sets up the Azure OpenAI and AI Search configurations, including the embeddings and vector store. \n",
    "Ensure that the endpoints, API keys, and deployment names in the `.env` file match your Azure resource setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure OpenAI and Azure Search clients initialized successfully.\n"
     ]
    }
   ],
   "source": [
    "# Azure OpenAI setup\n",
    "aoai_eastus_endpoint = os.getenv(\"AZURE_OPENAI_EASTUS_ENDPOINT\")\n",
    "aoai_eastus_api_key = os.getenv(\"AZURE_OPENAI_EASTUS_API_KEY\")\n",
    "aoai_gpt4o_deployment = os.getenv(\"AZURE_OPENAI_GPT4O_DEPLOYMENT\")\n",
    "aoai_gpt4o_mini_deployment = os.getenv(\"AZURE_OPENAI_GPT4O_MINI_DEPLOYMENT\")\n",
    "aoai_openai_api_version = os.getenv(\"AZURE_OPENAI_API_VERSION\")\n",
    "aoai_embedding_deployment = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT\")\n",
    "\n",
    "# Initialize the Azure OpenAI client\n",
    "aoai_client = AzureOpenAI(\n",
    "    azure_endpoint=aoai_eastus_endpoint,\n",
    "    api_version=aoai_openai_api_version,\n",
    "    api_key=aoai_eastus_api_key,\n",
    ")\n",
    "\n",
    "# Azure AI Search credentials\n",
    "search_service_endpoint = os.getenv(\"AZURE_SEARCH_ENDPOINT\")\n",
    "search_api_key = os.getenv(\"AZURE_SEARCH_API_KEY\")\n",
    "index_name = os.getenv(\"AZURE_SEARCH_INDEX\")\n",
    "search_client = SearchClient(endpoint=search_service_endpoint, index_name=index_name, credential=AzureKeyCredential(search_api_key))\n",
    "search_index_client = SearchIndexClient(endpoint=search_service_endpoint, index_name=index_name, credential=AzureKeyCredential(search_api_key))\n",
    "\n",
    "print(\"Azure OpenAI and Azure Search clients initialized successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5: Define and Create/Update Index Schema with Semantic Configuration\n",
    "\n",
    "### Description\n",
    "This cell defines the schema for the Azure AI Search index, including fields for semantic search and vector search capabilities. It also includes logic to delete the existing index if it exists and create or update the index schema with the new configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted existing index: coffee-chat2\n",
      "Created index: coffee-chat2\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Define and Create/Update Index Schema with Semantic Configuration\n",
    "index_schema = SearchIndex(\n",
    "    name=index_name,\n",
    "    fields=[\n",
    "        # Unique identifier for each menu item\n",
    "        SimpleField(name=\"id\", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True),\n",
    "        \n",
    "        # Fields for semantic search\n",
    "        SearchField(name=\"category\", type=SearchFieldDataType.String, sortable=True, filterable=True, facetable=True),\n",
    "        SearchField(name=\"name\", type=SearchFieldDataType.String, sortable=True, filterable=True, facetable=True),\n",
    "        SearchField(name=\"description\", type=SearchFieldDataType.String),\n",
    "        SearchField(name=\"longDescription\", type=SearchFieldDataType.String),  # Added long description\n",
    "        SearchField(name=\"origin\", type=SearchFieldDataType.String, filterable=True, facetable=True),\n",
    "        SearchField(name=\"caffeineContent\", type=SearchFieldDataType.String, filterable=True),  # Treat as string for now\n",
    "        SearchField(name=\"brewingMethod\", type=SearchFieldDataType.String, filterable=True),\n",
    "        SearchField(name=\"popularity\", type=SearchFieldDataType.String, filterable=True, facetable=True),\n",
    "        \n",
    "        # Sizes as JSON object\n",
    "        SearchField(name=\"sizes\", type=SearchFieldDataType.String, filterable=False, facetable=False),\n",
    "\n",
    "        # Embedding field for vector search\n",
    "        SearchField(\n",
    "            name=\"embedding\", \n",
    "            type=SearchFieldDataType.Collection(SearchFieldDataType.Single), \n",
    "            vector_search_dimensions=3072,  # Adjusted for most OpenAI embedding models\n",
    "            vector_search_profile_name=\"menuHnswProfile\"\n",
    "        )\n",
    "    ],\n",
    "    vector_search=VectorSearch(\n",
    "        algorithms=[\n",
    "            HnswAlgorithmConfiguration(\n",
    "                name=\"menuHnsw\",\n",
    "                kind=\"hnsw\",\n",
    "                parameters={\n",
    "                    \"m\": 10,  # Adjusted for accuracy/memory trade-off\n",
    "                    \"efConstruction\": 200  # Ensures recall during indexing\n",
    "                }\n",
    "            )\n",
    "        ],\n",
    "        profiles=[\n",
    "            VectorSearchProfile(\n",
    "                name=\"menuHnswProfile\",\n",
    "                algorithm_configuration_name=\"menuHnsw\",\n",
    "                vectorizer_name=\"menuVectorizer\"\n",
    "            )\n",
    "        ],\n",
    "        vectorizers=[\n",
    "            AzureOpenAIVectorizer(\n",
    "                vectorizer_name=\"menuVectorizer\",\n",
    "                parameters=AzureOpenAIVectorizerParameters(\n",
    "                    resource_url=aoai_eastus_endpoint,\n",
    "                    deployment_name=aoai_embedding_deployment,\n",
    "                    model_name=aoai_embedding_deployment,\n",
    "                    api_key=aoai_eastus_api_key\n",
    "                )\n",
    "            )\n",
    "        ]\n",
    "    ),\n",
    "    semantic_search=SemanticSearch(\n",
    "        configurations=[\n",
    "            SemanticConfiguration(\n",
    "                name=\"menuSemanticConfig\",\n",
    "                prioritized_fields=SemanticPrioritizedFields(\n",
    "                    title_field=SemanticField(field_name=\"name\"),  # Prioritize the \"name\" (e.g., \"Espresso\")\n",
    "                    content_fields=[\n",
    "                        SemanticField(field_name=\"description\"),  # Primary content field\n",
    "                        SemanticField(field_name=\"longDescription\"),  # Provide detailed context\n",
    "                        SemanticField(field_name=\"category\")  # Assist in grouping similar items\n",
    "                    ]\n",
    "                )\n",
    "            )\n",
    "        ]\n",
    "    ),\n",
    ")\n",
    "\n",
    "# Delete the existing index if it exists\n",
    "try:\n",
    "    search_index_client.delete_index(index_name)\n",
    "    print(f\"Deleted existing index: {index_name}\")\n",
    "except Exception as e:\n",
    "    print(f\"Index {index_name} does not exist or could not be deleted: {e}\")\n",
    "\n",
    "# Create or update the index schema\n",
    "search_index_client.create_or_update_index(index=index_schema)\n",
    "print(f\"Created index: {index_name}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6: Load and Process Menu Data\n",
    "\n",
    "### Description\n",
    "This cell reads the `menuItems.json` file, processes the data to ensure all fields are populated, and prints the structured JSON data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"menuItems\": [\n",
      "    {\n",
      "      \"category\": \"Coffee\",\n",
      "      \"items\": [\n",
      "        {\n",
      "          \"name\": \"Espresso\",\n",
      "          \"description\": \"Rich and bold single or double shot\",\n",
      "          \"longDescription\": \"Espresso is the quintessential coffee experience, brewed under pressure to extract its bold, rich flavor. A favorite in Italy and around the world, it forms the base for many coffee drinks, including cappuccinos and lattes. Perfect for those who enjoy an intense coffee kick with a creamy layer of crema.\",\n",
      "          \"origin\": \"Italy\",\n",
      "          \"caffeineContent\": \"63 mg per shot\",\n",
      "          \"brewingMethod\": \"Espresso Machine\",\n",
      "          \"popularity\": \"High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"single\",\n",
      "              \"price\": 1.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"double\",\n",
      "              \"price\": 2.0\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Americano\",\n",
      "          \"description\": \"Espresso with hot water\",\n",
      "          \"longDescription\": \"The Americano is a smooth, diluted espresso drink inspired by American GIs in Italy during WWII who added water to espresso to mimic drip coffee. It retains the rich essence of espresso while offering a milder flavor profile, perfect for those who prefer a less intense coffee experience.\",\n",
      "          \"origin\": \"United States\",\n",
      "          \"caffeineContent\": \"95 mg for medium\",\n",
      "          \"brewingMethod\": \"Espresso Machine with water dilution\",\n",
      "          \"popularity\": \"Moderate\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 2.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 4.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 5.0\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Cappuccino\",\n",
      "          \"description\": \"Espresso with steamed milk and foam\",\n",
      "          \"longDescription\": \"Cappuccino combines the boldness of espresso with a luxurious layer of steamed milk and a light, frothy foam topping. Originating in Italy, it's a classic choice for those who enjoy a balanced, creamy texture with every sip.\",\n",
      "          \"origin\": \"Italy\",\n",
      "          \"caffeineContent\": \"75 mg for medium\",\n",
      "          \"brewingMethod\": \"Espresso Machine with milk frothing\",\n",
      "          \"popularity\": \"High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 3.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 4.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 5.5\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Latte\",\n",
      "          \"description\": \"Espresso with steamed milk\",\n",
      "          \"longDescription\": \"The Latte, meaning 'milk' in Italian, is a creamy coffee beverage made with espresso and a generous amount of steamed milk, topped with a thin layer of foam. It's perfect for those who prefer a milder coffee taste and pairs well with flavored syrups.\",\n",
      "          \"origin\": \"Italy\",\n",
      "          \"caffeineContent\": \"75 mg for medium\",\n",
      "          \"brewingMethod\": \"Espresso Machine with milk steaming\",\n",
      "          \"popularity\": \"Very High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 3.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 4.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 5.5\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Mocha\",\n",
      "          \"description\": \"Espresso with chocolate and steamed milk\",\n",
      "          \"longDescription\": \"The Mocha is a delightful blend of espresso, steamed milk, and rich chocolate syrup, often topped with whipped cream. Originating from Yemen, it offers a sweet and indulgent coffee experience that's perfect for chocolate lovers.\",\n",
      "          \"origin\": \"Yemen\",\n",
      "          \"caffeineContent\": \"80 mg for medium\",\n",
      "          \"brewingMethod\": \"Espresso Machine with chocolate mix\",\n",
      "          \"popularity\": \"High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 4.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 5.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 6.0\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Coffee Infusion\",\n",
      "          \"description\": \"French Press infused at your table (about 3 cups; please wait 2 minutes to push the press).\",\n",
      "          \"longDescription\": \"Our Coffee Infusion is a table-side experience using the French Press method. This technique allows coffee grounds to steep in hot water, enhancing their natural flavors and oils for a rich, full-bodied brew. Ideal for sharing or savoring solo.\",\n",
      "          \"origin\": \"France\",\n",
      "          \"caffeineContent\": \"120 mg per serving\",\n",
      "          \"brewingMethod\": \"French Press\",\n",
      "          \"popularity\": \"Unique\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"kannchen\",\n",
      "              \"price\": 6.0\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Turkish Coffee\",\n",
      "          \"description\": \"Pulverized light roast beans blended with cardamom and sugar, boiled 3 times, as served traditionally.\",\n",
      "          \"longDescription\": \"Turkish Coffee is a traditional preparation involving finely ground coffee beans, cardamom, and sugar boiled together. It's a rich, aromatic brew served in small cups with grounds settled at the bottom. Perfect for savoring slowly.\",\n",
      "          \"origin\": \"Turkey\",\n",
      "          \"caffeineContent\": \"50 mg per small cup\",\n",
      "          \"brewingMethod\": \"Boiled\",\n",
      "          \"popularity\": \"Traditional\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"pot\",\n",
      "              \"price\": 4.5\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Caf\\u00e9 Cubano\",\n",
      "          \"description\": \"Double-rich espresso extraction with caramelized sugar.\",\n",
      "          \"longDescription\": \"Caf\\u00e9 Cubano is a Cuban specialty featuring espresso sweetened with sugar during the brewing process, creating a thick, syrupy coffee shot. It's a perfect blend of boldness and sweetness for an energizing treat.\",\n",
      "          \"origin\": \"Cuba\",\n",
      "          \"caffeineContent\": \"63 mg per shot\",\n",
      "          \"brewingMethod\": \"Espresso Machine\",\n",
      "          \"popularity\": \"High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"single\",\n",
      "              \"price\": 2.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"double\",\n",
      "              \"price\": 4.0\n",
      "            }\n",
      "          ]\n",
      "        }\n",
      "      ]\n",
      "    },\n",
      "    {\n",
      "      \"category\": \"Chilled Coffees\",\n",
      "      \"items\": [\n",
      "        {\n",
      "          \"name\": \"Caf\\u00e9 Johannes\",\n",
      "          \"description\": \"Espresso doppio over two scoops of vanilla ice cream, with whipped cream\",\n",
      "          \"longDescription\": \"Caf\\u00e9 Johannes combines the boldness of double espresso with the creaminess of vanilla ice cream and whipped cream. It's a dessert-like coffee treat that's both refreshing and indulgent.\",\n",
      "          \"origin\": \"Italy\",\n",
      "          \"caffeineContent\": \"75 mg per serving\",\n",
      "          \"brewingMethod\": \"Espresso Machine\",\n",
      "          \"popularity\": \"High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"single\",\n",
      "              \"price\": 5.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"double\",\n",
      "              \"price\": 6.5\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Granita Cappuccino\",\n",
      "          \"description\": \"Granules of sugar and ice mixed with espresso, cream, cocoa, and whipped cream\",\n",
      "          \"longDescription\": \"The Granita Cappuccino is a frozen coffee blend featuring espresso, cream, cocoa, and sugar. It's topped with whipped cream for a refreshing, sweet treat on hot days.\",\n",
      "          \"origin\": \"Custom House Recipe\",\n",
      "          \"caffeineContent\": \"80 mg for medium\",\n",
      "          \"brewingMethod\": \"Blended\",\n",
      "          \"popularity\": \"Refreshing\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 4.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 5.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 6.5\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Caff\\u00e9 Alpine\",\n",
      "          \"description\": \"Espresso, white chocolate, and cream, served over ice\",\n",
      "          \"longDescription\": \"Caff\\u00e9 Alpine offers a smooth combination of espresso, white chocolate, and cream served over ice. It's a creamy and cooling coffee option perfect for those who enjoy sweet iced drinks.\",\n",
      "          \"origin\": \"Custom House Recipe\",\n",
      "          \"caffeineContent\": \"70 mg for medium\",\n",
      "          \"brewingMethod\": \"Espresso Machine with ice\",\n",
      "          \"popularity\": \"Moderate\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 4.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 5.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 6.0\n",
      "            }\n",
      "          ]\n",
      "        }\n",
      "      ]\n",
      "    },\n",
      "    {\n",
      "      \"category\": \"Hot Chocolates\",\n",
      "      \"items\": [\n",
      "        {\n",
      "          \"name\": \"Hot Chocolate Deluxe\",\n",
      "          \"description\": \"Rich steamed cream and milk, Dutch chocolate, fresh whipped cream, and shaved dark chocolate\",\n",
      "          \"longDescription\": \"Hot Chocolate Deluxe is a decadent beverage featuring rich Dutch chocolate steamed with cream and milk. Topped with fresh whipped cream and shaved dark chocolate, it's a comforting indulgence.\",\n",
      "          \"origin\": \"Custom House Recipe\",\n",
      "          \"caffeineContent\": \"\",\n",
      "          \"brewingMethod\": \"\",\n",
      "          \"popularity\": \"Very High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 3.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 4.0\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 5.0\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Intermezzo Hot Chocolate\",\n",
      "          \"description\": \"Rich steamed cream, Dutch chocolate, fresh whipped cream, and the essence of almond\",\n",
      "          \"longDescription\": \"The Intermezzo Hot Chocolate adds a nutty twist to the classic hot chocolate with the essence of almond. Perfect for those who love a bit of sophistication in their chocolate beverages.\",\n",
      "          \"origin\": \"Custom House Recipe\",\n",
      "          \"caffeineContent\": \"\",\n",
      "          \"brewingMethod\": \"\",\n",
      "          \"popularity\": \"High\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 3.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 4.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 5.5\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Mexican Hot Chocolate\",\n",
      "          \"description\": \"Semi-sweet dark chocolate, Mexican vanilla, milk, and cinnamon\",\n",
      "          \"longDescription\": \"Mexican Hot Chocolate blends semi-sweet dark chocolate with Mexican vanilla and cinnamon, creating a warm, spiced drink that's perfect for cozy evenings.\",\n",
      "          \"origin\": \"Mexico\",\n",
      "          \"caffeineContent\": \"\",\n",
      "          \"brewingMethod\": \"\",\n",
      "          \"popularity\": \"Cultural Favorite\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"small\",\n",
      "              \"price\": 3.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"medium\",\n",
      "              \"price\": 4.5\n",
      "            },\n",
      "            {\n",
      "              \"size\": \"large\",\n",
      "              \"price\": 5.5\n",
      "            }\n",
      "          ]\n",
      "        }\n",
      "      ]\n",
      "    },\n",
      "    {\n",
      "      \"category\": \"Extras\",\n",
      "      \"items\": [\n",
      "        {\n",
      "          \"name\": \"Extra Shot\",\n",
      "          \"description\": \"Add an extra shot of espresso\",\n",
      "          \"longDescription\": \"Boost your beverage with an extra shot of espresso, adding richness and a caffeine kick to your favorite drink.\",\n",
      "          \"origin\": \"\",\n",
      "          \"caffeineContent\": \"63 mg\",\n",
      "          \"brewingMethod\": \"\",\n",
      "          \"popularity\": \"\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"standard\",\n",
      "              \"price\": 1.0\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Flavor Shot\",\n",
      "          \"description\": \"Add vanilla, caramel, or hazelnut\",\n",
      "          \"longDescription\": \"Customize your coffee with a flavor shot. Choose from vanilla, caramel, or hazelnut to enhance the taste of your drink.\",\n",
      "          \"origin\": \"\",\n",
      "          \"caffeineContent\": \"\",\n",
      "          \"brewingMethod\": \"\",\n",
      "          \"popularity\": \"\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"standard\",\n",
      "              \"price\": 0.75\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"Whipped Cream\",\n",
      "          \"description\": \"Add whipped cream topping\",\n",
      "          \"longDescription\": \"Top your coffee or hot chocolate with our fresh whipped cream, made from heavy cream, sugar, and a hint of vanilla for added indulgence.\",\n",
      "          \"origin\": \"\",\n",
      "          \"caffeineContent\": \"\",\n",
      "          \"brewingMethod\": \"\",\n",
      "          \"popularity\": \"\",\n",
      "          \"sizes\": [\n",
      "            {\n",
      "              \"size\": \"standard\",\n",
      "              \"price\": 0.5\n",
      "            }\n",
      "          ]\n",
      "        }\n",
      "      ]\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Define the path to the menuItems.json file\n",
    "menu_items_path = os.path.join('..', 'frontend', 'src', 'data', 'menuItems.json')\n",
    "\n",
    "# Read the JSON file\n",
    "with open(menu_items_path, 'r') as file:\n",
    "    menu_items = json.load(file)\n",
    "    \n",
    "\n",
    "# Build JSON with all fields, put empty string if not applicable\n",
    "structured_menu_items = {\"menuItems\": []}\n",
    "\n",
    "for category in menu_items['menuItems']:\n",
    "    category_dict = {\"category\": category['category'], \"items\": []}\n",
    "    for item in category['items']:\n",
    "        item_dict = {\n",
    "            \"name\": item['name'],\n",
    "            \"description\": item.get('description', ''),\n",
    "            \"longDescription\": item.get('longDescription', ''),\n",
    "            \"origin\": item.get('origin', ''),\n",
    "            \"caffeineContent\": item.get('caffeineContent', ''),\n",
    "            \"brewingMethod\": item.get('brewingMethod', ''),\n",
    "            \"popularity\": item.get('popularity', ''),\n",
    "            \"sizes\": [{\"size\": size['size'], \"price\": size['price']} for size in item.get('sizes', [])]\n",
    "        }\n",
    "        category_dict[\"items\"].append(item_dict)\n",
    "    structured_menu_items[\"menuItems\"].append(category_dict)\n",
    "\n",
    "print(json.dumps(structured_menu_items, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7: Data Preparation and Embedding Generation\n",
    "\n",
    "### Description\n",
    "This cell defines functions to sanitize document keys, generate embeddings using Azure OpenAI, and prepare the structured JSON data for ingestion into Azure AI Search. It includes retry logic for embedding generation and assigns embeddings to each document.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID: coffee_espresso\n",
      "Category: Coffee\n",
      "Name: Espresso\n",
      "Description: Rich and bold single or double shot\n",
      "Long Description: Espresso is the quintessential coffee experience, brewed under pressure to extract its bold, rich flavor. A favorite in Italy and around the world, it forms the base for many coffee drinks, including cappuccinos and lattes. Perfect for those who enjoy an intense coffee kick with a creamy layer of crema.\n",
      "Origin: Italy\n",
      "Caffeine Content: 63 mg per shot\n",
      "Brewing Method: Espresso Machine\n",
      "Popularity: High\n",
      "Sizes: [{\"size\": \"single\", \"price\": 1.0}, {\"size\": \"double\", \"price\": 2.0}]\n",
      "Embedding: [0.0025835344567894936, -0.03195502236485481, -0.023361852392554283, -0.012528417631983757, -0.06191285327076912, 0.020405488088726997, 0.006648536305874586, -0.025989733636379242, -0.04241398349404335, 0.008428924717009068]...\n",
      "\n",
      "ID: coffee_americano\n",
      "Category: Coffee\n",
      "Name: Americano\n",
      "Description: Espresso with hot water\n",
      "Long Description: The Americano is a smooth, diluted espresso drink inspired by American GIs in Italy during WWII who added water to espresso to mimic drip coffee. It retains the rich essence of espresso while offering a milder flavor profile, perfect for those who prefer a less intense coffee experience.\n",
      "Origin: United States\n",
      "Caffeine Content: 95 mg for medium\n",
      "Brewing Method: Espresso Machine with water dilution\n",
      "Popularity: Moderate\n",
      "Sizes: [{\"size\": \"small\", \"price\": 2.5}, {\"size\": \"medium\", \"price\": 4.0}, {\"size\": \"large\", \"price\": 5.0}]\n",
      "Embedding: [-0.01883688010275364, -0.01189530547708273, -0.021440763026475906, 0.006474780384451151, -0.04303395003080368, -0.0003119897737633437, 0.023396851494908333, -0.025975333526730537, -0.023561976850032806, 0.024425704032182693]...\n",
      "\n",
      "ID: coffee_cappuccino\n",
      "Category: Coffee\n",
      "Name: Cappuccino\n",
      "Description: Espresso with steamed milk and foam\n",
      "Long Description: Cappuccino combines the boldness of espresso with a luxurious layer of steamed milk and a light, frothy foam topping. Originating in Italy, it's a classic choice for those who enjoy a balanced, creamy texture with every sip.\n",
      "Origin: Italy\n",
      "Caffeine Content: 75 mg for medium\n",
      "Brewing Method: Espresso Machine with milk frothing\n",
      "Popularity: High\n",
      "Sizes: [{\"size\": \"small\", \"price\": 3.5}, {\"size\": \"medium\", \"price\": 4.5}, {\"size\": \"large\", \"price\": 5.5}]\n",
      "Embedding: [-0.025414273142814636, -0.032341182231903076, -0.020921140909194946, 0.03624926880002022, -0.042169906198978424, 0.0022129255812615156, -0.03367508202791214, -0.03725554421544075, -0.04518872871994972, 0.020277593284845352]...\n",
      "\n",
      "ID: coffee_latte\n",
      "Category: Coffee\n",
      "Name: Latte\n",
      "Description: Espresso with steamed milk\n",
      "Long Description: The Latte, meaning 'milk' in Italian, is a creamy coffee beverage made with espresso and a generous amount of steamed milk, topped with a thin layer of foam. It's perfect for those who prefer a milder coffee taste and pairs well with flavored syrups.\n",
      "Origin: Italy\n",
      "Caffeine Content: 75 mg for medium\n",
      "Brewing Method: Espresso Machine with milk steaming\n",
      "Popularity: Very High\n",
      "Sizes: [{\"size\": \"small\", \"price\": 3.5}, {\"size\": \"medium\", \"price\": 4.5}, {\"size\": \"large\", \"price\": 5.5}]\n",
      "Embedding: [-0.03617272898554802, -0.030359255149960518, -0.02643575705587864, 0.03825409710407257, -0.06086205318570137, 0.012667632661759853, -0.01790693588554859, -0.038684722036123276, -0.041627347469329834, 0.010191522538661957]...\n",
      "\n",
      "ID: coffee_mocha\n",
      "Category: Coffee\n",
      "Name: Mocha\n",
      "Description: Espresso with chocolate and steamed milk\n",
      "Long Description: The Mocha is a delightful blend of espresso, steamed milk, and rich chocolate syrup, often topped with whipped cream. Originating from Yemen, it offers a sweet and indulgent coffee experience that's perfect for chocolate lovers.\n",
      "Origin: Yemen\n",
      "Caffeine Content: 80 mg for medium\n",
      "Brewing Method: Espresso Machine with chocolate mix\n",
      "Popularity: High\n",
      "Sizes: [{\"size\": \"small\", \"price\": 4.0}, {\"size\": \"medium\", \"price\": 5.0}, {\"size\": \"large\", \"price\": 6.0}]\n",
      "Embedding: [0.005473959259688854, -0.03911183401942253, -0.02105928584933281, 0.02744860202074051, -0.040639448910951614, 0.029558168724179268, -0.011390442959964275, -0.042506538331508636, -0.022914249449968338, 0.0370507650077343]...\n",
      "\n",
      "ID: coffee_coffee_infusion\n",
      "Category: Coffee\n",
      "Name: Coffee Infusion\n",
      "Description: French Press infused at your table (about 3 cups; please wait 2 minutes to push the press).\n",
      "Long Description: Our Coffee Infusion is a table-side experience using the French Press method. This technique allows coffee grounds to steep in hot water, enhancing their natural flavors and oils for a rich, full-bodied brew. Ideal for sharing or savoring solo.\n",
      "Origin: France\n",
      "Caffeine Content: 120 mg per serving\n",
      "Brewing Method: French Press\n",
      "Popularity: Unique\n",
      "Sizes: [{\"size\": \"kannchen\", \"price\": 6.0}]\n",
      "Embedding: [-0.0220537930727005, -0.03403417021036148, -0.024106325581669807, -0.027512652799487114, -0.040002524852752686, -0.021078476682305336, 0.01952088251709938, -0.002163527999073267, -0.03260758891701698, 0.030132906511425972]...\n",
      "\n",
      "ID: coffee_turkish_coffee\n",
      "Category: Coffee\n",
      "Name: Turkish Coffee\n",
      "Description: Pulverized light roast beans blended with cardamom and sugar, boiled 3 times, as served traditionally.\n",
      "Long Description: Turkish Coffee is a traditional preparation involving finely ground coffee beans, cardamom, and sugar boiled together. It's a rich, aromatic brew served in small cups with grounds settled at the bottom. Perfect for savoring slowly.\n",
      "Origin: Turkey\n",
      "Caffeine Content: 50 mg per small cup\n",
      "Brewing Method: Boiled\n",
      "Popularity: Traditional\n",
      "Sizes: [{\"size\": \"pot\", \"price\": 4.5}]\n",
      "Embedding: [-0.0053310939110815525, -0.04494704306125641, -0.027366280555725098, 0.008689682930707932, -0.014287331141531467, 0.04340694844722748, -0.006438776385039091, -0.05316877365112305, -0.021667934954166412, 0.012190434150397778]...\n",
      "\n",
      "ID: coffee_caf__cubano\n",
      "Category: Coffee\n",
      "Name: Café Cubano\n",
      "Description: Double-rich espresso extraction with caramelized sugar.\n",
      "Long Description: Café Cubano is a Cuban specialty featuring espresso sweetened with sugar during the brewing process, creating a thick, syrupy coffee shot. It's a perfect blend of boldness and sweetness for an energizing treat.\n",
      "Origin: Cuba\n",
      "Caffeine Content: 63 mg per shot\n",
      "Brewing Method: Espresso Machine\n",
      "Popularity: High\n",
      "Sizes: [{\"size\": \"single\", \"price\": 2.5}, {\"size\": \"double\", \"price\": 4.0}]\n",
      "Embedding: [-0.008482846431434155, -0.028455426916480064, -0.026304157450795174, 0.0013743372401222587, -0.02082819864153862, 0.017393501475453377, 0.006016831379383802, -0.021647147834300995, -0.048990268260240555, 0.01974034123122692]...\n",
      "\n",
      "ID: chilled_coffees_caf__johannes\n",
      "Category: Chilled Coffees\n",
      "Name: Café Johannes\n",
      "Description: Espresso doppio over two scoops of vanilla ice cream, with whipped cream\n",
      "Long Description: Café Johannes combines the boldness of double espresso with the creaminess of vanilla ice cream and whipped cream. It's a dessert-like coffee treat that's both refreshing and indulgent.\n",
      "Origin: Italy\n",
      "Caffeine Content: 75 mg per serving\n",
      "Brewing Method: Espresso Machine\n",
      "Popularity: High\n",
      "Sizes: [{\"size\": \"single\", \"price\": 5.0}, {\"size\": \"double\", \"price\": 6.5}]\n",
      "Embedding: [0.004237833432853222, -0.04890267178416252, -0.02382744289934635, 0.027345143258571625, -0.059946924448013306, -0.013148235157132149, 7.252923751366325e-06, -0.026960186660289764, -0.036902669817209244, 0.05962834134697914]...\n",
      "\n",
      "ID: chilled_coffees_granita_cappuccino\n",
      "Category: Chilled Coffees\n",
      "Name: Granita Cappuccino\n",
      "Description: Granules of sugar and ice mixed with espresso, cream, cocoa, and whipped cream\n",
      "Long Description: The Granita Cappuccino is a frozen coffee blend featuring espresso, cream, cocoa, and sugar. It's topped with whipped cream for a refreshing, sweet treat on hot days.\n",
      "Origin: Custom House Recipe\n",
      "Caffeine Content: 80 mg for medium\n",
      "Brewing Method: Blended\n",
      "Popularity: Refreshing\n",
      "Sizes: [{\"size\": \"small\", \"price\": 4.5}, {\"size\": \"medium\", \"price\": 5.5}, {\"size\": \"large\", \"price\": 6.5}]\n",
      "Embedding: [-0.01660916954278946, -0.028453504666686058, -0.023688672110438347, 0.040625281631946564, -0.011251554824411869, -0.012984734028577805, -0.02316928282380104, -0.04963555932044983, -0.04227377846837044, 0.03805091604590416]...\n",
      "\n",
      "ID: chilled_coffees_caff__alpine\n",
      "Category: Chilled Coffees\n",
      "Name: Caffé Alpine\n",
      "Description: Espresso, white chocolate, and cream, served over ice\n",
      "Long Description: Caffé Alpine offers a smooth combination of espresso, white chocolate, and cream served over ice. It's a creamy and cooling coffee option perfect for those who enjoy sweet iced drinks.\n",
      "Origin: Custom House Recipe\n",
      "Caffeine Content: 70 mg for medium\n",
      "Brewing Method: Espresso Machine with ice\n",
      "Popularity: Moderate\n",
      "Sizes: [{\"size\": \"small\", \"price\": 4.0}, {\"size\": \"medium\", \"price\": 5.0}, {\"size\": \"large\", \"price\": 6.0}]\n",
      "Embedding: [-0.033293843269348145, -0.030324464663863182, -0.02518717385828495, 0.018854249268770218, -0.03639461472630501, -0.0049927630461752415, -0.019787106662988663, -0.0359216146171093, -0.024083511903882027, 0.024648481979966164]...\n",
      "\n",
      "ID: hot_chocolates_hot_chocolate_deluxe\n",
      "Category: Hot Chocolates\n",
      "Name: Hot Chocolate Deluxe\n",
      "Description: Rich steamed cream and milk, Dutch chocolate, fresh whipped cream, and shaved dark chocolate\n",
      "Long Description: Hot Chocolate Deluxe is a decadent beverage featuring rich Dutch chocolate steamed with cream and milk. Topped with fresh whipped cream and shaved dark chocolate, it's a comforting indulgence.\n",
      "Origin: Custom House Recipe\n",
      "Caffeine Content: \n",
      "Brewing Method: \n",
      "Popularity: Very High\n",
      "Sizes: [{\"size\": \"small\", \"price\": 3.0}, {\"size\": \"medium\", \"price\": 4.0}, {\"size\": \"large\", \"price\": 5.0}]\n",
      "Embedding: [0.0027125361375510693, -0.0428113155066967, -0.03051362931728363, 0.02594042755663395, -0.029693784192204475, -0.013117530383169651, -0.01798536255955696, -0.03217894211411476, -0.016256000846624374, 0.05067670717835426]...\n",
      "\n",
      "ID: hot_chocolates_intermezzo_hot_chocolate\n",
      "Category: Hot Chocolates\n",
      "Name: Intermezzo Hot Chocolate\n",
      "Description: Rich steamed cream, Dutch chocolate, fresh whipped cream, and the essence of almond\n",
      "Long Description: The Intermezzo Hot Chocolate adds a nutty twist to the classic hot chocolate with the essence of almond. Perfect for those who love a bit of sophistication in their chocolate beverages.\n",
      "Origin: Custom House Recipe\n",
      "Caffeine Content: \n",
      "Brewing Method: \n",
      "Popularity: High\n",
      "Sizes: [{\"size\": \"small\", \"price\": 3.5}, {\"size\": \"medium\", \"price\": 4.5}, {\"size\": \"large\", \"price\": 5.5}]\n",
      "Embedding: [-0.0033937476109713316, -0.025336317718029022, -0.031491778790950775, 0.032068852335214615, -0.019689232110977173, -0.02998039312660694, -0.0008029235759750009, -0.020884601399302483, -0.030392589047551155, 0.029156001284718513]...\n",
      "\n",
      "ID: hot_chocolates_mexican_hot_chocolate\n",
      "Category: Hot Chocolates\n",
      "Name: Mexican Hot Chocolate\n",
      "Description: Semi-sweet dark chocolate, Mexican vanilla, milk, and cinnamon\n",
      "Long Description: Mexican Hot Chocolate blends semi-sweet dark chocolate with Mexican vanilla and cinnamon, creating a warm, spiced drink that's perfect for cozy evenings.\n",
      "Origin: Mexico\n",
      "Caffeine Content: \n",
      "Brewing Method: \n",
      "Popularity: Cultural Favorite\n",
      "Sizes: [{\"size\": \"small\", \"price\": 3.5}, {\"size\": \"medium\", \"price\": 4.5}, {\"size\": \"large\", \"price\": 5.5}]\n",
      "Embedding: [0.002594430698081851, -0.03536994382739067, -0.024135345593094826, 0.02570628561079502, -0.02668217197060585, 0.015340464189648628, -0.009312578476965427, -0.05379278212785721, -0.014078952372074127, 0.03748833388090134]...\n",
      "\n",
      "ID: extras_extra_shot\n",
      "Category: Extras\n",
      "Name: Extra Shot\n",
      "Description: Add an extra shot of espresso\n",
      "Long Description: Boost your beverage with an extra shot of espresso, adding richness and a caffeine kick to your favorite drink.\n",
      "Origin: \n",
      "Caffeine Content: 63 mg\n",
      "Brewing Method: \n",
      "Popularity: \n",
      "Sizes: [{\"size\": \"standard\", \"price\": 1.0}]\n",
      "Embedding: [0.018976276740431786, -0.01118418574333191, -0.016487812623381615, 0.006050860043615103, -0.0022973110899329185, -0.010266652330756187, 0.003150548320263624, 0.00807708129286766, -0.029444506391882896, 0.011893189512193203]...\n",
      "\n",
      "ID: extras_flavor_shot\n",
      "Category: Extras\n",
      "Name: Flavor Shot\n",
      "Description: Add vanilla, caramel, or hazelnut\n",
      "Long Description: Customize your coffee with a flavor shot. Choose from vanilla, caramel, or hazelnut to enhance the taste of your drink.\n",
      "Origin: \n",
      "Caffeine Content: \n",
      "Brewing Method: \n",
      "Popularity: \n",
      "Sizes: [{\"size\": \"standard\", \"price\": 0.75}]\n",
      "Embedding: [-0.0009499499574303627, -0.034819550812244415, -0.02148440293967724, 0.030780769884586334, -0.03883443400263786, 0.00916492659598589, -0.008011842146515846, -0.012940827757120132, -0.025332000106573105, -0.002134400187060237]...\n",
      "\n",
      "ID: extras_whipped_cream\n",
      "Category: Extras\n",
      "Name: Whipped Cream\n",
      "Description: Add whipped cream topping\n",
      "Long Description: Top your coffee or hot chocolate with our fresh whipped cream, made from heavy cream, sugar, and a hint of vanilla for added indulgence.\n",
      "Origin: \n",
      "Caffeine Content: \n",
      "Brewing Method: \n",
      "Popularity: \n",
      "Sizes: [{\"size\": \"standard\", \"price\": 0.5}]\n",
      "Embedding: [-0.024600524455308914, -0.03466380387544632, -0.021454965695738792, 0.025690818205475807, -0.011272629722952843, -0.0012438115663826466, 0.005592452362179756, -0.03942600637674332, -0.006281718146055937, 0.03336046636104584]...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def sanitize_key(key):\n",
    "    \"\"\"Sanitize the document key to contain only valid characters.\"\"\"\n",
    "    return re.sub(r'[^a-zA-Z0-9_\\-]', '_', key)\n",
    "\n",
    "@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=5, max=60))\n",
    "def generate_embeddings(texts):\n",
    "    \"\"\"\n",
    "    Generate embeddings using Azure OpenAI with retry logic for a batch of texts.\n",
    "    \"\"\"\n",
    "    response = aoai_client.embeddings.create(input=texts, model=aoai_embedding_deployment)\n",
    "    return [res.embedding for res in response.data]\n",
    "\n",
    "def prepare_data_for_azure_search(menu_items_data):\n",
    "    \"\"\"Transform parsed data for ingestion into Azure AI Search.\"\"\"\n",
    "    azure_search_documents = []\n",
    "    menu_items = menu_items_data[\"menuItems\"]  # Extract the list of categories\n",
    "    texts_for_embedding = []\n",
    "    document_keys = []\n",
    "\n",
    "    for category in menu_items:\n",
    "        for item in category[\"items\"]:\n",
    "            # Combine relevant fields for embedding\n",
    "            combined_text = f\"{category['category']} {item['name']} {item['description']} {item.get('longDescription', '')}\"\n",
    "            document_key = sanitize_key(f\"{category['category']}_{item['name'].replace(' ', '_')}\".lower())\n",
    "            \n",
    "            # Collect texts and keys for batch embedding\n",
    "            texts_for_embedding.append(combined_text)\n",
    "            document_keys.append(document_key)\n",
    "            \n",
    "            # Add the document to Azure Search format without embedding\n",
    "            azure_search_documents.append({\n",
    "                \"id\": document_key,  # Use sanitized document key\n",
    "                \"category\": category[\"category\"],\n",
    "                \"name\": item[\"name\"],\n",
    "                \"description\": item[\"description\"],\n",
    "                \"longDescription\": item.get(\"longDescription\", \"\"),  # Include long description if available\n",
    "                \"origin\": item.get(\"origin\", \"\"),  # Include origin if available\n",
    "                \"caffeineContent\": item.get(\"caffeineContent\", \"\"),  # Include caffeine content if available\n",
    "                \"brewingMethod\": item.get(\"brewingMethod\", \"\"),  # Include brewing method if available\n",
    "                \"popularity\": item.get(\"popularity\", \"\"),  # Include popularity if available\n",
    "                \"sizes\": json.dumps(item[\"sizes\"]),  # Convert sizes to JSON string\n",
    "            })\n",
    "\n",
    "    # Generate embeddings in batch\n",
    "    embeddings = generate_embeddings(texts_for_embedding)\n",
    "\n",
    "    # Assign embeddings to documents\n",
    "    for i, embedding in enumerate(embeddings):\n",
    "        azure_search_documents[i][\"embedding\"] = embedding\n",
    "\n",
    "    return azure_search_documents\n",
    "\n",
    "# Example usage\n",
    "documents_for_index = prepare_data_for_azure_search(structured_menu_items)\n",
    "\n",
    "for doc in documents_for_index:\n",
    "    print(f\"ID: {doc['id']}\")\n",
    "    print(f\"Category: {doc['category']}\")\n",
    "    print(f\"Name: {doc['name']}\")\n",
    "    print(f\"Description: {doc['description']}\")\n",
    "    print(f\"Long Description: {doc['longDescription']}\")\n",
    "    print(f\"Origin: {doc['origin']}\")\n",
    "    print(f\"Caffeine Content: {doc['caffeineContent']}\")\n",
    "    print(f\"Brewing Method: {doc['brewingMethod']}\")\n",
    "    print(f\"Popularity: {doc['popularity']}\")\n",
    "    print(f\"Sizes: {doc['sizes']}\")\n",
    "    print(f\"Embedding: {doc['embedding'][:10]}...\")  # Print the first 10 dimensions for brevity\n",
    "    print()  # Add a blank line between documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8: Upload to Azure AI Search\n",
    "\n",
    "### Description\n",
    "This cell defines and calls a function to upload the prepared data to Azure AI Search.\n",
    "Ensure the Azure AI Search index is properly configured before running this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploaded batch 1/2 successfully. Batch size: 15\n",
      "Uploaded batch 2/2 successfully. Batch size: 2\n",
      "Embedding index created and documents uploaded successfully. Total successful uploads: 17/17\n"
     ]
    }
   ],
   "source": [
    "def upload_documents_to_search(documents):\n",
    "    batch_size = 15\n",
    "    total_batches = (len(documents) + batch_size - 1) // batch_size  # Calculate total number of batches\n",
    "    successful_uploads = 0\n",
    "\n",
    "    for i in range(0, len(documents), batch_size):\n",
    "        batch = documents[i:i + batch_size]\n",
    "        try:\n",
    "            # Upload the batch\n",
    "            response = search_client.upload_documents(documents=batch)\n",
    "            successful_uploads += len(batch)\n",
    "            print(f\"Uploaded batch {i // batch_size + 1}/{total_batches} successfully. Batch size: {len(batch)}\")\n",
    "        except HttpResponseError as e:\n",
    "            print(f\"Error uploading batch {i // batch_size + 1}/{total_batches}: {e}\")\n",
    "            # Log the problematic batch for further inspection\n",
    "            # print(f\"Problematic batch: {batch}\")\n",
    "            continue\n",
    "\n",
    "    print(f\"Embedding index created and documents uploaded successfully. Total successful uploads: {successful_uploads}/{len(documents)}\")\n",
    "\n",
    "upload_documents_to_search(documents_for_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9: Final Summary\n",
    "\n",
    "### Summary\n",
    "- Installed required packages.\n",
    "- Configured Azure OpenAI and Azure AI Search services.\n",
    "- Defined and created/updated the index schema with semantic configuration.\n",
    "- Loaded and processed menu data from `menuItems.json`.\n",
    "- Prepared data for ingestion, including generating embeddings using Azure OpenAI.\n",
    "- Uploaded the structured data into Azure AI Search.\n",
    "\n",
    "The pipeline is now complete! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}