demos/deepeval_rag_eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![deepeval_image](images/deepeval.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAG Evaluations Demonstration: DeepEval\n",
    "\n",
    "DeepEval is an open-source evaluation framework for LLMs. DeepEval makes it extremely easy to build and iterate on LLM (applications) and was built with the following principles in mind:\n",
    "\n",
    "- Easily \"unit test\" LLM outputs in a similar way to Pytest.\n",
    "- Plug-and-use 14+ LLM-evaluated metrics, most with research backing.\n",
    "- Synthetic dataset generation with state-of-the-art evolution techniques.\n",
    "- Metrics are simple to customize and covers all use cases.\n",
    "- Real-time evaluations in production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Evaluation Dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Repo card metadata block was not found. Setting CardData to empty.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'question': 'Which private companies in the Americas are the largest GHG emitters according to the Carbon Majors database?',\n",
       " 'ground_truth': 'The largest private companies in the Americas that are the largest GHG emitters according to the Carbon Majors database are ExxonMobil, Chevron, and Peabody.',\n",
       " 'answer': 'According to the Carbon Majors database, the largest private companies in the Americas that are the largest GHG emitters are:\\n\\n1. Chevron Corporation (United States)\\n2. ExxonMobil Corporation (United States)\\n3. ConocoPhillips Company (United States)\\n4. BP plc (United Kingdom, but with significant operations in the Americas)\\n5. Royal Dutch Shell plc (Netherlands, but with significant operations in the Americas)\\n6. Peabody Energy Corporation (United States)\\n7. Duke Energy Corporation (United States)\\n8. TotalEnergies SE (France, but with significant operations in the Americas)\\n9. BHP Group Limited (Australia, but with significant operations in the Americas)\\n10. Rio Tinto Group (United Kingdom/Australia, but with significant operations in the Americas)\\n\\nPlease note that the rankings may change over time as new data becomes available.',\n",
       " 'contexts': ['The issue of greenhouse gas emissions has become a major concern for environmentalists and policymakers alike, as the impact of climate change becomes more apparent. Private companies in the Americas play a significant role in contributing to these emissions, with some being identified as the largest emitters in the region according to the Carbon Majors database.',\n",
       "  'Reducing greenhouse gas emissions from private companies is a complex challenge that requires cooperation between governments, businesses, and consumers. Many companies are now taking steps to reduce their carbon footprint through initiatives such as investing in renewable energy, improving energy efficiency, and offsetting emissions through carbon credits.',\n",
       "  'The private companies responsible for the most emissions during this period, according to the database, are from the United States: ExxonMobil, Chevron and Peabody.\\nThe largest emitter amongst state-owned companies in the Americas is Mexican company Pemex, followed by Venezuelan company Petróleos de Venezuela, S.A.']}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download amnesty_qa dataset\n",
    "from datasets import load_dataset\n",
    "\n",
    "amnesty_qa = load_dataset(\"explodinggradients/amnesty_qa\", \"english_v2\", trust_remote_code=True)\n",
    "eval_data = amnesty_qa['eval']\n",
    "eval_data[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build a local LLM-as-a-judge directly from HuggingFace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/johnalling-desktop/education/RAG_eval_tests/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b1300bdb8094f988c40dfd04d2c753c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/johnalling-desktop/education/RAG_eval_tests/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "# Create a using a custom model from huggingface, create an llm-as-a-judge\n",
    "from llama3_deepeval import Llama3_8B\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "model_str = \"solidrust/Meta-Llama-3-8B-Instruct-hf-AWQ\"\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_str, device_map=\"auto\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_str, device_map=\"auto\")\n",
    "\n",
    "llama_3 = Llama3_8B(model=model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sky appears blue because of a phenomenon called Rayleigh scattering, named after the British physicist Lord Rayleigh. He discovered that shorter wavelengths of light, such as blue and violet, are scattered more than longer wavelengths, like red and orange, when they interact with tiny molecules of gases in the atmosphere.\n",
      "\n",
      "Here's what happens:\n",
      "\n",
      "1. When sunlight enters Earth's atmosphere, it encounters tiny molecules of gases like nitrogen (N2) and oxygen (O2).\n",
      "2. These molecules scatter the light in all directions, but they scatter shorter wavelengths (like blue and violet) more than longer wavelengths (like red and orange).\n",
      "3. As a result, the blue and violet light is dispersed throughout the atmosphere, reaching our eyes from all directions.\n",
      "4. Our brains perceive this scattered blue light as the color of the sky, making it appear blue during the daytime.\n",
      "\n",
      "The color of the sky can vary depending on several factors, such as:\n",
      "\n",
      "* Time of day: The sky can take on hues of red, orange, or pink during sunrise and sunset due to the scattering of longer wavelengths.\n",
      "* Atmospheric conditions: Dust, pollution, and water vapor in the air can scatter light in different ways, changing the apparent color of the sky.\n",
      "* Altitude and atmospheric pressure: The color of the sky can change at higher elevations or in areas with different atmospheric pressure.\n",
      "\n",
      "So, to summarize, the sky appears blue because of the scattering of shorter wavelengths of light by tiny molecules in the atmosphere!\n"
     ]
    }
   ],
   "source": [
    "# test basic prompting of the local llm\n",
    "gen_output = llama_3.generate(\"Why is the sky blue?\")\n",
    "\n",
    "print(gen_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create DeepEval Test Cases\n",
    "\n",
    "DeepEval uses a `LLMTestCase` class to handle evaluations. This class has built-in fields that closely match what we've already seen:\n",
    "- input\n",
    "- expected_output\n",
    "- actual_output\n",
    "- retrieval_context\n",
    "\n",
    "Plus a field for additional metadata that can be utilized for *custom* metrics\n",
    "\n",
    "Here we define a function that translates our dataset into a list of test cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create deepeval evaluation dataset from the downloaded dataset\n",
    "from deepeval.test_case import LLMTestCase\n",
    "from random import uniform\n",
    "\n",
    "def test_case_from_data(data_point):\n",
    "    test_case = LLMTestCase(\n",
    "        input = data_point['question'],\n",
    "        actual_output = data_point['answer'],\n",
    "        expected_output = data_point['ground_truth'],\n",
    "        retrieval_context = data_point['contexts'],\n",
    "        additional_metadata = {'latency': uniform(0,20)}\n",
    "    )\n",
    "    return test_case\n",
    "\n",
    "\n",
    "test_cases = [test_case_from_data(data_point) for data_point in eval_data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import the Evaluation Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create metrics to measure from deepeval\n",
    "from deepeval.metrics import (\n",
    "    ContextualPrecisionMetric,\n",
    "    ContextualRecallMetric,\n",
    "    AnswerRelevancyMetric\n",
    ")\n",
    "\n",
    "# Evaluate whether nodes in retrieval_context that are relevant to the given input are ranked higher than irrelevant ones.\n",
    "contextual_precision = ContextualPrecisionMetric(model=llama_3, threshold=0.5)\n",
    "\n",
    "# Evaluate the quality of the retriever by evaluating the extent of which the retrieval_context aligns with the expected_output\n",
    "contextual_recall = ContextualRecallMetric(model=llama_3, threshold=0.5)\n",
    "\n",
    "# Evaluate how relevant the actual_output is to the provided input\n",
    "answer_relevancy = AnswerRelevancyMetric(model=llama_3, threshold=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make our own custom metrics as well!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from custom_deepeval_metric import LatencyMetric\n",
    "\n",
    "# Evaluate the latency of the test_case run\n",
    "latency = LatencyMetric(max_seconds=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Evaluation\n",
    "\n",
    "First, some housekeeping functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retry_five_times(func, *args, **kwargs):\n",
    "    for ii in range(5):\n",
    "        try:\n",
    "            func(*args, **kwargs)\n",
    "            # If function succeeds, break the loop\n",
    "            break\n",
    "        except Exception as e:\n",
    "            if ii == 4:\n",
    "                print(f\"Function failed all attempts, returning null\")\n",
    "        \n",
    "            print(f\"Function call failed with error: {e}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the evaluation of a single test case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41e56d160aa043009debd0c37e094f53",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "31df269208e44eb4891463eeda32b5d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Function call failed with error: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Contextual Precision Score:  1.0\n",
      "Reason:  The score is 1.00 because the retrieval contexts are perfectly aligned with the input, with the most relevant nodes ranked higher and the irrelevant nodes correctly placed lower. The first two nodes provide direct and specific information about the largest private companies in the Americas that are the largest GHG emitters according to the Carbon Majors database, while the third node is a general statement about the issue of greenhouse gas emissions, making it less relevant to the question.\n"
     ]
    }
   ],
   "source": [
    "# evaluate one at a time\n",
    "retry_five_times(contextual_precision.measure, test_cases[2])\n",
    "print(\"Contextual Precision Score: \", contextual_precision.score)\n",
    "print(\"Reason: \", contextual_precision.reason)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the evaluation of multiple test cases on multiple metrics\n",
    "\n",
    "**Note: This is not the fastest way to run this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def evaluation_grid(metrics, test_cases):\n",
    "    df = pd.DataFrame()\n",
    "    df[\"Questions\"] = [test_case.input for test_case in test_cases]\n",
    "    df[\"Answers\"] = [test_case.actual_output for test_case in test_cases]\n",
    "    for metric in metrics:\n",
    "        metric_scores = []\n",
    "        metric_reasons = []\n",
    "        for test_case in test_cases:\n",
    "            retry_five_times(metric.measure,test_case)\n",
    "            metric_scores.append(metric.score)\n",
    "            metric_reasons.append(metric.reason)\n",
    "        df[metric.__name__+\" Score\"] = metric_scores\n",
    "        df[metric.__name__+\" Reason\"] = metric_reasons\n",
    "\n",
    "    return df\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f267cee9b6964967a3e528638aa370fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6927b532f8414d7e90b5c6e23c396168",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54aa9a81184c4181865b260e26904fb0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2216599c82444602907a388ba93a7e06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f534c7db8ea94a5fac9ce86ee0fdb77e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "265057b7db5642be9d50f17092a20f3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a72371650bf4a978831c4feaf50a509",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4792930288f04ffdb56f4f68ae2684d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6710be962274d75bc1dafe24b5a1110",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Event loop is already running. Applying nest_asyncio patch to allow async execution...\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()\n",
    "\n",
    "evaluation_data = evaluation_grid([contextual_recall,contextual_precision,answer_relevancy,latency], test_cases[2:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Questions</th>\n",
       "      <th>Answers</th>\n",
       "      <th>Contextual Recall Score</th>\n",
       "      <th>Contextual Recall Reason</th>\n",
       "      <th>Contextual Precision Score</th>\n",
       "      <th>Contextual Precision Reason</th>\n",
       "      <th>Answer Relevancy Score</th>\n",
       "      <th>Answer Relevancy Reason</th>\n",
       "      <th>Latency Score</th>\n",
       "      <th>Latency Reason</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Which private companies in the Americas are th...</td>\n",
       "      <td>According to the Carbon Majors database, the l...</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>The score is 0.67 because the model partially ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The score is 1.00 because all the relevant nod...</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>The score is 0.91 because the output is mostly...</td>\n",
       "      <td>1</td>\n",
       "      <td>Latency was below the acceptable limit of 1 se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What action did Amnesty International urge its...</td>\n",
       "      <td>Amnesty International urged its supporters to ...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>The score is 1.00 because the expected output ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The score is 1.00 because all the relevant nod...</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>The score is 0.75 because, although the output...</td>\n",
       "      <td>1</td>\n",
       "      <td>Latency was below the acceptable limit of 1 se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What are the recommendations made by Amnesty I...</td>\n",
       "      <td>Amnesty International made several recommendat...</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>The score is 0.80 because the output accuratel...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The score is 1.00 because all relevant nodes i...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>The score is 1.00 because the actual output di...</td>\n",
       "      <td>1</td>\n",
       "      <td>Latency was below the acceptable limit of 1 se...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Questions  \\\n",
       "0  Which private companies in the Americas are th...   \n",
       "1  What action did Amnesty International urge its...   \n",
       "2  What are the recommendations made by Amnesty I...   \n",
       "\n",
       "                                             Answers  Contextual Recall Score  \\\n",
       "0  According to the Carbon Majors database, the l...                 0.666667   \n",
       "1  Amnesty International urged its supporters to ...                 1.000000   \n",
       "2  Amnesty International made several recommendat...                 0.800000   \n",
       "\n",
       "                            Contextual Recall Reason  \\\n",
       "0  The score is 0.67 because the model partially ...   \n",
       "1  The score is 1.00 because the expected output ...   \n",
       "2  The score is 0.80 because the output accuratel...   \n",
       "\n",
       "   Contextual Precision Score  \\\n",
       "0                         1.0   \n",
       "1                         1.0   \n",
       "2                         1.0   \n",
       "\n",
       "                         Contextual Precision Reason  Answer Relevancy Score  \\\n",
       "0  The score is 1.00 because all the relevant nod...                0.909091   \n",
       "1  The score is 1.00 because all the relevant nod...                0.750000   \n",
       "2  The score is 1.00 because all relevant nodes i...                1.000000   \n",
       "\n",
       "                             Answer Relevancy Reason  Latency Score  \\\n",
       "0  The score is 0.91 because the output is mostly...              1   \n",
       "1  The score is 0.75 because, although the output...              1   \n",
       "2  The score is 1.00 because the actual output di...              1   \n",
       "\n",
       "                                      Latency Reason  \n",
       "0  Latency was below the acceptable limit of 1 se...  \n",
       "1  Latency was below the acceptable limit of 1 se...  \n",
       "2  Latency was below the acceptable limit of 1 se...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It should be noted that you can also create an evaluation dataset of all your test cases and evaluate them at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepeval.dataset import EvaluationDataset\n",
    "\n",
    "deepeval_ds = EvaluationDataset(test_cases=test_cases[2:5])\n",
    "\n",
    "deepeval_ds.evaluate(metrics=[contextual_recall,contextual_precision,answer_relevancy,latency])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running Tests with DeepEval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7453a1633eff4b61a7a92f153dfd2d3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Event loop is already running. Applying nest_asyncio patch to allow async execution...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test passed!\n"
     ]
    }
   ],
   "source": [
    "from deepeval import assert_test\n",
    "\n",
    "try:\n",
    "    assert_test(test_cases[4], metrics=[latency])\n",
    "    print(\"Test passed!\")\n",
    "except Exception as e:\n",
    "    print('Error found: {}'.format(e))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DeepEval also offers integration with the ConfidentAI app, offering a centralized place to log evals, change hyperparameters, debug via LLM traces, and monitor in production. However, this doesnt appear to be open-source or able to be self-hosted and wouldn't be available in an Air-gapped scenario most likely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![confident_ai](images/confident_ai.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DeepEval Pros\n",
    "- Lots of built-in metrics (includes everything provided by RAGAS)\n",
    "- Lots of customization (handles LLM and non-LLM evals)\n",
    "- Applicable to RAG and non-RAG specific evals\n",
    "- Metrics supply reasoning (more transparency than just a score)\n",
    "- Easy to customize evaluator LLMs\n",
    "- Test set generation capabilities\n",
    "\n",
    "#### DeepEval Cons\n",
    "- Slower than RAGAS (getting reasonings takes time)\n",
    "- Requires a really good evaluator LLM\n",
    "- If run as a batch (using EvaluationDataset for example), if one eval errors, they all fail\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}