adaptive_rag.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Adaptive RAG**\n",
        "Adaptive Retrieval-Augmented Generation (RAG) is a smart approach that adjusts how information is retrieved and used based on the complexity of a query. Instead of treating all queries the same way, Adaptive RAG chooses the best strategy for each query type.\n",
        "\n",
        "In our case, system analyzes the query and decides the best way to handle it:\n",
        "\n",
        "**Self-corrective RAG:** For questions that can be answered using the indexed data.\n",
        "\n",
        "**Web Search:** For questions that require information not present in the index.\n",
        "\n",
        "Research Paper: [Adaptive RAG](https://arxiv.org/pdf/2403.14403)"
      ],
      "metadata": {
        "id": "yBG5q1VlPYMk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Initial Setup**"
      ],
      "metadata": {
        "id": "_N--dPv3Qz3L"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UEwv4s_ApMPM"
      },
      "outputs": [],
      "source": [
        "! pip install --q athina faiss-gpu langgraph"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# set API keys\n",
        "import os\n",
        "from google.colab import userdata\n",
        "os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')\n",
        "os.environ[\"TAVILY_API_KEY\"] = userdata.get('TAVILY_API_KEY')\n",
        "os.environ['ATHINA_API_KEY'] = userdata.get('ATHINA_API_KEY')"
      ],
      "metadata": {
        "id": "cyMBUpkzwII-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Indexing**"
      ],
      "metadata": {
        "id": "7VB05ui3SeoQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load embedding model\n",
        "from langchain_openai import OpenAIEmbeddings\n",
        "embeddings = OpenAIEmbeddings()"
      ],
      "metadata": {
        "id": "rpfSXcV8wW6F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load data\n",
        "from langchain.document_loaders import CSVLoader\n",
        "loader = CSVLoader(\"./context.csv\")\n",
        "documents = loader.load()"
      ],
      "metadata": {
        "id": "W1z59llHwYwn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# split documents\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
        "documents = text_splitter.split_documents(documents)"
      ],
      "metadata": {
        "id": "nxhl7_K2wed0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vectorstore\n",
        "from langchain.vectorstores import FAISS\n",
        "vectorstore = FAISS.from_documents(documents, embeddings)"
      ],
      "metadata": {
        "id": "Nx3euIviwgxs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Retriever**"
      ],
      "metadata": {
        "id": "jH3CkEuWTNhn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create retriever\n",
        "retriever = vectorstore.as_retriever()"
      ],
      "metadata": {
        "id": "wXxm05JBwi3v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question Router**\n",
        "The question router determines whether a query should be answered using our internal index or through a web search.\n",
        "\n",
        "Please change the prompt based on your topics"
      ],
      "metadata": {
        "id": "Ar5EApjmS_7i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create question router\n",
        "\n",
        "from typing import Literal\n",
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "from langchain_core.pydantic_v1 import BaseModel, Field\n",
        "from langchain_openai import ChatOpenAI\n",
        "\n",
        "\n",
        "# define a data class\n",
        "class RouteQuery(BaseModel):\n",
        "    \"\"\"Route a user query to the most relevant datasource.\"\"\"\n",
        "\n",
        "    datasource: Literal[\"vectorstore\", \"web_search\"] = Field(\n",
        "        ...,\n",
        "        description=\"Given a user question choose to route it to web search or a vectorstore.\",\n",
        "    )\n",
        "\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "structured_llm_router = llm.with_structured_output(RouteQuery)\n",
        "\n",
        "# Prompt (add only topics that are present in you vectorstore)\n",
        "system = \"\"\"You are an expert at routing a user question to either a vectorstore or web search.\n",
        "The vectorstore contains information on the following topics:\n",
        "- Finance and real estate\n",
        "- Library and research topics\n",
        "- Biology and microbiology\n",
        "- Literature and writing\n",
        "- Movies and entertainment\n",
        "- Animals and nature\n",
        "- History and geography\n",
        "- Astronomy\n",
        "\n",
        "If the question is related to these topics, route it to the vectorstore. Otherwise, use web search.\"\"\"\n",
        "route_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"{question}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "question_router = route_prompt | structured_llm_router"
      ],
      "metadata": {
        "id": "6b-3iM8zw3Sh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "question_router.invoke({\"question\": \"how does interlibrary loan work?\"})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4Oj50WltygLX",
        "outputId": "c649ba1a-c22b-46e7-8cbb-4e09ae5e4c4d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RouteQuery(datasource='vectorstore')"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "question_router.invoke({\"question\": \"What is RAG?\"})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IyEhloce0I1e",
        "outputId": "d8f0d73d-635b-4633-d45c-676c4061d2f2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RouteQuery(datasource='web_search')"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Document Grader**\n",
        "The document grader evaluates whether a document is relevant to the given query."
      ],
      "metadata": {
        "id": "0FXBmV9pTkoP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for doc retriever\n",
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "from langchain_core.pydantic_v1 import BaseModel, Field\n",
        "from langchain_openai import ChatOpenAI\n",
        "\n",
        "# define a data class\n",
        "class GradeDocuments(BaseModel):\n",
        "    binary_score: str = Field(\n",
        "        description=\"Documents are relevant to the question, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeDocuments)\n",
        "\n",
        "# Prompt for the grader\n",
        "system = \"\"\"You are a grader assessing relevance of a retrieved document to a user question. \\n\n",
        "    If the document contains keyword(s) or semantic meaning related to the question, grade it as relevant. \\n\n",
        "    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.\"\"\"\n",
        "grade_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"Retrieved document: \\n\\n {document} \\n\\n User question: {question}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "retrieval_grader = grade_prompt | structured_llm_grader"
      ],
      "metadata": {
        "id": "RxCdVNUQ0ZqB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# testing grader example 1\n",
        "question = \"how does interlibrary loan work\"\n",
        "docs = retriever.get_relevant_documents(question)\n",
        "print(retrieval_grader.invoke({\"question\": question, \"document\": docs}))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3T6eCGDZ0cCK",
        "outputId": "7430a2c6-d6f2-4e26-ed8a-598f9911f003"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The method `BaseRetriever.get_relevant_documents` was deprecated in langchain-core 0.1.46 and will be removed in 0.3.0. Use invoke instead.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "binary_score='yes'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# # testing grader example 2\n",
        "# question = \"What is RAG?\"\n",
        "# docs = retriever.get_relevant_documents(question)\n",
        "# print(retrieval_grader.invoke({\"question\": question, \"document\": docs}))"
      ],
      "metadata": {
        "id": "IvAKg34jUobu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **RAG Chain**"
      ],
      "metadata": {
        "id": "qL-MGdNGVBRV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create document chain\n",
        "from langchain import hub\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "from langchain.prompts import ChatPromptTemplate\n",
        "\n",
        "template = \"\"\"\"\n",
        "You are a helpful assistant that answers questions based on the following context.\n",
        "Use the provided context to answer the question.\n",
        "Context: {context}\n",
        "Question: {question}\n",
        "Answer:\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "\n",
        "def format_docs(docs):\n",
        "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
        "\n",
        "\n",
        "rag_chain = prompt | llm | StrOutputParser()"
      ],
      "metadata": {
        "id": "Z7ensSY41KNY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# response\n",
        "generation = rag_chain.invoke({\"context\": docs, \"question\": question})\n",
        "generation"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "id": "CelkRXJj1Mjx",
        "outputId": "135037a1-9238-413a-d8f2-e9992fd21dd3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Interlibrary loan works by allowing patrons of one library to borrow physical materials or receive electronic documents from another library that holds the desired item. The borrowing library identifies potential lending libraries, which then deliver the item either physically or electronically. The borrowing library receives the item, delivers it to their patron, and arranges for its return if necessary. Fees may accompany interlibrary loan services, and requests can be managed through semi-automated or manual systems. Libraries negotiate for interlibrary loan eligibility, and with the increasing demand for digital materials, libraries are exploring the legal, technical, and licensing aspects of lending and borrowing ebooks through interlibrary loan.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Hallucination Grader**\n",
        "The hallucination grader checks whether the answer is grounded in or supported by the given set of facts"
      ],
      "metadata": {
        "id": "RauainozVsks"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for hallucination\n",
        "# define a data class\n",
        "class GradeHallucinations(BaseModel):\n",
        "    \"\"\"Binary score for hallucination present in generation answer.\"\"\"\n",
        "\n",
        "    binary_score: str = Field(\n",
        "        description=\"Answer is grounded in the facts, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeHallucinations)\n",
        "\n",
        "# prompt for the grader\n",
        "system = \"\"\"You are a grader assessing whether an LLM generation is grounded in / supported by a set of retrieved facts. \\n\n",
        "     Give a binary score 'yes' or 'no'. 'Yes' means that the answer is grounded in / supported by the set of facts.\"\"\"\n",
        "hallucination_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"Set of facts: \\n\\n {documents} \\n\\n LLM generation: {generation}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "hallucination_grader = hallucination_prompt | structured_llm_grader\n",
        "hallucination_grader.invoke({\"documents\": docs, \"generation\": generation})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IQooXxa21QS7",
        "outputId": "b089e4f8-04c1-43de-8bfa-0469cbf14e42"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GradeHallucinations(binary_score='yes')"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Answer Grader**\n",
        "The answer grader evaluates whether an answer effectively addresses the given question."
      ],
      "metadata": {
        "id": "SGrdVm7PWpo3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for answer\n",
        "# define a data class\n",
        "class GradeAnswer(BaseModel):\n",
        "    \"\"\"Binary score to assess answer addresses question.\"\"\"\n",
        "\n",
        "    binary_score: str = Field(\n",
        "        description=\"Answer addresses the question, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeAnswer)\n",
        "\n",
        "# prompt for the grader\n",
        "system = \"\"\"You are a grader assessing whether an answer addresses / resolves a question \\n\n",
        "     Give a binary score 'yes' or 'no'. Yes' means that the answer resolves the question.\"\"\"\n",
        "answer_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"User question: \\n\\n {question} \\n\\n LLM generation: {generation}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "answer_grader = answer_prompt | structured_llm_grader\n",
        "answer_grader.invoke({\"question\": question, \"generation\": generation})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TE6G2fUr1SqG",
        "outputId": "08954708-1a54-42db-e801-6ba5594c011c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GradeAnswer(binary_score='yes')"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Web Search**"
      ],
      "metadata": {
        "id": "dPZShXJBW7an"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define web search\n",
        "from langchain_community.tools.tavily_search import TavilySearchResults\n",
        "web_search_tool = TavilySearchResults(k=3)"
      ],
      "metadata": {
        "id": "PqmibD231WH8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Create Graph**"
      ],
      "metadata": {
        "id": "jG4ifXapXIsq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Define Graph State**"
      ],
      "metadata": {
        "id": "6orlhNYWXNwg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define a data class for state\n",
        "from typing import List\n",
        "from typing_extensions import TypedDict\n",
        "\n",
        "class GraphState(TypedDict):\n",
        "    question: str\n",
        "    generation: str\n",
        "    documents: List[str]"
      ],
      "metadata": {
        "id": "jGi0YHC21eMs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# define graph steps\n",
        "from langchain.schema import Document\n",
        "\n",
        "# all nodes\n",
        "def retrieve(state):\n",
        "\n",
        "    print(\"---RETRIEVE---\")\n",
        "    question = state[\"question\"]\n",
        "\n",
        "    # Retrieval\n",
        "    documents = retriever.invoke(question)\n",
        "    return {\"documents\": documents, \"question\": question}\n",
        "\n",
        "\n",
        "def generate(state):\n",
        "\n",
        "    print(\"---GENERATE---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # RAG generation\n",
        "    generation = rag_chain.invoke({\"context\": documents, \"question\": question})\n",
        "    return {\"documents\": documents, \"question\": question, \"generation\": generation}\n",
        "\n",
        "\n",
        "def grade_documents(state):\n",
        "\n",
        "    print(\"---CHECK DOCUMENT RELEVANCE TO QUESTION---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # Score each doc\n",
        "    filtered_docs = []\n",
        "    for d in documents:\n",
        "        score = retrieval_grader.invoke(\n",
        "            {\"question\": question, \"document\": d.page_content}\n",
        "        )\n",
        "        grade = score.binary_score\n",
        "        if grade == \"yes\":\n",
        "            print(\"---GRADE: DOCUMENT RELEVANT---\")\n",
        "            filtered_docs.append(d)\n",
        "        else:\n",
        "            print(\"---GRADE: DOCUMENT NOT RELEVANT---\")\n",
        "            continue\n",
        "    return {\"documents\": filtered_docs, \"question\": question}\n",
        "\n",
        "\n",
        "def web_search(state):\n",
        "\n",
        "    print(\"---WEB SEARCH---\")\n",
        "    question = state[\"question\"]\n",
        "\n",
        "    # Web search\n",
        "    docs = web_search_tool.invoke({\"query\": question})\n",
        "    web_results = \"\\n\".join([d[\"content\"] for d in docs])\n",
        "    web_results = Document(page_content=web_results)\n",
        "\n",
        "    return {\"documents\": web_results, \"question\": question}\n",
        "\n",
        "\n",
        "# edges\n",
        "def route_question(state):\n",
        "\n",
        "    print(\"---ROUTE QUESTION---\")\n",
        "    question = state[\"question\"]\n",
        "    source = question_router.invoke({\"question\": question})\n",
        "    if source.datasource == \"web_search\":\n",
        "        print(\"---ROUTE QUESTION TO WEB SEARCH---\")\n",
        "        return \"web_search\"\n",
        "    elif source.datasource == \"vectorstore\":\n",
        "        print(\"---ROUTE QUESTION TO RAG---\")\n",
        "        return \"vectorstore\"\n",
        "\n",
        "\n",
        "def decide_to_generate(state):\n",
        "\n",
        "    print(\"---ASSESS GRADED DOCUMENTS---\")\n",
        "    state[\"question\"]\n",
        "    filtered_documents = state[\"documents\"]\n",
        "\n",
        "    if not filtered_documents:\n",
        "        # All documents have been filtered check_relevance\n",
        "        # We will re-generate a new query\n",
        "        print(\n",
        "            \"---DECISION: ALL DOCUMENTS ARE NOT RELEVANT TO QUESTION, TRANSFORM QUERY---\"\n",
        "        )\n",
        "        return \"transform_query\"\n",
        "    else:\n",
        "        # We have relevant documents, so generate answer\n",
        "        print(\"---DECISION: GENERATE---\")\n",
        "        return \"generate\"\n",
        "\n",
        "\n",
        "def grade_generation_v_documents_and_question(state):\n",
        "\n",
        "    print(\"---CHECK HALLUCINATIONS---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "    generation = state[\"generation\"]\n",
        "\n",
        "    score = hallucination_grader.invoke(\n",
        "        {\"documents\": documents, \"generation\": generation}\n",
        "    )\n",
        "    grade = score.binary_score\n",
        "\n",
        "    # Check hallucination\n",
        "    if grade == \"yes\":\n",
        "        print(\"---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\")\n",
        "        # Check question-answering\n",
        "        print(\"---GRADE GENERATION vs QUESTION---\")\n",
        "        score = answer_grader.invoke({\"question\": question, \"generation\": generation})\n",
        "        grade = score.binary_score\n",
        "        if grade == \"yes\":\n",
        "            print(\"---DECISION: GENERATION ADDRESSES QUESTION---\")\n",
        "            return \"useful\"\n",
        "        else:\n",
        "            print(\"---DECISION: GENERATION DOES NOT ADDRESS QUESTION---\")\n",
        "            return \"not useful\"\n",
        "    else:\n",
        "        pprint(\"---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---\")\n",
        "        return \"not supported\""
      ],
      "metadata": {
        "id": "-gL4bV451kKC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Build Graph**"
      ],
      "metadata": {
        "id": "J0bLxwTFX7Gx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Build graph\n",
        "from langgraph.graph import END, StateGraph, START\n",
        "\n",
        "workflow = StateGraph(GraphState)\n",
        "\n",
        "# Define the nodes\n",
        "workflow.add_node(\"web_search\", web_search)  # web search\n",
        "workflow.add_node(\"retrieve\", retrieve)  # retrieve\n",
        "workflow.add_node(\"grade_documents\", grade_documents)  # grade documents\n",
        "workflow.add_node(\"generate\", generate)  # generatae\n",
        "  # transform_query\n",
        "\n",
        "# Build graph\n",
        "workflow.add_conditional_edges(\n",
        "    START,\n",
        "    route_question,\n",
        "    {\n",
        "        \"web_search\": \"web_search\",\n",
        "        \"vectorstore\": \"retrieve\",\n",
        "    },\n",
        ")\n",
        "workflow.add_edge(\"web_search\", \"generate\")\n",
        "workflow.add_edge(\"retrieve\", \"grade_documents\")\n",
        "workflow.add_conditional_edges(\n",
        "    \"grade_documents\",\n",
        "    decide_to_generate,\n",
        "    {\n",
        "        \"generate\": \"generate\",\n",
        "    },\n",
        ")\n",
        "\n",
        "workflow.add_conditional_edges(\n",
        "    \"generate\",\n",
        "    grade_generation_v_documents_and_question,\n",
        "    {\n",
        "        \"not supported\": \"generate\",\n",
        "        \"useful\": END\n",
        "    },\n",
        ")\n",
        "\n",
        "# Compile\n",
        "app = workflow.compile()"
      ],
      "metadata": {
        "id": "_PTiM-4f14a3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Final generation example 1 (relevant documents)\n",
        "from pprint import pprint\n",
        "\n",
        "inputs = {\"question\": \"how does interlibrary loan work\"}\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        pprint(f\"Node '{key}':\")\n",
        "    pprint(\"\\n---\\n\")\n",
        "\n",
        "pprint(value[\"generation\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oMOJmPyo3NcY",
        "outputId": "9fac66ee-5a64-4f95-8096-fc7c74b70050"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---ROUTE QUESTION---\n",
            "---ROUTE QUESTION TO RAG---\n",
            "---RETRIEVE---\n",
            "\"Node 'retrieve':\"\n",
            "'\\n---\\n'\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: GENERATE---\n",
            "\"Node 'grade_documents':\"\n",
            "'\\n---\\n'\n",
            "---GENERATE---\n",
            "---CHECK HALLUCINATIONS---\n",
            "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
            "---GRADE GENERATION vs QUESTION---\n",
            "---DECISION: GENERATION ADDRESSES QUESTION---\n",
            "\"Node 'generate':\"\n",
            "'\\n---\\n'\n",
            "('Interlibrary loan works by allowing patrons of one library to borrow '\n",
            " 'physical materials or receive electronic documents from another library that '\n",
            " 'holds the desired item. The borrowing library identifies potential lending '\n",
            " 'libraries, and the lending library delivers the item either physically or '\n",
            " 'electronically. The borrowing library then receives the item, delivers it to '\n",
            " 'their patron, and arranges for its return if necessary. Fees may accompany '\n",
            " 'interlibrary loan services, and requests can be managed through '\n",
            " 'semi-automated or manual systems.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Final generation example 2 (non-relevant documents)\n",
        "from pprint import pprint\n",
        "\n",
        "inputs = {\"question\": \"What is RAG?\"}\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        pprint(f\"Node '{key}':\")\n",
        "    pprint(\"\\n---\\n\")\n",
        "\n",
        "pprint(value[\"generation\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AcGSVcJ04WPz",
        "outputId": "3302154b-f6f9-4cdc-9def-e38219865e7e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---ROUTE QUESTION---\n",
            "---ROUTE QUESTION TO WEB SEARCH---\n",
            "---WEB SEARCH---\n",
            "\"Node 'web_search':\"\n",
            "'\\n---\\n'\n",
            "---GENERATE---\n",
            "---CHECK HALLUCINATIONS---\n",
            "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
            "---GRADE GENERATION vs QUESTION---\n",
            "---DECISION: GENERATION ADDRESSES QUESTION---\n",
            "\"Node 'generate':\"\n",
            "'\\n---\\n'\n",
            "('RAG, or retrieval-augmented generation, is a method that combines retrieval '\n",
            " 'models and generative models in natural language processing (NLP) to produce '\n",
            " 'coherent, contextually relevant text. It allows AI models to access external '\n",
            " 'knowledge bases and incorporate up-to-date information into generated '\n",
            " 'responses. RAG is significant in NLP as it bridges the gap between retrieval '\n",
            " 'and generative models, enhancing the accuracy and relevance of AI-generated '\n",
            " 'content.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Preparing Data for Evaluation**"
      ],
      "metadata": {
        "id": "pf1GVRBqZGrF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a dataframe to store the question, context, and response\n",
        "inputs = {\"question\": \"how does interlibrary loan work\"}\n",
        "outputs = []\n",
        "\n",
        "\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        if key == \"generate\":\n",
        "            question = value[\"question\"]\n",
        "            documents = value[\"documents\"]\n",
        "            generation = value[\"generation\"]\n",
        "\n",
        "            # Combine document\n",
        "            context = \"\\n\".join(doc.page_content for doc in documents)\n",
        "\n",
        "            # Append the result\n",
        "            outputs.append({\n",
        "                \"query\": question,\n",
        "                \"context\": context,\n",
        "                \"response\": generation,\n",
        "            })"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oS0TKWm2ZVvM",
        "outputId": "44d30dfb-9c72-4029-a118-27de60a37538"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---ROUTE QUESTION---\n",
            "---ROUTE QUESTION TO RAG---\n",
            "---RETRIEVE---\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: GENERATE---\n",
            "---GENERATE---\n",
            "---CHECK HALLUCINATIONS---\n",
            "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
            "---GRADE GENERATION vs QUESTION---\n",
            "---DECISION: GENERATION ADDRESSES QUESTION---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to DataFrame\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(outputs)"
      ],
      "metadata": {
        "id": "BpEUW4e-Z3Y7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "id": "GVSbxLf7Z9z1",
        "outputId": "edfbf64e-94a3-4347-b863-613942120781"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                             query  \\\n",
              "0  how does interlibrary loan work   \n",
              "\n",
              "                                             context  \\\n",
              "0  Procedures and methods ==\\n\\nAfter receiving a...   \n",
              "\n",
              "                                            response  \n",
              "0  Interlibrary loan works by allowing patrons of...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f726b48c-4e7c-48cf-8c6c-1bb99e932bc5\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>how does interlibrary loan work</td>\n",
              "      <td>Procedures and methods ==\\n\\nAfter receiving a...</td>\n",
              "      <td>Interlibrary loan works by allowing patrons of...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f726b48c-4e7c-48cf-8c6c-1bb99e932bc5')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f726b48c-4e7c-48cf-8c6c-1bb99e932bc5 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f726b48c-4e7c-48cf-8c6c-1bb99e932bc5');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_3083c885-0228-4a6c-aac3-e4c487758b6f\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_3083c885-0228-4a6c-aac3-e4c487758b6f button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"query\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"how does interlibrary loan work\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Procedures and methods ==\\\\n\\\\nAfter receiving a request from their patron, the borrowing library identifies potential lending libraries with the desired item. The lending library then delivers the item physically or electronically, and the borrowing library receives the item, delivers it to their patron, and if necessary, arranges for its return. In some cases, fees accompany interlibrary loan services. While the majority of interlibrary loan requests are now managed through semi-automated\\ncontext: ['Interlibrary loan (abbreviated ILL, and sometimes called document delivery, document supply, interlending, interlibrary services, interloan, or resource sharing) is a service that enables patrons of one library to borrow physical materials and receive electronic documents that are held by another library. The service expands library patrons\\\\' access to resources beyond their local library\\\\'s holdings, serving as \\\"an integral element of collection development\\\" for libraries.\\\\n\\\\n\\\\n==\\nloan. Many journal or database licenses specify whether a library can or cannot supply journal articles via ILL, with libraries negotiating for ILL eligibility. With increasing demand for digital materials, libraries have begun exploring the legal, technical, and licensing aspects of lending and borrowing ebooks through interlibrary loan.\\\\n\\\\n\\\\n== History ==\\\\nInformal borrowing and lending between libraries has a long history, with examples in Western Europe as early as the 8th century CE. In\\nelectronic systems, libraries can also submit requests by postal mail, fax, email, or telephone call, referred to as manual requests. Manual requests can be submitted in the United States through the American Library Association. \\\\nInterlibrary loan and resource sharing have a variety of systems and workflows, which vary based on a library\\\\'s scale of service, regional networks, and library management systems. Processes are automated by electronic systems, including VDX, Tipasa, ILLiad, Ex\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Interlibrary loan works by allowing patrons of one library to borrow physical materials or receive electronic documents from another library that holds the desired item. The borrowing library identifies potential lending libraries, which then deliver the item either physically or electronically. The borrowing library receives the item, delivers it to their patron, and arranges for its return if necessary. Fees may accompany interlibrary loan services, and requests can be managed through semi-automated or manual systems. Libraries negotiate for interlibrary loan eligibility, and with the increasing demand for digital materials, libraries are exploring the legal, technical, and licensing aspects of lending and borrowing ebooks through interlibrary loan.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to dictionary\n",
        "df_dict = df.to_dict(orient='records')\n",
        "\n",
        "# Convert context to list\n",
        "for record in df_dict:\n",
        "    if not isinstance(record.get('context'), list):\n",
        "        if record.get('context') is None:\n",
        "            record['context'] = []\n",
        "        else:\n",
        "            record['context'] = [record['context']]"
      ],
      "metadata": {
        "id": "hviE9dWVadoq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluation in Athina AI**\n",
        "\n",
        "We will use **Context Relevancy** eval here. It Measures the relevancy of the retrieved context, calculated based on both the query and contexts. To learn more about this. Please refer to our [documentation](https://docs.athina.ai/api-reference/evals/preset-evals/overview) for further details"
      ],
      "metadata": {
        "id": "ISQy9RQbYbtY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# set api keys for Athina evals\n",
        "from athina.keys import AthinaApiKey, OpenAiApiKey\n",
        "OpenAiApiKey.set_key(os.getenv('OPENAI_API_KEY'))\n",
        "AthinaApiKey.set_key(os.getenv('ATHINA_API_KEY'))"
      ],
      "metadata": {
        "id": "fmx599yJayWg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load dataset\n",
        "from athina.loaders import Loader\n",
        "dataset = Loader().load_dict(df_dict)"
      ],
      "metadata": {
        "id": "H-TycfUTanch"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# evaluate\n",
        "from athina.evals import RagasContextRelevancy\n",
        "RagasContextRelevancy(model=\"gpt-4o\").run_batch(data=dataset).to_df()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 462
        },
        "id": "Uz0UohC4arbF",
        "outputId": "e7fbfadc-2027-4943-8e11-31cabc0841db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [context_relevancy]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:01<00:00,  1.07s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "You can view your dataset at: https://app.athina.ai/develop/e6867bf1-6320-474b-8dca-3284b41df567\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                             query  \\\n",
              "0  how does interlibrary loan work   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \\\n",
              "0  [Procedures and methods ==\\n\\nAfter receiving a request from their patron, the borrowing library identifies potential lending libraries with the desired item. The lending library then delivers the item physically or electronically, and the borrowing library receives the item, delivers it to their patron, and if necessary, arranges for its return. In some cases, fees accompany interlibrary loan services. While the majority of interlibrary loan requests are now managed through semi-automated\\n...   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              response  \\\n",
              "0  Interlibrary loan works by allowing patrons of one library to borrow physical materials or receive electronic documents from another library that holds the desired item. The borrowing library identifies potential lending libraries, which then deliver the item either physically or electronically. The borrowing library receives the item, delivers it to their patron, and arranges for its return if necessary. Fees may accompany interlibrary loan services, and requests can be managed through semi...   \n",
              "\n",
              "  expected_response             display_name failed  \\\n",
              "0              None  Ragas Context Relevancy   None   \n",
              "\n",
              "                                                                                                                                                                        grade_reason  \\\n",
              "0  This metric is calulated by dividing the number of sentences in context that are relevant for answering the given query by the total number of sentences in the retrieved context   \n",
              "\n",
              "   runtime   model  ragas_context_relevancy  \n",
              "0     1420  gpt-4o                   0.1875  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0da1ce2c-8d6a-4ebb-b035-601366599a70\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "      <th>expected_response</th>\n",
              "      <th>display_name</th>\n",
              "      <th>failed</th>\n",
              "      <th>grade_reason</th>\n",
              "      <th>runtime</th>\n",
              "      <th>model</th>\n",
              "      <th>ragas_context_relevancy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>how does interlibrary loan work</td>\n",
              "      <td>[Procedures and methods ==\\n\\nAfter receiving a request from their patron, the borrowing library identifies potential lending libraries with the desired item. The lending library then delivers the item physically or electronically, and the borrowing library receives the item, delivers it to their patron, and if necessary, arranges for its return. In some cases, fees accompany interlibrary loan services. While the majority of interlibrary loan requests are now managed through semi-automated\\n...</td>\n",
              "      <td>Interlibrary loan works by allowing patrons of one library to borrow physical materials or receive electronic documents from another library that holds the desired item. The borrowing library identifies potential lending libraries, which then deliver the item either physically or electronically. The borrowing library receives the item, delivers it to their patron, and arranges for its return if necessary. Fees may accompany interlibrary loan services, and requests can be managed through semi...</td>\n",
              "      <td>None</td>\n",
              "      <td>Ragas Context Relevancy</td>\n",
              "      <td>None</td>\n",
              "      <td>This metric is calulated by dividing the number of sentences in context that are relevant for answering the given query by the total number of sentences in the retrieved context</td>\n",
              "      <td>1420</td>\n",
              "      <td>gpt-4o</td>\n",
              "      <td>0.1875</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0da1ce2c-8d6a-4ebb-b035-601366599a70')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-0da1ce2c-8d6a-4ebb-b035-601366599a70 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-0da1ce2c-8d6a-4ebb-b035-601366599a70');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "Out of range float values are not JSON compliant: nan"
            }
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    }
  ]
}