self_rag.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## **Self RAG**\n",
        "\n",
        "SELF-RAG is a method that improves the accuracy and quality of text generated by a language model (LM). It does this by using retrieval to find relevant information and allowing the model to reflect on its output.\n",
        "\n",
        "The model generates text with the help of retrieved passages, and then it checks its own response by creating reflection tokens. These tokens tell the model if it needs more information or if the answer is complete and supported by the retrieved data.\n",
        "\n",
        "Research Paper: [Self RAG](https://arxiv.org/pdf/2310.11511)\n"
      ],
      "metadata": {
        "id": "3hIY5SZAbuFB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Initial Setup**"
      ],
      "metadata": {
        "id": "ou7ws3vpebm8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install --q athina faiss-gpu langgraph"
      ],
      "metadata": {
        "id": "tq806kcBbtdu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')\n",
        "os.environ['ATHINA_API_KEY'] = userdata.get('ATHINA_API_KEY')"
      ],
      "metadata": {
        "id": "7kPfeR5K-kuh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Indexing**"
      ],
      "metadata": {
        "id": "DMzSiRwAeha8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load embedding model\n",
        "from langchain_openai import OpenAIEmbeddings\n",
        "embeddings = OpenAIEmbeddings()"
      ],
      "metadata": {
        "id": "lvUy2gkY_IVZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load data\n",
        "from langchain.document_loaders import CSVLoader\n",
        "loader = CSVLoader(\"./context.csv\")\n",
        "documents = loader.load()"
      ],
      "metadata": {
        "id": "yMy0cM6T_Kcf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# split documents\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
        "documents = text_splitter.split_documents(documents)"
      ],
      "metadata": {
        "id": "OvHF-0F5_PqV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vectorstore\n",
        "from langchain.vectorstores import FAISS\n",
        "vectorstore = FAISS.from_documents(documents, embeddings)"
      ],
      "metadata": {
        "id": "IlOS0yB3_Rwa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Retriever**"
      ],
      "metadata": {
        "id": "sQReFeMZeknp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create retriever\n",
        "retriever = vectorstore.as_retriever()"
      ],
      "metadata": {
        "id": "UlSnVlO7_bsb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Document Grader**\n",
        "The document grader evaluates whether a document is relevant to the given query."
      ],
      "metadata": {
        "id": "JEoJgzTOeppN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for doc retriever\n",
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "from langchain_core.pydantic_v1 import BaseModel, Field\n",
        "from langchain_openai import ChatOpenAI\n",
        "\n",
        "# define a data class\n",
        "class GradeDocuments(BaseModel):\n",
        "    binary_score: str = Field(\n",
        "        description=\"Documents are relevant to the question, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeDocuments)\n",
        "\n",
        "# Prompt for the grader\n",
        "system = \"\"\"You are a grader assessing relevance of a retrieved document to a user question. \\n\n",
        "    If the document contains keyword(s) or semantic meaning related to the question, grade it as relevant. \\n\n",
        "    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.\"\"\"\n",
        "grade_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"Retrieved document: \\n\\n {document} \\n\\n User question: {question}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "retrieval_grader = grade_prompt | structured_llm_grader"
      ],
      "metadata": {
        "id": "PWiF0_1v_g_y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# testing grader\n",
        "question = \"what are points on a mortgage\"\n",
        "docs = retriever.invoke(question)\n",
        "print(retrieval_grader.invoke({\"question\": question, \"document\": docs}))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "74oYPlRFANUt",
        "outputId": "15cf8b4c-9df1-4cda-e019-6968e3196b3f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "binary_score='yes'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **RAG Chain**"
      ],
      "metadata": {
        "id": "qg6Git55e0fp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create document chain\n",
        "from langchain import hub\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "from langchain.prompts import ChatPromptTemplate\n",
        "\n",
        "template = \"\"\"\"\n",
        "You are a helpful assistant that answers questions based on the following context\n",
        "Context: {context}\n",
        "Question: {question}\n",
        "Answer:\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "\n",
        "def format_docs(docs):\n",
        "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
        "\n",
        "\n",
        "rag_chain = prompt | llm | StrOutputParser()"
      ],
      "metadata": {
        "id": "locILw-xASan"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# response\n",
        "generation = rag_chain.invoke({\"context\": docs, \"question\": question})\n",
        "generation"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "GuagLQrkAUYA",
        "outputId": "d064f70a-68c9-42bd-c1da-fddc111e03ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"Points on a mortgage, also known as discount points, mortgage points, or simply points, are a form of pre-paid interest that can be paid by a borrower to a lender when arranging a mortgage. One point is equal to one percent of the loan amount. By paying points, a borrower can effectively reduce the interest rate on the loan, resulting in a lower monthly payment. Points can also be used to qualify for a loan based on monthly income versus the monthly loan payment. It's important to note that points are different from origination fees, mortgage arrangement fees, or broker fees.\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Hallucination Grader**\n",
        "The hallucination grader checks whether the answer is grounded in or supported by the given set of facts"
      ],
      "metadata": {
        "id": "rRQTc6VmfDPL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for hallucination\n",
        "# define a data class\n",
        "class GradeHallucinations(BaseModel):\n",
        "    \"\"\"Binary score for hallucination present in generation answer.\"\"\"\n",
        "\n",
        "    binary_score: str = Field(\n",
        "        description=\"Answer is grounded in the facts, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeHallucinations)\n",
        "\n",
        "# prompt for the grader\n",
        "system = \"\"\"You are a grader assessing whether an LLM generation is grounded in / supported by a set of retrieved facts. \\n\n",
        "     Give a binary score 'yes' or 'no'. 'Yes' means that the answer is grounded in / supported by the set of facts.\"\"\"\n",
        "hallucination_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"Set of facts: \\n\\n {documents} \\n\\n LLM generation: {generation}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "hallucination_grader = hallucination_prompt | structured_llm_grader\n",
        "hallucination_grader.invoke({\"documents\": docs, \"generation\": generation})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FXJJC7yvhiVA",
        "outputId": "31625d2b-d4be-485f-cea3-e46fbd26d976"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GradeHallucinations(binary_score='yes')"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Answer Grader**\n",
        "The answer grader evaluates whether an answer effectively addresses the given question."
      ],
      "metadata": {
        "id": "FmLXpPFkfeM8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for answer\n",
        "# define a data class\n",
        "class GradeAnswer(BaseModel):\n",
        "    \"\"\"Binary score to assess answer addresses question.\"\"\"\n",
        "\n",
        "    binary_score: str = Field(\n",
        "        description=\"Answer addresses the question, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeAnswer)\n",
        "\n",
        "# prompt for the grader\n",
        "system = \"\"\"You are a grader assessing whether an answer addresses / resolves a question \\n\n",
        "     Give a binary score 'yes' or 'no'. Yes' means that the answer resolves the question.\"\"\"\n",
        "answer_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"User question: \\n\\n {question} \\n\\n LLM generation: {generation}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "answer_grader = answer_prompt | structured_llm_grader\n",
        "answer_grader.invoke({\"question\": question, \"generation\": generation})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R65ciWyXnMad",
        "outputId": "50f46208-e912-4a8d-c45b-6659383a2bbb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GradeAnswer(binary_score='yes')"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Create Graph**"
      ],
      "metadata": {
        "id": "4Hc_X6y_fm3v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Define Graph State**"
      ],
      "metadata": {
        "id": "Y5XeBloBfoiG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define a data class for state\n",
        "from typing import List\n",
        "from typing_extensions import TypedDict\n",
        "\n",
        "class GraphState(TypedDict):\n",
        "    question: str\n",
        "    generation: str\n",
        "    documents: List[str]"
      ],
      "metadata": {
        "id": "SkHTZwxpndjl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# # define graph steps\n",
        "# from langchain.schema import Document\n",
        "\n",
        "# all nodes\n",
        "def retrieve(state):\n",
        "\n",
        "    print(\"---RETRIEVE---\")\n",
        "    question = state[\"question\"]\n",
        "\n",
        "    # Retrieval\n",
        "    documents = retriever.get_relevant_documents(question)\n",
        "    return {\"documents\": documents, \"question\": question}\n",
        "\n",
        "\n",
        "def generate(state):\n",
        "\n",
        "    print(\"---GENERATE---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # RAG generation\n",
        "    generation = rag_chain.invoke({\"context\": documents, \"question\": question})\n",
        "    return {\"documents\": documents, \"question\": question, \"generation\": generation}\n",
        "\n",
        "\n",
        "def grade_documents(state):\n",
        "\n",
        "    print(\"---CHECK DOCUMENT RELEVANCE TO QUESTION---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # Score each doc\n",
        "    filtered_docs = []\n",
        "    for d in documents:\n",
        "        score = retrieval_grader.invoke(\n",
        "            {\"question\": question, \"document\": d.page_content}\n",
        "        )\n",
        "        grade = score.binary_score\n",
        "        if grade == \"yes\":\n",
        "            print(\"---GRADE: DOCUMENT RELEVANT---\")\n",
        "            filtered_docs.append(d)\n",
        "        else:\n",
        "            print(\"---GRADE: DOCUMENT NOT RELEVANT---\")\n",
        "            continue\n",
        "    return {\"documents\": filtered_docs, \"question\": question}\n",
        "\n",
        "\n",
        "# edges\n",
        "def decide_to_generate(state):\n",
        "\n",
        "    print(\"---ASSESS GRADED DOCUMENTS---\")\n",
        "    state[\"question\"]\n",
        "    filtered_documents = state[\"documents\"]\n",
        "\n",
        "    if not filtered_documents:\n",
        "        # All documents have been filtered check_relevance\n",
        "        print(\n",
        "            \"---DECISION: ALL DOCUMENTS ARE NOT RELEVANT TO QUESTION---\"\n",
        "        )\n",
        "        return \"no_relevant_documents\"\n",
        "    else:\n",
        "        # We have relevant documents, so generate answer\n",
        "        print(\"---DECISION: GENERATE---\")\n",
        "        return \"generate\"\n",
        "\n",
        "\n",
        "def grade_generation_v_documents_and_question(state):\n",
        "\n",
        "    print(\"---CHECK HALLUCINATIONS---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "    generation = state[\"generation\"]\n",
        "\n",
        "    score = hallucination_grader.invoke(\n",
        "        {\"documents\": documents, \"generation\": generation}\n",
        "    )\n",
        "    grade = score.binary_score\n",
        "\n",
        "    # Check hallucination\n",
        "    if grade == \"yes\":\n",
        "        print(\"---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\")\n",
        "        # Check question-answering\n",
        "        print(\"---GRADE GENERATION vs QUESTION---\")\n",
        "        score = answer_grader.invoke({\"question\": question, \"generation\": generation})\n",
        "        grade = score.binary_score\n",
        "        if grade == \"yes\":\n",
        "            print(\"---DECISION: GENERATION ADDRESSES QUESTION---\")\n",
        "            return \"useful\"\n",
        "        else:\n",
        "            print(\"---DECISION: GENERATION DOES NOT ADDRESS QUESTION---\")\n",
        "            return \"not useful\"\n",
        "    else:\n",
        "        pprint(\"---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---\")\n",
        "        return \"not supported\""
      ],
      "metadata": {
        "id": "BtShAQXSnh87"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Build Graph**"
      ],
      "metadata": {
        "id": "FWaVqzLDfwXC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Build graph\n",
        "from langgraph.graph import END, StateGraph, START\n",
        "\n",
        "workflow = StateGraph(GraphState)\n",
        "\n",
        "# Define the nodes\n",
        "workflow.add_node(\"retrieve\", retrieve)\n",
        "workflow.add_node(\"grade_documents\", grade_documents)\n",
        "workflow.add_node(\"generate\", generate)\n",
        "\n",
        "# Build graph\n",
        "workflow.add_edge(START, \"retrieve\")\n",
        "workflow.add_edge(\"retrieve\", \"grade_documents\")\n",
        "workflow.add_conditional_edges(\n",
        "    \"grade_documents\",\n",
        "    decide_to_generate,\n",
        "    {\n",
        "        \"generate\": \"generate\",\n",
        "        \"no_relevant_documents\": END,\n",
        "    },\n",
        ")\n",
        "\n",
        "workflow.add_conditional_edges(\n",
        "    \"generate\",\n",
        "    grade_generation_v_documents_and_question,\n",
        "    {\n",
        "        \"not supported\": \"generate\",\n",
        "        \"useful\": END,\n",
        "    },\n",
        ")\n",
        "\n",
        "# Compile\n",
        "app = workflow.compile()"
      ],
      "metadata": {
        "id": "nZpTvXjnnnmb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Final generation example 1 (relevant documents)\n",
        "from pprint import pprint\n",
        "\n",
        "inputs = {\"question\": \"what are points on a mortgage\"}\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        pprint(f\"Node '{key}':\")\n",
        "    pprint(\"\\n---\\n\")\n",
        "\n",
        "if \"generation\" in value:\n",
        "    pprint(value[\"generation\"])\n",
        "else:\n",
        "    pprint(\"No relevant documents found or no generation produced.\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CwyI8NftnqYG",
        "outputId": "27ea7116-befa-4f9b-a408-b7df47f81aab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---RETRIEVE---\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The method `BaseRetriever.get_relevant_documents` was deprecated in langchain-core 0.1.46 and will be removed in 0.3.0. Use invoke instead.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\"Node 'retrieve':\"\n",
            "'\\n---\\n'\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: GENERATE---\n",
            "\"Node 'grade_documents':\"\n",
            "'\\n---\\n'\n",
            "---GENERATE---\n",
            "---CHECK HALLUCINATIONS---\n",
            "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
            "---GRADE GENERATION vs QUESTION---\n",
            "---DECISION: GENERATION ADDRESSES QUESTION---\n",
            "\"Node 'generate':\"\n",
            "'\\n---\\n'\n",
            "('Points on a mortgage, also known as discount points, are a form of pre-paid '\n",
            " 'interest that borrowers can pay to a lender when arranging a mortgage in the '\n",
            " 'United States. One point equals one percent of the loan amount. By paying '\n",
            " 'points, a borrower can reduce the interest rate on the loan, resulting in a '\n",
            " 'lower monthly payment. Points can also be used to qualify for a loan based '\n",
            " \"on monthly income versus the monthly loan payment. It's important to note \"\n",
            " 'that points are different from origination fees, mortgage arrangement fees, '\n",
            " 'or broker fees.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# example 2 (no relevant documents)\n",
        "from pprint import pprint\n",
        "\n",
        "inputs = {\"question\": \"Explain how the different types of agent memory work?\"}\n",
        "\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        pprint(f\"Node '{key}':\")\n",
        "    pprint(\"\\n---\\n\")\n",
        "\n",
        "if \"generation\" in value:\n",
        "    pprint(value[\"generation\"])\n",
        "else:\n",
        "    pprint(\"No relevant documents found or no generation produced.\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pBKmsfNBn6I1",
        "outputId": "e7b84866-a564-4373-e4b9-1a26b03e0592"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---RETRIEVE---\n",
            "\"Node 'retrieve':\"\n",
            "'\\n---\\n'\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: ALL DOCUMENTS ARE NOT RELEVANT TO QUESTION---\n",
            "\"Node 'grade_documents':\"\n",
            "'\\n---\\n'\n",
            "'No relevant documents found or no generation produced.'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Preparing Data for Evaluation**"
      ],
      "metadata": {
        "id": "mTtzrhJ9Y5UT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a dataframe to store the question, context, and response\n",
        "inputs = {\"question\": \"what are points on a mortgage\"}\n",
        "outputs = []\n",
        "\n",
        "\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        if key == \"generate\":  # Assuming 'generate' node produces the final output\n",
        "            question = value[\"question\"]\n",
        "            documents = value[\"documents\"]\n",
        "            generation = value[\"generation\"]\n",
        "\n",
        "            # Combine document contents into a single string\n",
        "            context = \"\\n\".join(doc.page_content for doc in documents)\n",
        "\n",
        "            # Append the result\n",
        "            outputs.append({\n",
        "                \"query\": question,\n",
        "                \"context\": context,\n",
        "                \"response\": generation,\n",
        "            })"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jq96Il_ux_mX",
        "outputId": "444a9dc6-7f85-48e8-df3d-62d428a0a63f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---RETRIEVE---\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: GENERATE---\n",
            "---GENERATE---\n",
            "---CHECK HALLUCINATIONS---\n",
            "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
            "---GRADE GENERATION vs QUESTION---\n",
            "---DECISION: GENERATION ADDRESSES QUESTION---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to DataFrame\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(outputs)"
      ],
      "metadata": {
        "id": "_N5Giy-myGTG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "id": "bAgfG0LOyH9T",
        "outputId": "565e8822-2b9d-40b3-f72d-c15341b35a92"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           query  \\\n",
              "0  what are points on a mortgage   \n",
              "\n",
              "                                             context  \\\n",
              "0  context: [\"Discount points, also called mortga...   \n",
              "\n",
              "                                            response  \n",
              "0  Points on a mortgage, also known as discount p...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a09b53bc-81b1-4711-892b-16beddeb3907\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>what are points on a mortgage</td>\n",
              "      <td>context: [\"Discount points, also called mortga...</td>\n",
              "      <td>Points on a mortgage, also known as discount p...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a09b53bc-81b1-4711-892b-16beddeb3907')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a09b53bc-81b1-4711-892b-16beddeb3907 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a09b53bc-81b1-4711-892b-16beddeb3907');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_9e1ebcae-eae0-464b-8955-c385d6184c1d\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_9e1ebcae-eae0-464b-8955-c385d6184c1d button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"query\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"what are points on a mortgage\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"context: [\\\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for this\\npoints is the concept of the 'no closing cost loan', in which the consumer accepts a higher interest rate in return for the lender paying the loan's closing costs up front.  In some cases a purchaser can negotiate with the seller to get them to pay seller's points which can be used to pay mortgage points.\\\\n\\\\n\\\\n== References ==\\\\n\\\\n\\\\n== External links ==\\\\nirs.gov/publications/p936 \\u2013 IRS Form 936 defines a point for the purpose of deducting mortgage interest for U.S. income taxes\\\"\\nrate.Points may also be purchased to reduce the monthly payment for the purpose of qualifying for a loan.  Loan qualification based on monthly income versus the monthly loan payment may sometimes only be achievable by reducing the monthly payment through the purchasing of points to buy down the interest rate, thereby reducing the monthly loan payment.\\\\nDiscount points may be different from origination fee, mortgage arrangement fee or broker fee. Discount points are always used to buy down the\\nup-front payment. For each point purchased, the loan rate is typically reduced by anywhere from 1/8% (0.125%) to 1/4% (0.25%).Selling the property or refinancing prior to this break-even point will result in a net financial loss for the buyer while keeping the loan for longer than this break-even point will result in a net financial savings for the buyer. Accordingly, if the intention is to buy and sell the property or refinance, paying points will cost more than just paying the higher interest\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to a lender when arranging a mortgage in the United States. One point equals one percent of the loan amount. By paying points, a borrower can reduce the interest rate on the loan, resulting in a lower monthly payment. Points can also be used to qualify for a loan based on monthly income versus the monthly loan payment. It's important to note that points are different from origination fees, mortgage arrangement fees, or broker fees.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to dictionary\n",
        "df_dict = df.to_dict(orient='records')\n",
        "\n",
        "# Convert context to list\n",
        "for record in df_dict:\n",
        "    if not isinstance(record.get('context'), list):\n",
        "        if record.get('context') is None:\n",
        "            record['context'] = []\n",
        "        else:\n",
        "            record['context'] = [record['context']]"
      ],
      "metadata": {
        "id": "V_MkBZv5JcXQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluation in Athina AI**\n",
        "\n",
        "We will use **Does Response Answer Query** eval here. It Checks if the response answer the user's query. To learn more about this. Please refer to our [documentation](https://docs.athina.ai/api-reference/evals/preset-evals/overview) for further details."
      ],
      "metadata": {
        "id": "Jrj1b6MPyLid"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# set api keys for Athina evals\n",
        "from athina.keys import AthinaApiKey, OpenAiApiKey\n",
        "OpenAiApiKey.set_key(os.getenv('OPENAI_API_KEY'))\n",
        "AthinaApiKey.set_key(os.getenv('ATHINA_API_KEY'))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9jXkYgJ1Jf8K",
        "outputId": "70e9ad8b-2278-4e8e-ad2a-528839c28839"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/pydantic/_internal/_generate_schema.py:386: UserWarning: <built-in function any> is not a Python type (it may be an instance of an object), Pydantic will allow any object with no validation since we cannot even enforce that the input is an instance of the given type. To get rid of this error wrap the type with `pydantic.SkipValidation`.\n",
            "  warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# load dataset\n",
        "from athina.loaders import Loader\n",
        "dataset = Loader().load_dict(df_dict)"
      ],
      "metadata": {
        "id": "PESrn6XCyVhL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# evaluate\n",
        "from athina.evals import DoesResponseAnswerQuery\n",
        "DoesResponseAnswerQuery(model=\"gpt-4o\").run_batch(data=dataset).to_df()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 393
        },
        "id": "fAIq316ZyjKi",
        "outputId": "21d3e154-a896-409d-defe-a3e3717f043f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "You can view your dataset at: https://app.athina.ai/develop/7ad4a012-9202-4a5e-a131-769f0e6c620c\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           query  \\\n",
              "0  what are points on a mortgage   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \\\n",
              "0  [context: [\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for ...   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              response  \\\n",
              "0  Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to a lender when arranging a mortgage in the United States. One point equals one percent of the loan amount. By paying points, a borrower can reduce the interest rate on the loan, resulting in a lower monthly payment. Points can also be used to qualify for a loan based on monthly income versus the monthly loan payment. It's important to note that points are different from origination f...   \n",
              "\n",
              "  expected_response                display_name  failed  \\\n",
              "0              None  Does Response Answer Query   False   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         grade_reason  \\\n",
              "0  The response provides a clear and detailed explanation of what points on a mortgage are. It explains that points are a form of pre-paid interest, how they are calculated (one point equals one percent of the loan amount), and their purpose (to reduce the interest rate and monthly payment). Additionally, it distinguishes points from other fees such as origination fees, mortgage arrangement fees, or broker fees. This sufficiently covers the user's query about points on a mortgage.   \n",
              "\n",
              "   runtime   model  passed  \n",
              "0     1704  gpt-4o     1.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d07260bf-1e70-465e-ae8c-d89a9044d3d3\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "      <th>expected_response</th>\n",
              "      <th>display_name</th>\n",
              "      <th>failed</th>\n",
              "      <th>grade_reason</th>\n",
              "      <th>runtime</th>\n",
              "      <th>model</th>\n",
              "      <th>passed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>what are points on a mortgage</td>\n",
              "      <td>[context: [\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for ...</td>\n",
              "      <td>Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to a lender when arranging a mortgage in the United States. One point equals one percent of the loan amount. By paying points, a borrower can reduce the interest rate on the loan, resulting in a lower monthly payment. Points can also be used to qualify for a loan based on monthly income versus the monthly loan payment. It's important to note that points are different from origination f...</td>\n",
              "      <td>None</td>\n",
              "      <td>Does Response Answer Query</td>\n",
              "      <td>False</td>\n",
              "      <td>The response provides a clear and detailed explanation of what points on a mortgage are. It explains that points are a form of pre-paid interest, how they are calculated (one point equals one percent of the loan amount), and their purpose (to reduce the interest rate and monthly payment). Additionally, it distinguishes points from other fees such as origination fees, mortgage arrangement fees, or broker fees. This sufficiently covers the user's query about points on a mortgage.</td>\n",
              "      <td>1704</td>\n",
              "      <td>gpt-4o</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d07260bf-1e70-465e-ae8c-d89a9044d3d3')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d07260bf-1e70-465e-ae8c-d89a9044d3d3 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d07260bf-1e70-465e-ae8c-d89a9044d3d3');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "Out of range float values are not JSON compliant: nan"
            }
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    }
  ]
}