custom-models/import_models/fmeval_imported_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8b8f4f7c-c45f-4477-99a5-288544114eb4",
   "metadata": {},
   "source": [
    "# Evaluate Bedrock Imported Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64f03356-3bb7-4558-bf0f-51998d65cc5c",
   "metadata": {},
   "source": [
    "In this notebook we will walk through evaluation of the custom imported models that were fine tuned using SageMaker, EC2 or others. \n",
    "\n",
    "To evaluate the models imported into Bedrock we will [FMEval](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-foundation-model-evaluate-auto-lib-custom.html) library. In this notebook we will evaluate an LLM fine tuned for Question Answering. We will use [SpeedOfMagic/trivia_qa_tiny](https://huggingface.co/datasets/SpeedOfMagic/trivia_qa_tiny) dataset for evaluating the Q&A Fine Tuned LLM.\n",
    "\n",
    "Before proceeding futher kindly review the [FMEval License](https://github.com/aws/fmeval/blob/main/LICENSE) and [SpeedOfMagic/trivia_qa_tiny License](https://huggingface.co/datasets/SpeedOfMagic/trivia_qa_tiny)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f8c75c-0c20-4e74-9100-572b37b3073f",
   "metadata": {},
   "source": [
    "## Install the pre-requisites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68d73111-c5fa-43e4-82e4-330ad6844e15",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -Rf ~/.cache/pip/*\n",
    "!rm -Rf /opt/conda/lib/python3.10/site-packages/fsspec*\n",
    "!rm -Rf /opt/conda/lib/python3.10/site-packages/pytz*\n",
    "!pip3 uninstall autogluon --y\n",
    "!pip3 install fmeval --upgrade-strategy only-if-needed --force-reinstall --quiet\n",
    "!pip3 install pyarrow --upgrade --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfdbb520-5d6b-46a4-b05c-9cb569dc5141",
   "metadata": {},
   "source": [
    "## Prepare the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bc9f377-c889-4b02-b631-b5365b69bfd9",
   "metadata": {},
   "source": [
    "We will download the [SpeedOfMagic/trivia_qa_tiny](https://huggingface.co/datasets/SpeedOfMagic/trivia_qa_tiny) dataset and reformat to OAI format. We will save the dataset to disk in JSON format which will be used for evaluation in FMEval evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab879151-760c-4d11-a6c9-6fb7cfb01160",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset, Dataset\n",
    "\n",
    "def create_conversation(row):\n",
    "    row[\"messages\"] = [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": row[\"question\"],\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"assistant\",\n",
    "                \"content\": row[\"answer\"][\"value\"]\n",
    "            },\n",
    "    ]\n",
    "    return row\n",
    "    \n",
    "# Load dataset from the hub\n",
    "dataset = load_dataset(\"SpeedOfMagic/trivia_qa_tiny\")\n",
    "\n",
    "# save datasets\n",
    "dataset[\"test\"].to_json(f\"data/test_dataset.json\", orient=\"records\", force_ascii=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e602c9e-9706-45bb-b541-de3f028d3e52",
   "metadata": {},
   "source": [
    "Lets check if the dataset to be used exists on the disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6112fa52-307d-4330-a74b-b1b117d21f4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "# Check that the dataset file to be used by the evaluation is present\n",
    "if not glob.glob(\"data/test_dataset.json\"):\n",
    "    print(\"ERROR - please make sure the file, trex_sample.jsonl, exists.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "870377cd-bd85-451d-acdf-d1f622a29c56",
   "metadata": {},
   "source": [
    "## FMEval Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3915fd23-028d-48d8-a6e1-9e85b71eedc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fmeval.data_loaders.data_config import DataConfig\n",
    "from fmeval.model_runners.bedrock_model_runner import BedrockModelRunner\n",
    "from fmeval.constants import MIME_TYPE_JSONLINES\n",
    "from fmeval.eval_algorithms.qa_accuracy import QAAccuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8813fee2-7e80-4623-ad95-2d4d53280e98",
   "metadata": {},
   "source": [
    "### Define the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b656ac82-3b3f-41af-86f3-0320b4438fa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"<<bedrock_imported_model_arn>>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f2c717e-f738-41ea-9952-4a328bfb6446",
   "metadata": {},
   "source": [
    "### Dataset Config "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96eb11be-bb00-48da-9e9f-adc333ceea32",
   "metadata": {},
   "source": [
    "Below we will set up a Dataset Config for the local dataset file that we wrote above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf5c3306-f07c-4586-a3c7-9c04fbbf9e49",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = DataConfig(\n",
    "    dataset_name=\"trex_sample\",\n",
    "    dataset_uri=\"data/test_dataset.json\",\n",
    "    dataset_mime_type=MIME_TYPE_JSONLINES,\n",
    "    model_input_location=\"question\",\n",
    "    target_output_location=\"answer\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c62f92-7f3b-44c5-96ec-a699c0db21a1",
   "metadata": {},
   "source": [
    "### Model Runner Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41180206-f7e8-49c2-b48e-cbb710d8f4e3",
   "metadata": {},
   "source": [
    "The will create a Bedrock Model Runner below which will be used to perform inference on the dataset file using the Dataset config above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8c7eca3-22dd-457f-8003-4e1a239ac7eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "bedrock_model_runner = BedrockModelRunner(\n",
    "    model_id=model_id,\n",
    "    output='outputs[0].text',\n",
    "    content_template='{\"prompt\": $prompt, \"max_tokens\": 500}',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc33dff3-ab79-4216-955d-37264c55c407",
   "metadata": {},
   "source": [
    "### Run Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c4ab226-fe06-47d2-89d2-cddf6a506804",
   "metadata": {},
   "source": [
    "We will use [QA Accuracy Evaluation Algorithm](https://aws.github.io/fmeval/fmeval/eval_algorithms.html#EvalAlgorithm.QA_ACCURACY) for evaluating the model given that the model has been fine tuned for Question and Answering.\n",
    "\n",
    "Following are the different Evaluation Algorithms currently supported by FMEval Library:\n",
    "-  prompt_stereotyping\n",
    "-  factual_knowledge\n",
    "-  toxicity\n",
    "-  qa_toxicity\n",
    "-  summarization_toxicity\n",
    "-  general_semantic_robustness\n",
    "-  accuracy\n",
    "-  qa_accuracy\n",
    "-  qa_accuracy_semantic_robustness\n",
    "-  summarization_accuracy\n",
    "-  summarization_accuracy_semantic_robustness\n",
    "-  classification_accuracy\n",
    "-  classification_accuracy_semantic_robustness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9d7617d-df91-4382-9379-2e353b974820",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "with warnings.catch_warnings(record=True) as w:\n",
    "    eval_algo = QAAccuracy()\n",
    "    eval_output = eval_algo.evaluate(model=bedrock_model_runner, dataset_config=config, \n",
    "                                     prompt_template=\"[INST]$model_input[/INST]\", save=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fde9da0a-1246-4d85-97ec-521dac63b03c",
   "metadata": {},
   "source": [
    "#### Parse Evaluation Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46b69661-69a5-4b32-b578-c460442c9031",
   "metadata": {},
   "outputs": [],
   "source": [
    "for op in eval_output:\n",
    "    print(f\"Eval Name: {op.eval_name}\")\n",
    "    for score in op.dataset_scores:\n",
    "        print(f\"{score.name} : {score.value}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efc156d2-d87d-4a1b-b069-53f9e0bb80e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Pandas DataFrame to visualize the results\n",
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "data = []\n",
    "\n",
    "# We obtain the path to the results file from \"output_path\" in the cell above\n",
    "with open(eval_output[0].output_path, \"r\") as file:\n",
    "    for line in file:\n",
    "        data.append(json.loads(line))\n",
    "df = pd.DataFrame(data)\n",
    "df['f1_score'] = df['scores'].apply(lambda x: x[0]['value'])\n",
    "df['exact_match_score'] = df['scores'].apply(lambda x: x[1]['value'])\n",
    "df['quasi_exact_match_score'] = df['scores'].apply(lambda x: x[2]['value'])\n",
    "df['precision_over_words'] = df['scores'].apply(lambda x: x[3]['value'])\n",
    "df['recall_over_words'] = df['scores'].apply(lambda x: x[4]['value'])\n",
    "df = df.drop(['scores'], axis=1)\n",
    "df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}