examples/basic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example will walk you throught the basic usage of PromptBench. We hope that you can get familiar with the APIs and use it in your own projects later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, there is a unified import of `import promptbench as pb` that easily imports the package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import promptbench as pb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "First, PromptBench supports easy load of datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All supported datasets: \n",
      "['cola', 'sst2', 'qqp', 'mnli', 'mnli_matched', 'mnli_mismatched', 'qnli', 'wnli', 'rte', 'mrpc', 'mmlu', 'squad_v2', 'un_multi', 'iwslt', 'math', 'bool_logic', 'valid_parentheses', 'gsm8k', 'csqa', 'bigbench_date', 'bigbench_object_tracking']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'content': \"it 's a charming and often affecting journey . \", 'label': 1},\n",
       " {'content': 'unflinchingly bleak and desperate ', 'label': 0},\n",
       " {'content': 'allows us to hope that nolan is poised to embark a major career as a commercial yet inventive filmmaker . ',\n",
       "  'label': 1},\n",
       " {'content': \"the acting , costumes , music , cinematography and sound are all astounding given the production 's austere locales . \",\n",
       "  'label': 1},\n",
       " {'content': \"it 's slow -- very , very slow . \", 'label': 0}]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print all supported datasets in promptbench\n",
    "print('All supported datasets: ')\n",
    "print(pb.SUPPORTED_DATASETS)\n",
    "\n",
    "# load a dataset, sst2, for instance.\n",
    "# if the dataset is not available locally, it will be downloaded automatically.\n",
    "dataset = pb.DatasetLoader.load_dataset(\"sst2\")\n",
    "# dataset = pb.DatasetLoader.load_dataset(\"mmlu\")\n",
    "# dataset = pb.DatasetLoader.load_dataset(\"un_multi\")\n",
    "# dataset = pb.DatasetLoader.load_dataset(\"iwslt\", [\"ar-en\", \"de-en\", \"en-ar\"])\n",
    "# dataset = pb.DatasetLoader.load_dataset(\"math\")\n",
    "# dataset = pb.DatasetLoader.load_dataset(\"bool_logic\")\n",
    "# dataset = pb.DatasetLoader.load_dataset(\"valid_parenthesesss\")\n",
    "\n",
    "# print the first 5 examples\n",
    "dataset[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load models\n",
    "\n",
    "Then, you can easily load LLM models via promptbench."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All supported models: \n",
      "['google/flan-t5-large', 'llama2-7b', 'llama2-7b-chat', 'llama2-13b', 'llama2-13b-chat', 'llama2-70b', 'llama2-70b-chat', 'phi-1.5', 'gpt-3.5-turbo', 'gpt-4', 'gpt-4-1106-preview', 'gpt-3.5-turbo-1106', 'vicuna-7b', 'vicuna-13b', 'vicuna-13b-v1.3', 'google/flan-ul2']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "# print all supported models in promptbench\n",
    "print('All supported models: ')\n",
    "print(pb.SUPPORTED_MODELS)\n",
    "\n",
    "# load a model, flan-t5-large, for instance.\n",
    "model = pb.LLMModel(model='google/flan-t5-large', max_new_tokens=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construct prompts\n",
    "\n",
    "Prompts are the key interaction interface to LLMs. You can easily construct a prompt by call the Prompt API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt API supports a list, so you can pass multiple prompts at once.\n",
    "prompts = pb.Prompt([\"Classify the sentence as positive or negative: {content}\",\n",
    "                     \"Determine the emotion of the following sentence as positive or negative: {content}\"\n",
    "                     ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to define the projection function for the model output.\n",
    "Since the output format defined in your prompts may be different from the model output.\n",
    "For example, for sst2 dataset, the label are '0' and '1' to represent 'negative' and 'positive'.\n",
    "But the model output is 'negative' and 'positive'.\n",
    "So we need to define a projection function to map the model output to the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def proj_func(pred):\n",
    "    mapping = {\n",
    "        \"positive\": 1,\n",
    "        \"negative\": 0\n",
    "    }\n",
    "    return mapping.get(pred, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform evaluation using prompts, datasets, and models\n",
    "\n",
    "Finally, you can perform standard evaluation using the loaded prompts, datasets, and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 872/872 [02:16<00:00,  6.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.947, Classify the sentence as positive or negative: {content}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 872/872 [02:18<00:00,  6.29it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.947, Determine the emotion of the following sentence as positive or negative: {content}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "for prompt in prompts:\n",
    "    preds = []\n",
    "    labels = []\n",
    "    for data in tqdm(dataset):\n",
    "        # process input\n",
    "        input_text = pb.InputProcess.basic_format(prompt, data)\n",
    "        label = data['label']\n",
    "        raw_pred = model(input_text)\n",
    "        # process output\n",
    "        pred = pb.OutputProcess.cls(raw_pred, proj_func)\n",
    "        preds.append(pred)\n",
    "        labels.append(label)\n",
    "    \n",
    "    # evaluate\n",
    "    score = pb.Eval.compute_cls_accuracy(preds, labels)\n",
    "    print(f\"{score:.3f}, {prompt}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "promptbench",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}