fine-tuning_distilgpt2_for_text-generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7k8m6a0R15sz"
   },
   "source": [
    "# Fine Tuning DistilGPT2 for Text Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1C9zObeOoEoN"
   },
   "source": [
    "The DistilGPT2 is lighter in weight and faster in language generation than the original OpenAI GPT2. It is created by process of distillation applied to GPT2. Here, we will generate emotions by fine-tuning DistilGPT2 on a sample of \"emotion\" dataset from Hugging Face Hub. We can train a language generation model so that it can generate text for any subject in English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n",
      "Reusing dataset emotion (/root/.cache/huggingface/datasets/emotion/default/0.0.0/348f63ca8e27b3713b6c04d723efe6d824a56fb3d1449794716c0f0296072705)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d634c72b8e24bc3b82131eee16f9927",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "emotions = load_dataset(\"emotion\")\n",
    "emotions.set_format(\"pandas\")\n",
    "train, valid, test = emotions[\"train\"][:], emotions[\"validation\"][:], emotions[\"test\"][:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For fine-tuning distilgpt2, just need the text field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = list(train['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nUXGPS5Li-E4"
   },
   "source": [
    "Store the emotions in a txt file where each line of txt file is a single expression "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1647869742314,
     "user": {
      "displayName": "Okyong Choi",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgXt3H1vBv3hDstzKZEIDl0RZQP9ouHdNHms_-n8g=s64",
      "userId": "02726249487150490039"
     },
     "user_tz": 240
    },
    "id": "LotaG9qgZmHy"
   },
   "outputs": [],
   "source": [
    "file_name = 'testing.txt'\n",
    "with open(file_name, 'w') as f:\n",
    "    f.write(\" |EndOfText|\\n\".join(texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9KwFvrFRjOwo"
   },
   "source": [
    "Now, let's come to Transformers by Huggingface, and unleash the Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YbIJfTnDjmG6"
   },
   "source": [
    "Make 2 directories. \n",
    "\n",
    "1) weights - for storing the weights of distilgpt2\n",
    "\n",
    "2) tokenizer - for storing the tokenizer of distilgpt2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q1HiZgs-kFLg"
   },
   "source": [
    "# Fine-Tuning of DistilGPT2\n",
    "Now, its time for Training (or fine tuning) distilgpt2 with IMDB reviews.  \n",
    "Given below is a command containing few parameters to help Transformers finetune distilgpt2. Now, let's understand what these parameters mean\n",
    "\n",
    "1. output_dir: It is the weights_dir we made where our finetuned model will be stored in the form of checkpoints\n",
    "\n",
    "2. model_name_or_path: It tells the kind of model we are currently dealing with\n",
    "\n",
    "3. per_device_train_batch_size: It tells the batch size for each gpu\n",
    "\n",
    "4. do_train: It tells pytorch to start training mode\n",
    "\n",
    "5. train_file: This is where we give the input text data \n",
    "\n",
    "6. num_train_epochs: Number of epochs for finetuning\n",
    "\n",
    "\n",
    "Now, let the training begin..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1647870081262,
     "user": {
      "displayName": "Okyong Choi",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgXt3H1vBv3hDstzKZEIDl0RZQP9ouHdNHms_-n8g=s64",
      "userId": "02726249487150490039"
     },
     "user_tz": 240
    },
    "id": "VQ5ZpOMGtH2W"
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(level=logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "executionInfo": {
     "elapsed": 308,
     "status": "ok",
     "timestamp": 1647870083515,
     "user": {
      "displayName": "Okyong Choi",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgXt3H1vBv3hDstzKZEIDl0RZQP9ouHdNHms_-n8g=s64",
      "userId": "02726249487150490039"
     },
     "user_tz": 240
    },
    "id": "J_rFZRe_2KAW"
   },
   "outputs": [],
   "source": [
    "weights_dir = \"output\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine-tuning the DistilGPT2 for causal language modeling (GPT-2 in this notebooek) on a IMDB dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 44710,
     "status": "ok",
     "timestamp": 1647869790110,
     "user": {
      "displayName": "Okyong Choi",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgXt3H1vBv3hDstzKZEIDl0RZQP9ouHdNHms_-n8g=s64",
      "userId": "02726249487150490039"
     },
     "user_tz": 240
    },
    "id": "EkocIBHfaZul",
    "outputId": "598edd12-1a8f-4daa-e54a-4939551e71e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'transformers' already exists and is not an empty directory.\n"
     ]
    }
   ],
   "source": [
    "!git clone --depth=1 --branch v4.6.0-release https://github.com/huggingface/transformers.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1647870084486,
     "user": {
      "displayName": "Okyong Choi",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgXt3H1vBv3hDstzKZEIDl0RZQP9ouHdNHms_-n8g=s64",
      "userId": "02726249487150490039"
     },
     "user_tz": 240
    },
    "id": "42gszj3gkUU1"
   },
   "outputs": [],
   "source": [
    "cmd = f'''\n",
    "python transformers/examples/pytorch/language-modeling/run_clm.py \\\n",
    "    --model_name_or_path distilgpt2 \\\n",
    "    --train_file {file_name} \\\n",
    "    --do_train \\\n",
    "    --num_train_epochs 3 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --per_device_train_batch_size 2 \\\n",
    "    --output_dir {weights_dir}\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"WANDB_DISABLED\"] = \"true\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 538846,
     "status": "ok",
     "timestamp": 1647870625377,
     "user": {
      "displayName": "Okyong Choi",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgXt3H1vBv3hDstzKZEIDl0RZQP9ouHdNHms_-n8g=s64",
      "userId": "02726249487150490039"
     },
     "user_tz": 240
    },
    "id": "mQKT9jlOcnjY",
    "outputId": "f8e15643-b4ca-40bb-e46c-c2ff24ef6d70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using the `WAND_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).\n",
      "03/22/2022 13:51:04 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 2distributed training: False, 16-bits training: False\n",
      "03/22/2022 13:51:04 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir=output, overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, evaluation_strategy=IntervalStrategy.NO, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=8, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, lr_scheduler_type=SchedulerType.LINEAR, warmup_ratio=0.0, warmup_steps=0, logging_dir=runs/Mar22_13-51-04_61e3c5fc8419, logging_strategy=IntervalStrategy.STEPS, logging_first_step=False, logging_steps=500, save_strategy=IntervalStrategy.STEPS, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level=O1, fp16_backend=auto, fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=[], dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=output, disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name=length, report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, use_legacy_prediction_loop=False, push_to_hub=False, resume_from_checkpoint=None, _n_gpu=2, mp_parameters=)\n",
      "03/22/2022 13:51:05 - WARNING - datasets.builder -   Using custom data configuration default-958516cacc7d5a7e\n",
      "Downloading and preparing dataset text/default to /root/.cache/huggingface/datasets/text/default-958516cacc7d5a7e/0.0.0/4b86d314f7236db91f0a0f5cda32d4375445e64c5eda2692655dd99c2dac68e8...\n",
      "Downloading data files: 100%|███████████████████| 1/1 [00:00<00:00, 2051.00it/s]\n",
      "Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 702.56it/s]\n",
      "Dataset text downloaded and prepared to /root/.cache/huggingface/datasets/text/default-958516cacc7d5a7e/0.0.0/4b86d314f7236db91f0a0f5cda32d4375445e64c5eda2692655dd99c2dac68e8. Subsequent calls will reuse this data.\n",
      "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 579.00it/s]\n",
      "[INFO|configuration_utils.py:517] 2022-03-22 13:51:06,517 >> loading configuration file https://huggingface.co/distilgpt2/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/f985248d2791fcff97732e4ee263617adec1edb5429a2b8421734c6d14e39bee.422318838d1ec4e061efb4ea29671cb2a044e244dc69229682bebd7cacc81631\n",
      "[INFO|configuration_utils.py:553] 2022-03-22 13:51:06,518 >> Model config GPT2Config {\n",
      "  \"_num_labels\": 1,\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0\n",
      "  },\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_inner\": null,\n",
      "  \"n_layer\": 6,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"scale_attn_weights\": true,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"transformers_version\": \"4.6.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "[INFO|configuration_utils.py:517] 2022-03-22 13:51:07,391 >> loading configuration file https://huggingface.co/distilgpt2/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/f985248d2791fcff97732e4ee263617adec1edb5429a2b8421734c6d14e39bee.422318838d1ec4e061efb4ea29671cb2a044e244dc69229682bebd7cacc81631\n",
      "[INFO|configuration_utils.py:553] 2022-03-22 13:51:07,393 >> Model config GPT2Config {\n",
      "  \"_num_labels\": 1,\n",
      "  \"activation_function\": \"gelu_new\",\n",
      "  \"architectures\": [\n",
      "    \"GPT2LMHeadModel\"\n",
      "  ],\n",
      "  \"attn_pdrop\": 0.1,\n",
      "  \"bos_token_id\": 50256,\n",
      "  \"embd_pdrop\": 0.1,\n",
      "  \"eos_token_id\": 50256,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0\n",
      "  },\n",
      "  \"layer_norm_epsilon\": 1e-05,\n",
      "  \"model_type\": \"gpt2\",\n",
      "  \"n_ctx\": 1024,\n",
      "  \"n_embd\": 768,\n",
      "  \"n_head\": 12,\n",
      "  \"n_inner\": null,\n",
      "  \"n_layer\": 6,\n",
      "  \"n_positions\": 1024,\n",
      "  \"resid_pdrop\": 0.1,\n",
      "  \"scale_attn_weights\": true,\n",
      "  \"summary_activation\": null,\n",
      "  \"summary_first_dropout\": 0.1,\n",
      "  \"summary_proj_to_labels\": true,\n",
      "  \"summary_type\": \"cls_index\",\n",
      "  \"summary_use_proj\": true,\n",
      "  \"task_specific_params\": {\n",
      "    \"text-generation\": {\n",
      "      \"do_sample\": true,\n",
      "      \"max_length\": 50\n",
      "    }\n",
      "  },\n",
      "  \"transformers_version\": \"4.6.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50257\n",
      "}\n",
      "\n",
      "[INFO|tokenization_utils_base.py:1717] 2022-03-22 13:51:12,177 >> loading file https://huggingface.co/distilgpt2/resolve/main/vocab.json from cache at /root/.cache/huggingface/transformers/55051ac97dcc32f0a736d21a32a4d42b0d9b90f117ca7c38e65038b04bd5c3f5.c7ed1f96aac49e745788faa77ba0a26a392643a50bb388b9c04ff469e555241f\n",
      "[INFO|tokenization_utils_base.py:1717] 2022-03-22 13:51:12,177 >> loading file https://huggingface.co/distilgpt2/resolve/main/merges.txt from cache at /root/.cache/huggingface/transformers/9dfb299b74cdf7601ba7cd3a8073dbdac351caec0ed7ab5849b098b3c8ae3d57.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b\n",
      "[INFO|tokenization_utils_base.py:1717] 2022-03-22 13:51:12,177 >> loading file https://huggingface.co/distilgpt2/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/accb287b5a5396b2597382916b6cc939fdab1366e89475a92338d3971b3d02b7.cf2d0ecb83b6df91b3dbb53f1d1e4c311578bfd3aa0e04934215a49bf9898df0\n",
      "[INFO|tokenization_utils_base.py:1717] 2022-03-22 13:51:12,177 >> loading file https://huggingface.co/distilgpt2/resolve/main/added_tokens.json from cache at None\n",
      "[INFO|tokenization_utils_base.py:1717] 2022-03-22 13:51:12,177 >> loading file https://huggingface.co/distilgpt2/resolve/main/special_tokens_map.json from cache at None\n",
      "[INFO|tokenization_utils_base.py:1717] 2022-03-22 13:51:12,177 >> loading file https://huggingface.co/distilgpt2/resolve/main/tokenizer_config.json from cache at None\n",
      "[INFO|modeling_utils.py:1155] 2022-03-22 13:51:12,989 >> loading weights file https://huggingface.co/distilgpt2/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/43a212e83e76bcb07f45be584cf100676bdbbbe9c13f9e5c1c050049143a832f.a83d881ec4d624fd4b5826dd026e315246c48c67504ff91c0500570e291a54ba\n",
      "[INFO|modeling_utils.py:1339] 2022-03-22 13:51:13,759 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.\n",
      "\n",
      "[INFO|modeling_utils.py:1347] 2022-03-22 13:51:13,759 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at distilgpt2.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.\n",
      "100%|███████████████████████████████████████████| 16/16 [00:00<00:00, 58.43ba/s]\n",
      "100%|███████████████████████████████████████████| 16/16 [00:01<00:00,  9.35ba/s]\n",
      "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py:30: UserWarning: \n",
      "    There is an imbalance between your GPUs. You may want to exclude GPU 1 which\n",
      "    has less than 75% of the memory or cores of GPU 0. You can do so by setting\n",
      "    the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES\n",
      "    environment variable.\n",
      "  warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))\n",
      "[INFO|trainer.py:1156] 2022-03-22 13:51:17,028 >> ***** Running training *****\n",
      "[INFO|trainer.py:1157] 2022-03-22 13:51:17,028 >>   Num examples = 382\n",
      "[INFO|trainer.py:1158] 2022-03-22 13:51:17,028 >>   Num Epochs = 3\n",
      "[INFO|trainer.py:1159] 2022-03-22 13:51:17,028 >>   Instantaneous batch size per device = 1\n",
      "[INFO|trainer.py:1160] 2022-03-22 13:51:17,028 >>   Total train batch size (w. parallel, distributed & accumulation) = 2\n",
      "[INFO|trainer.py:1161] 2022-03-22 13:51:17,028 >>   Gradient Accumulation steps = 1\n",
      "[INFO|trainer.py:1162] 2022-03-22 13:51:17,028 >>   Total optimization steps = 573\n",
      "  0%|                                                   | 0/573 [00:00<?, ?it/s]/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
      "{'loss': 3.3475, 'learning_rate': 6.369982547993019e-06, 'epoch': 2.62}         \n",
      " 87%|███████████████████████████████████▊     | 500/573 [04:13<00:37,  1.94it/s][INFO|trainer.py:1885] 2022-03-22 13:55:30,535 >> Saving model checkpoint to output/checkpoint-500\n",
      "[INFO|configuration_utils.py:351] 2022-03-22 13:55:30,536 >> Configuration saved in output/checkpoint-500/config.json\n",
      "[INFO|modeling_utils.py:889] 2022-03-22 13:55:31,036 >> Model weights saved in output/checkpoint-500/pytorch_model.bin\n",
      "[INFO|tokenization_utils_base.py:1924] 2022-03-22 13:55:31,036 >> tokenizer config file saved in output/checkpoint-500/tokenizer_config.json\n",
      "[INFO|tokenization_utils_base.py:1930] 2022-03-22 13:55:31,036 >> Special tokens file saved in output/checkpoint-500/special_tokens_map.json\n",
      "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
      "100%|█████████████████████████████████████████| 573/573 [04:53<00:00,  1.97it/s][INFO|trainer.py:1352] 2022-03-22 13:56:10,276 >> \n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "{'train_runtime': 293.2478, 'train_samples_per_second': 1.954, 'epoch': 3.0}    \n",
      "100%|█████████████████████████████████████████| 573/573 [04:53<00:00,  1.95it/s]\n",
      "[INFO|trainer.py:1885] 2022-03-22 13:56:10,370 >> Saving model checkpoint to output\n",
      "[INFO|configuration_utils.py:351] 2022-03-22 13:56:10,371 >> Configuration saved in output/config.json\n",
      "[INFO|modeling_utils.py:889] 2022-03-22 13:56:10,784 >> Model weights saved in output/pytorch_model.bin\n",
      "[INFO|tokenization_utils_base.py:1924] 2022-03-22 13:56:10,784 >> tokenizer config file saved in output/tokenizer_config.json\n",
      "[INFO|tokenization_utils_base.py:1930] 2022-03-22 13:56:10,784 >> Special tokens file saved in output/special_tokens_map.json\n",
      "[INFO|trainer_pt_utils.py:907] 2022-03-22 13:56:10,855 >> ***** train metrics *****\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   epoch                      =        3.0\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   init_mem_cpu_alloc_delta   =     1315MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   init_mem_cpu_peaked_delta  =      138MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   init_mem_gpu_alloc_delta   =      319MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   init_mem_gpu_peaked_delta  =        0MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_mem_cpu_alloc_delta  =     2602MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_mem_cpu_peaked_delta =      362MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_mem_gpu_alloc_delta  =      946MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_mem_gpu_peaked_delta =     2373MB\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_runtime              = 0:04:53.24\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_samples              =        382\n",
      "[INFO|trainer_pt_utils.py:912] 2022-03-22 13:56:10,855 >>   train_samples_per_second   =      1.954\n"
     ]
    }
   ],
   "source": [
    "!{cmd}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FKl2zr2Gm2Db"
   },
   "source": [
    "Although, Huggingface provides a run_generation.py file for language generation. Running it from a command (as it takes the input), makes it load the model and the tokenizer everytime you run the file which slows downs generation. To reduce the I/O overhead, I have restructured the run_generation.py file in the following code which only loads the model and tokenizer once in a model and a tokenizer object and we can use these objects to generate text over and over again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "M7_-L9gGziiv"
   },
   "outputs": [],
   "source": [
    "from transformers import GPT2LMHeadModel, GPT2Tokenizer\n",
    "\n",
    "def get_model_tokenizer(weights_dir, device = 'cuda'):\n",
    "    print(\"Loading Model ...\")\n",
    "    model = GPT2LMHeadModel.from_pretrained(weights_dir)\n",
    "    model.to('cuda')\n",
    "    print(\"Model Loaded ...\")\n",
    "    tokenizer = GPT2Tokenizer.from_pretrained(weights_dir)\n",
    "    return model, tokenizer\n",
    "\n",
    "def generate_messages(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    prompt_text,\n",
    "    stop_token,\n",
    "    length,\n",
    "    num_return_sequences,\n",
    "    temperature = 0.7,\n",
    "    k=20,\n",
    "    p=0.9,\n",
    "    repetition_penalty = 1.0,\n",
    "    device = 'cuda'\n",
    "):\n",
    "\n",
    "    MAX_LENGTH = int(10000)\n",
    "    def adjust_length_to_model(length, max_sequence_length):\n",
    "        if length < 0 and max_sequence_length > 0:\n",
    "            length = max_sequence_length\n",
    "        elif 0 < max_sequence_length < length:\n",
    "            length = max_sequence_length  # No generation bigger than model size\n",
    "        elif length < 0:\n",
    "            length = MAX_LENGTH  # avoid infinite loop\n",
    "        return length\n",
    "        \n",
    "    length = adjust_length_to_model(length=length, max_sequence_length=model.config.max_position_embeddings)\n",
    "\n",
    "    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors=\"pt\")\n",
    "\n",
    "    encoded_prompt = encoded_prompt.to(device)\n",
    "\n",
    "    output_sequences = model.generate(\n",
    "            input_ids=encoded_prompt,\n",
    "            max_length=length + len(encoded_prompt[0]),\n",
    "            temperature=temperature,\n",
    "            top_k=k,\n",
    "            top_p=p,\n",
    "            repetition_penalty=repetition_penalty,\n",
    "            do_sample=True,\n",
    "            num_return_sequences=num_return_sequences,\n",
    "        )\n",
    "\n",
    "    if len(output_sequences.shape) > 2:\n",
    "        output_sequences.squeeze_()\n",
    "\n",
    "    generated_sequences = []\n",
    "\n",
    "    for generated_sequence_idx, generated_sequence in enumerate(output_sequences):\n",
    "        # print(\"=== GENERATED SEQUENCE {} ===\".format(generated_sequence_idx + 1))\n",
    "        generated_sequence = generated_sequence.tolist()\n",
    "\n",
    "        # Decode text\n",
    "        text = tokenizer.decode(generated_sequence, clean_up_tokenization_spaces=True)\n",
    "\n",
    "        # Remove all text after the stop token\n",
    "        text = text[: text.find(stop_token) if stop_token else None]\n",
    "\n",
    "        # Add the prompt at the beginning of the sequence. Remove the excess text that was used for pre-processing\n",
    "        total_sequence = (\n",
    "            prompt_text + text[len(tokenizer.decode(encoded_prompt[0], clean_up_tokenization_spaces=True)) :]\n",
    "        )\n",
    "\n",
    "        generated_sequences.append(total_sequence)\n",
    "    return generated_sequences\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4297,
     "status": "ok",
     "timestamp": 1647867395691,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 240
    },
    "id": "Q3fwyPATznLp",
    "outputId": "22414c14-2c31-4109-ae61-5d0bdbacffa8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Model ...\n",
      "Model Loaded ...\n"
     ]
    }
   ],
   "source": [
    "model, tokenizer = get_model_tokenizer(weights_dir, device = 'cuda')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "qqpCwzTUz0wZ"
   },
   "outputs": [],
   "source": [
    "temperature = 1.0\n",
    "k = 400\n",
    "p = 0.9\n",
    "repetition_penalty = 1.0\n",
    "num_return_sequences = 5\n",
    "length = 1000\n",
    "stop_token = '|EndOfText|'\n",
    "prompt_text = \"this is\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 27760,
     "status": "ok",
     "timestamp": 1647867432090,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 240
    },
    "id": "YQv52LcT0Iyi",
    "outputId": "10a40532-a5b5-4767-d669-1c916d281294"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.84 s, sys: 50.6 ms, total: 6.89 s\n",
      "Wall time: 6.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['this is the good health of women in this country it is the same as the average man should have ',\n",
       " 'this is only going to change my mind if i feel punished for it and something i will do differently for the better because my precious daughter isnt hurt ',\n",
       " 'this is too early to tell this just how dumb it was to have that information but i am completely numb to it so i was impressed with this fact that it was presented in my own way to feel accepted in my world ',\n",
       " 'this is useful for helping reduce headaches and in some cases, pain to patients of any age group who may not have had the stress disorder ',\n",
       " 'this is a gentle reminder that you will always enjoy taking part in what you do with your child i feel that i am giving her pleasure which is to do justice for her the way i did before her so that her life will never be ruined like this again in the long run ']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "generate_messages(\n",
    "    model,\n",
    "    tokenizer,\n",
    "    prompt_text,\n",
    "    stop_token,\n",
    "    length,\n",
    "    num_return_sequences,\n",
    "    temperature=temperature,\n",
    "    k=k,\n",
    "    p=p,\n",
    "    repetition_penalty=repetition_penalty\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "JpXtDNZ4dunQ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "ckpt_dir = 'emotions_distilgpt2_model'\n",
    "try:\n",
    "    os.mkdir(ckpt_dir)\n",
    "except FileExistsError: pass\n",
    "\n",
    "output_model_file = os.path.join(ckpt_dir, 'torch_distilgpt2_news.pt')\n",
    "torch.save(model, output_model_file)\n",
    "tokenizer.save.save_vocabulary(ckpt_dir)\n",
    "print('Saved')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Fine-Tune DistilGPT2 and Generate Text",
   "provenance": [
    {
     "file_id": "https://github.com/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb",
     "timestamp": 1647867558304
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}