DistriBERT_for_SA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# <span style=\"color:blue\">**Project Overview: DistriBERT for Sentiment Analysis**</span>\n",
        "\n",
        "## <span style=\"color:green\">**Introduction**</span>\n",
        "\n",
        "This project focuses on implementing a sentiment analysis (SA) model using DistilBERT, a distilled version of BERT, for analyzing sentiment in textual data.\n",
        "\n",
        "## <span style=\"color:green\">**Objectives**</span>\n",
        "\n",
        "### <span style=\"color:purple\">**Main Objectives**</span>\n",
        "- <span style=\"color:purple\">**Implementation:**</span> Implement DistilBERT for sentiment analysis using the Hugging Face Transformers library.\n",
        "- <span style=\"color:purple\">**Evaluation:**</span> Evaluate the model's performance using metrics such as accuracy, precision, recall, and F1-score.\n",
        "- <span style=\"color:purple\">**Cross-Validation:**</span> Perform cross-validation to ensure the robustness and generalization of the model.\n",
        "\n",
        "### <span style=\"color:purple\">**Specific Goals**</span>\n",
        "- <span style=\"color:purple\">**Dataset Preparation:**</span> Load and preprocess the Amazon Reviews dataset for training and validation.\n",
        "- <span style=\"color:purple\">**Model Training:**</span> Train the DistilBERT model for sentiment classification.\n",
        "- <span style=\"color:purple\">**Evaluation Metrics:**</span> Calculate and report metrics to assess model performance.\n",
        "- <span style=\"color:purple\">**Cross-Validation:**</span> Conduct cross-validation to validate model performance across different folds.\n",
        "\n",
        "## <span style=\"color:green\">**Implementation Steps**</span>\n",
        "\n",
        "### <span style=\"color:orange\">**Install Necessary Libraries**</span>\n",
        "- Install PyTorch, Transformers, Datasets, and other required libraries for model development.\n",
        "\n",
        "### <span style=\"color:orange\">**Import Statements**</span>\n",
        "- Import essential libraries such as PyTorch, Hugging Face Transformers, and scikit-learn for dataset handling, model training, and evaluation.\n",
        "\n",
        "### <span style=\"color:orange\">**Dataset Loading and Preprocessing**</span>\n",
        "- Load a subset of the Amazon Reviews dataset and preprocess it using DistilBERT's tokenizer.\n",
        "- Split the dataset into training and validation sets.\n",
        "\n",
        "### <span style=\"color:orange\">**Model Initialization**</span>\n",
        "- Initialize the DistilBERT model for sequence classification.\n",
        "- Define training arguments such as batch size, number of epochs, and logging configurations.\n",
        "\n",
        "### <span style=\"color:orange\">**Model Training**</span>\n",
        "- Train the DistilBERT model using the Trainer class from Transformers.\n",
        "- Save the trained model and evaluation metrics to files for further analysis.\n",
        "\n",
        "### <span style=\"color:orange\">**Model Evaluation**</span>\n",
        "- Evaluate the trained model on the validation dataset.\n",
        "- Calculate accuracy, precision, recall, and F1-score to assess model performance.\n",
        "\n",
        "### <span style=\"color:orange\">**Cross-Validation**</span>\n",
        "- Implement cross-validation to validate the model across multiple folds.\n",
        "- Compute average metrics like accuracy, precision, recall, and F1-score across cross-validation runs.\n",
        "- Save cross-validation results to a CSV file.\n",
        "\n",
        "## <span style=\"color:green\">**Conclusion**</span>\n",
        "\n",
        "This project aims to demonstrate the effectiveness of DistilBERT for sentiment analysis tasks. By following these steps, we can build, train, evaluate, and validate a robust sentiment analysis model using state-of-the-art techniques.\n"
      ],
      "metadata": {
        "id": "oB9oKsmYV4kX"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "sBpvOkCtWYMI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Qxr1DI0JWYWH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:blue\">Step 1: Install Necessary Libraries</span>\n",
        "\n",
        "- **Purpose:**\n",
        "  - The goal of this step is to install essential Python libraries required for the project. These libraries are foundational for tasks such as data preprocessing, model training, evaluation, and optimization.\n",
        "\n",
        "- **Actions:**\n",
        "  - **PyTorch Libraries:** Install `torch`, `torchvision`, and `torchaudio` libraries. These are core components for building and training neural networks using PyTorch framework. `torchvision` provides utilities for computer vision tasks, while `torchaudio` supports audio data processing.\n",
        "  \n",
        "  - **Transformers Library:** Install `transformers` from Hugging Face. This library is crucial for leveraging pre-trained transformer models like BERT, DistilBERT, and others. It simplifies model loading, fine-tuning, and inference.\n",
        "  \n",
        "  - **Datasets Library:** Install `datasets` library. It provides access to various datasets commonly used in machine learning research and applications. This facilitates seamless integration of datasets into your training and evaluation pipelines.\n",
        "  \n",
        "  - **Accelerate Library:** Install `accelerate`, ensuring it is version 0.21.0 or higher. This library optimizes PyTorch training, particularly beneficial for scaling training on multiple GPUs efficiently.\n",
        "\n",
        "- **Code Example:**\n",
        "  ```python\n",
        "  # Install PyTorch and related libraries\n",
        "  !pip install torch torchvision torchaudio\n",
        "  \n",
        "  # Install Hugging Face Transformers and Datasets\n",
        "  !pip install transformers datasets\n",
        "  \n",
        "  # Install Accelerate library (version 0.21.0 or higher)\n",
        "  !pip install accelerate>=0.21.0"
      ],
      "metadata": {
        "id": "gSYIaTKWWY2J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install necessary libraries"
      ],
      "metadata": {
        "id": "4jr5RlQdATpx"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install torch torchvision torchaudio transformers datasets  # Install required libraries"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9qJtj3MaCe-L",
        "outputId": "a5343870-b600-437d-f330-ef17826b9f80"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.3.0+cu121)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (0.18.0+cu121)\n",
            "Requirement already satisfied: torchaudio in /usr/local/lib/python3.10/dist-packages (2.3.0+cu121)\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.41.2)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.20.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.15.4)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.12.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2023.6.0)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch) (2.20.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch) (12.1.105)\n",
            "Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch) (2.3.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.5.82)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision) (1.25.2)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision) (9.4.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.5.15)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.4)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (16.1.0)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets) (0.6)\n",
            "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.8)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.0.3)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.4.1)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.16)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.9.5)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.6.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (2.1.5)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)\n",
            "Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "PRPCcqvHCfBg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "_2XLGuqQC41y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install accelerate>=0.21.0  # Install accelerate library with version >=0.21.0"
      ],
      "metadata": {
        "id": "uzqoRxLwCfEZ"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "0eLcOqbdCfGy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "qg1YecJuC8QB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### <span style=\"color:blue\">Step 2: Import Statements</span>\n",
        "\n",
        "#### <span style=\"color:green\">Library Import</span>\n",
        "\n",
        "- **PyTorch**: Import PyTorch (`import torch`) to leverage its capabilities for deep learning model training and operations.\n",
        "- **NumPy**: Import NumPy (`import numpy as np`) for efficient numerical computations and array operations.\n",
        "- **Pandas**: Import Pandas (`import pandas as pd`) for handling and manipulating data in tabular form.\n",
        "- **Hugging Face Transformers**: Import necessary classes from Hugging Face Transformers (`from transformers import DistilBertTokenizer, DistilBertForSequenceClassification, Trainer, TrainingArguments`) for using DistilBERT model and related components.\n",
        "- **Hugging Face Datasets**: Import the function to load datasets from Hugging Face Datasets (`from datasets import load_dataset`) to conveniently access the Amazon Reviews dataset.\n",
        "- **Scikit-learn Metrics**: Import evaluation metrics (`from sklearn.metrics import accuracy_score, precision_recall_fscore_support`) from scikit-learn to evaluate model performance.\n",
        "\n",
        "#### <span style=\"color:green\">Purpose</span>\n",
        "\n",
        "- **PyTorch**: Utilize PyTorch for building and training neural network models, including DistilBERT.\n",
        "- **NumPy**: Leverage NumPy for numerical computations needed during data preprocessing and evaluation.\n",
        "- **Pandas**: Use Pandas for handling and manipulating structured data, including loading datasets and storing results.\n",
        "- **Hugging Face Transformers**: Access the DistilBERT model and utilities for tokenization, training, and evaluation.\n",
        "- **Hugging Face Datasets**: Load the Amazon Reviews dataset using a convenient interface provided by Hugging Face.\n",
        "- **Scikit-learn Metrics**: Compute evaluation metrics such as accuracy, precision, recall, and F1-score to assess model performance.\n",
        "\n",
        "#### <span style=\"color:green\">Integration</span>\n",
        "\n",
        "- **Integration Strategy**: Integrate these libraries and modules to streamline the development and evaluation of the sentiment analysis model using DistilBERT.\n",
        "- **Compatibility**: Ensure compatibility and functionality across different components, facilitating efficient data handling, model training, and performance evaluation.\n",
        "\n",
        "---\n",
        "\n",
        "These import statements are crucial for setting up the environment and accessing necessary tools and utilities to proceed with the development of the DistriBERT model for sentiment analysis on the Amazon Reviews dataset."
      ],
      "metadata": {
        "id": "m60PgoIkapJB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import statements"
      ],
      "metadata": {
        "id": "WkiJePOAC8Sm"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch  # Import PyTorch library"
      ],
      "metadata": {
        "id": "-pJ_gXVHC8Ub"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np  # Import NumPy library for numerical computations"
      ],
      "metadata": {
        "id": "CLG83hc3KJrs"
      },
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd  # Import pandas for handling dataframes"
      ],
      "metadata": {
        "id": "-AXFMDOjGlTJ"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import DistilBertTokenizer, DistilBertForSequenceClassification, Trainer, TrainingArguments  # Import classes from Hugging Face Transformers\n"
      ],
      "metadata": {
        "id": "PPClEp9RCfJb"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset  # Import function to load datasets from Hugging Face Datasets"
      ],
      "metadata": {
        "id": "f7S7RicE_pD6"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support  # Import evaluation metrics from scikit-learn"
      ],
      "metadata": {
        "id": "r1-khOeg_Q2p"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "MDi885I9-eo4"
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "vLqyo6Ea-LrB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 3: Dataset Loading and Preprocessing</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - This step involves loading and preprocessing a subset of the Amazon Reviews dataset for training and validation purposes.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Dataset Loading:</span>**\n",
        "    - Load a smaller subset of the Amazon Reviews dataset using the `load_dataset` function from the `datasets` module. Specify the dataset name (`\"amazon_polarity\"`) and the split (`\"train[:100]\"`) to load 100 samples for initial testing.\n",
        "\n",
        "  - **<span style=\"color:blue\">Data Preprocessing:</span>**\n",
        "    - **<span style=\"color:green\">Tokenization:</span>** Use the DistilBERT tokenizer (`DistilBertTokenizer`) to tokenize and encode the dataset. Set parameters such as `truncation=True` for handling long sequences and `padding='max_length'` to ensure uniform sequence length.\n",
        "\n",
        "  - **<span style=\"color:blue\">Dataset Format:</span>**\n",
        "    - Convert tokenized datasets into a format compatible with PyTorch (`type='torch'`). Specify columns to include (`['input_ids', 'attention_mask', 'label']`) for model input (input IDs and attention masks) and labels (sentiment labels).\n",
        "\n",
        "  - **<span style=\"color:blue\">Splitting into Training and Validation Sets:</span>**\n",
        "    - Determine the sizes for the training (`train_size`) and validation (`val_size`) datasets based on a predefined split ratio (e.g., 80% for training, 20% for validation).\n",
        "    \n",
        "    - Shuffle and select samples for the training dataset (`train_dataset`) to ensure randomization and avoid bias.\n",
        "    \n",
        "    - Select samples for the validation dataset (`val_dataset`) using the remaining samples after selecting the training set.\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Dataset Size Considerations:</span>** Adjust the subset size (`100` samples in this case) based on computational resources and initial testing requirements.\n",
        "  \n",
        "  - **<span style=\"color:green\">Data Integrity:</span>** Ensure data integrity by verifying the successful loading and preprocessing of the dataset before proceeding to the next steps.\n",
        "  \n",
        "  - **<span style=\"color:green\">Error Handling:</span>** Handle potential errors such as missing data or incompatible formats during dataset loading and preprocessing stages.\n",
        "  \n",
        "  - **<span style=\"color:green\">Documentation:</span>** Refer to documentation and examples provided by Hugging Face and other relevant sources for detailed usage and parameter configurations of the `datasets` and `transformers` libraries.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "K4h61F_Mbtbg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load a smaller subset of the Amazon Reviews dataset for initial testing"
      ],
      "metadata": {
        "id": "DUI827YW-Lto"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "dataset = load_dataset(\"amazon_polarity\", split=\"train[:100]\")  # Load a subset of Amazon Reviews dataset"
      ],
      "metadata": {
        "id": "TK-05u3WDMkD"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "677kLf3MDMnD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "DBid9PB3b2U4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenizer initialization"
      ],
      "metadata": {
        "id": "RR_Y_jKQDMoE"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')  # Initialize DistilBERT tokenizer"
      ],
      "metadata": {
        "id": "a4E49truDMqc"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "TOHohtn0DMs-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Function for preprocessing data"
      ],
      "metadata": {
        "id": "Auf5rgztDMvo"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def preprocess(example):\n",
        "    return tokenizer(example['content'], truncation=True, padding='max_length')  # Preprocess dataset examples"
      ],
      "metadata": {
        "id": "jbO5RHmQDZmE"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "2yxMlKJQDZph"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenize datasets in batches"
      ],
      "metadata": {
        "id": "j1KZhDJ-DZxZ"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenized_datasets = dataset.map(preprocess, batched=True)  # Tokenize dataset in batches"
      ],
      "metadata": {
        "id": "F51iaqnDDZ0b"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenized_datasets.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label'])  # Set dataset format for PyTorch\n"
      ],
      "metadata": {
        "id": "zf64cb9ZDZ3K"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "NktYs5cwDnXi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "7v7Bku4UcwwJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 4: Split Datasets into Training and Validation Sets</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - Split the preprocessed dataset into training and validation sets to facilitate model training and evaluation.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Dataset Splitting:</span>**\n",
        "    - Calculate the sizes for training (`train_size`) and validation (`val_size`) datasets based on a specified ratio (e.g., 80% training, 20% validation).\n",
        "    \n",
        "  - **<span style=\"color:blue\">Shuffling:</span>**\n",
        "    - Use the `shuffle` method (`tokenized_datasets.shuffle`) with a seed for reproducibility to randomize the dataset before splitting.\n",
        "    \n",
        "  - **<span style=\"color:blue\">Selection:</span>**\n",
        "    - Select data ranges (`range`) for training and validation sets using the `select` method (`tokenized_datasets.select`).\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Data Distribution:</span>** Ensure a representative distribution of data across training and validation sets to maintain model performance and generalization.\n",
        "  \n",
        "  - **<span style=\"color:green\">Shuffling:</span>** Shuffle the dataset to randomize the order of examples before splitting to prevent any inherent bias in the data sequence.\n",
        "  \n",
        "  - **<span style=\"color:green\">Seed Selection:</span>** Use a consistent seed (e.g., `seed=42`) for reproducibility in dataset shuffling and selection across different runs or environments.\n",
        "  \n",
        "  - **<span style=\"color:green\">Validation Set:</span>** Validate the selected validation set range (`train_size` to `train_size + val_size`) to ensure correct separation from the training data.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "SDnw7AtecxD4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Split datasets into training and validation sets"
      ],
      "metadata": {
        "id": "MZ11q1udDnaU"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_size = int(0.8 * len(tokenized_datasets))  # Calculate size of training dataset"
      ],
      "metadata": {
        "id": "UsgG_fowDnfY"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "val_size = len(tokenized_datasets) - train_size  # Calculate size of validation dataset"
      ],
      "metadata": {
        "id": "UA9YS3LHDnhy"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_dataset = tokenized_datasets.shuffle(seed=42).select(range(train_size))  # Shuffle and select training dataset\n"
      ],
      "metadata": {
        "id": "8reEAvuODnl6"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "val_dataset = tokenized_datasets.shuffle(seed=42).select(range(train_size, train_size + val_size))  # Select validation dataset\n"
      ],
      "metadata": {
        "id": "DalT6UXxDnpC"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "r5fDASKsDnsK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "P6iAeLJKc6iJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 5: Initialize and Train DistilBERT Model</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - Initialize the DistilBERT model for sequence classification and train it on the prepared training dataset.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Model Initialization:</span>**\n",
        "    - Initialize the DistilBERT model for sequence classification using the `DistilBertForSequenceClassification.from_pretrained` method with appropriate parameters (e.g., `num_labels`).\n",
        "\n",
        "  - **<span style=\"color:blue\">Training Arguments:</span>**\n",
        "    - Define training arguments such as `TrainingArguments` specifying parameters like `output_dir`, `num_train_epochs`, `per_device_train_batch_size`, `per_device_eval_batch_size`, `warmup_steps`, `weight_decay`, `logging_dir`, `logging_steps`, `evaluation_strategy`, and `save_strategy`.\n",
        "\n",
        "  - **<span style=\"color:blue\">Trainer Initialization:</span>**\n",
        "    - Initialize the `Trainer` object with the defined DistilBERT model, training arguments, and the prepared training dataset (`train_dataset`) and validation dataset (`val_dataset`).\n",
        "\n",
        "  - **<span style=\"color:blue\">Model Training:</span>**\n",
        "    - Use the `trainer.train()` method to commence training the initialized DistilBERT model on the training dataset.\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Model Selection:</span>** Choose an appropriate pre-trained DistilBERT model configuration (`distilbert-base-uncased`, etc.) for sequence classification based on the task requirements.\n",
        "  \n",
        "  - **<span style=\"color:green\">Training Configuration:</span>** Configure training parameters (`num_train_epochs`, batch sizes, logging settings, etc.) to optimize model performance and monitor training progress.\n",
        "  \n",
        "  - **<span style=\"color:green\">Evaluation Strategy:</span>** Specify the `evaluation_strategy` to evaluate the model at the end of each epoch for performance assessment on the validation dataset.\n",
        "  \n",
        "  - **<span style=\"color:green\">Save Strategy:</span>** Define the `save_strategy` to save the model checkpoints at the end of each epoch for potential further evaluation or deployment.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "VpLffZIoc62j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load pre-trained DistilBERT model for sequence classification"
      ],
      "metadata": {
        "id": "x5OKz92WD28Q"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)  # Initialize DistilBERT for sequence classification\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9EmEJGrND2_o",
        "outputId": "8fc67c11-8cbe-4b07-db61-132e01d37786"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "5sZCYaHRD3Ct"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define training arguments, adjusting batch sizes and epochs for faster execution"
      ],
      "metadata": {
        "id": "Y-JyBLdOD3Fr"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "training_args = TrainingArguments(\n",
        "    output_dir='./results',  # Directory to save training results\n",
        "    num_train_epochs=1,  # Number of training epochs\n",
        "    per_device_train_batch_size=4,  # Batch size for training\n",
        "    per_device_eval_batch_size=4,  # Batch size for evaluation\n",
        "    warmup_steps=500,  # Number of warmup steps\n",
        "    weight_decay=0.01,  # Weight decay coefficient\n",
        "    logging_dir='./logs',  # Directory to save logs\n",
        "    logging_steps=10,  # Log every 10 steps\n",
        "    evaluation_strategy='epoch',  # Evaluate at the end of each epoch\n",
        "    save_strategy='epoch',  # Save model at the end of each epoch\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vs1iUx48D3Ig",
        "outputId": "e725fd5e-ada5-46ea-b4dc-53a5cc985b43"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py:1474: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "YUSe-Dv-D3Jl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize Trainer with defined model, training arguments, and datasets"
      ],
      "metadata": {
        "id": "5OkzTjoMD3MJ"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "trainer = Trainer(\n",
        "    model=model,  # Pass model to Trainer\n",
        "    args=training_args,  # Pass training arguments to Trainer\n",
        "    train_dataset=train_dataset,  # Pass training dataset to Trainer\n",
        "    eval_dataset=val_dataset,  # Pass validation dataset to Trainer\n",
        ")"
      ],
      "metadata": {
        "id": "_Z0yXOP1EMpx"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "y0heiZhAEMtU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the model"
      ],
      "metadata": {
        "id": "HKGxlzV6EMwR"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "trainer.train()  # Train the model"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        },
        "id": "tb_N95LpEMzV",
        "outputId": "20d05b5b-ef03-4067-9955-316670b55d84"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='20' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [20/20 06:46, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.694200</td>\n",
              "      <td>0.694923</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TrainOutput(global_step=20, training_loss=0.6982547283172608, metrics={'train_runtime': 436.1211, 'train_samples_per_second': 0.183, 'train_steps_per_second': 0.046, 'total_flos': 10597391892480.0, 'train_loss': 0.6982547283172608, 'epoch': 1.0})"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "9oT6W2NlEM2Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "hEsZ8ypyEM42"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 6: Evaluate the Trained Model</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - Evaluate the performance of the trained DistilBERT model on the validation dataset using evaluation metrics such as accuracy, precision, recall, and F1-score.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Model Loading:</span>**\n",
        "    - Load the trained DistilBERT model using `DistilBertForSequenceClassification.from_pretrained` from the saved directory or checkpoint.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Prediction:</span>**\n",
        "    - Use the loaded model to make predictions on the validation dataset (`val_dataset`) using `trainer.predict()` method.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Metrics Calculation:</span>**\n",
        "    - Compute evaluation metrics such as accuracy, precision, recall, and F1-score using `accuracy_score`, `precision_recall_fscore_support`, or other appropriate functions from scikit-learn.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Results Display:</span>**\n",
        "    - Print or display the calculated evaluation metrics to assess the model's performance on the validation dataset.\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Model Loading:</span>** Ensure the correct path or directory is provided to load the trained model from the saved checkpoint or directory.\n",
        "  \n",
        "  - **<span style=\"color:green\">Prediction:</span>** Utilize the `trainer.predict()` method to obtain model predictions on the validation dataset efficiently.\n",
        "  \n",
        "  - **<span style=\"color:green\">Metric Selection:</span>** Select appropriate evaluation metrics (`accuracy`, `precision`, `recall`, `F1-score`) based on the task requirements and dataset characteristics.\n",
        "  \n",
        "  - **<span style=\"color:green\">Performance Assessment:</span>** Interpret and analyze the evaluation metrics to assess the model's performance and identify areas for potential improvement.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "sOLFWAzedQHd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate the trained model on validation dataset"
      ],
      "metadata": {
        "id": "FAOgtwtUEZ7U"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "predictions = trainer.predict(val_dataset)  # Make predictions on validation dataset"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "UNkGPWDkEZ-g",
        "outputId": "0553541e-17bb-4a19-92d0-c6389f62c6f4"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "preds = predictions.predictions.argmax(-1)  # Get predicted labels"
      ],
      "metadata": {
        "id": "zNhLgi0VEaBP"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "labels = predictions.label_ids  # Get true labels"
      ],
      "metadata": {
        "id": "VriWEgknEaD7"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "lh52JhR1EaGw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate evaluation metrics (accuracy, precision, recall, F1-score)"
      ],
      "metadata": {
        "id": "FJbNe6AgEiv8"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "accuracy = accuracy_score(labels, preds)  # Calculate accuracy"
      ],
      "metadata": {
        "id": "j0uKjeUeEixV"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')  # Calculate precision, recall, F1-score\n"
      ],
      "metadata": {
        "id": "hNq0pLQgEiyz"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "S464kDtlEi2Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print evaluation metrics"
      ],
      "metadata": {
        "id": "gKjL6OrMEi5Q"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"Accuracy: {accuracy:.4f}\")  # Print accuracy"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EETfqb8XEi9M",
        "outputId": "ed107683-0c55-4650-ea4a-fc69200a418d"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 0.6000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "vUoJYaWjEwwv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"Precision: {precision:.4f}, Recall: {recall:.4f}, F1: {f1:.4f}\")  # Print precision, recall, F1-score"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z-i-G4EUEw0s",
        "outputId": "c9355a9c-6a88-4068-dc84-99f0219a8d91"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Precision: 1.0000, Recall: 0.1111, F1: 0.2000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "0Re2QtgdEw3l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "9DJqU_uwFqHk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 7: Save Evaluation Metrics and Predictions</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - Save the evaluation metrics (accuracy, precision, recall, F1-score) and model predictions (true labels and predicted labels) to files for further analysis and reporting.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Metrics Saving:</span>**\n",
        "    - Create a DataFrame or data structure to store evaluation metrics (accuracy, precision, recall, F1-score) calculated during model evaluation.\n",
        "    - Save the evaluation metrics to a CSV file using `to_csv()` method of Pandas DataFrame.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Predictions Saving:</span>**\n",
        "    - Create a DataFrame to store true labels and predicted labels obtained from model predictions.\n",
        "    - Save the predictions DataFrame to a CSV file using `to_csv()` method of Pandas DataFrame.\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Metrics Storage:</span>** Ensure the CSV file path is correctly specified to save evaluation metrics for future reference or reporting.\n",
        "  \n",
        "  - **<span style=\"color:green\">Predictions Storage:</span>** Save true labels and predicted labels in a structured format to facilitate comparison and analysis.\n",
        "  \n",
        "  - **<span style=\"color:green\">File Naming:</span>** Choose meaningful names for CSV files (e.g., `evaluation_metrics.csv`, `predictions.csv`) to easily identify stored data.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "PqmNmpCFdiN6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save evaluation metrics to CSV file"
      ],
      "metadata": {
        "id": "GnTmFumMFqJL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_metrics_df = pd.DataFrame({  # Create dataframe for evaluation metrics\n",
        "    'Metric': ['Accuracy', 'Precision', 'Recall', 'F1-score'],  # Define metric names\n",
        "    'Score': [accuracy, precision, recall, f1]  # Define corresponding scores\n",
        "})"
      ],
      "metadata": {
        "id": "EjoQkgZIFqKy"
      },
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "O2PwUCmcFqNw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_metrics_df.to_csv('./evaluation_metrics.csv', index=False)  # Save evaluation metrics to CSV file"
      ],
      "metadata": {
        "id": "ahx08Lq5FqPS"
      },
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "fznpmrtLFqQ1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Save predictions to CSV file"
      ],
      "metadata": {
        "id": "Z0JCA-eDGuLW"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "predictions_df = pd.DataFrame({  # Create dataframe for predictions\n",
        "    'True Labels': labels,  # True labels column\n",
        "    'Predicted Labels': preds  # Predicted labels column\n",
        "})"
      ],
      "metadata": {
        "id": "FykVstQoGuOd"
      },
      "execution_count": 58,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "yoY_Pq-HGuRn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "predictions_df.to_csv('./predictions.csv', index=False)  # Save predictions to CSV file"
      ],
      "metadata": {
        "id": "AQkw5j5oFqSM"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "9dH5OM9gG2XC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "n2sAxi4XG2e1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 8: Save and Load the Trained Model</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - Save the trained DistilBERT model to disk for future use and load it back into memory for inference or further training.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Model Saving:</span>**\n",
        "    - Use the `save_model()` method of the Trainer class to save the trained DistilBERT model to a specified directory.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Model Loading:</span>**\n",
        "    - Utilize the `from_pretrained()` method of DistilBertForSequenceClassification to load the saved model from the directory path.\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Model Storage:</span>** Ensure the directory path provided during model saving is accessible and descriptive.\n",
        "  \n",
        "  - **<span style=\"color:green\">Model Loading:</span>** Verify that the correct directory path is used to load the saved model back into memory.\n",
        "  \n",
        "  - **<span style=\"color:green\">Reusability:</span>** Saved models can be reused for inference on new data or continued training without retraining from scratch.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "QkyOSAEcd3oy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save the trained model"
      ],
      "metadata": {
        "id": "pyWsjyGTEw55"
      },
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "trainer.save_model('./distilbert-amazon-reviews')  # Save the trained model"
      ],
      "metadata": {
        "id": "u0yrn9d-E2qA"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "jvqTmENME2tU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the saved model"
      ],
      "metadata": {
        "id": "sEKzwUPmE2v-"
      },
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "loaded_model = DistilBertForSequenceClassification.from_pretrained('./distilbert-amazon-reviews')  # Load the saved model\n"
      ],
      "metadata": {
        "id": "DJGamuZxE22C"
      },
      "execution_count": 54,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "iGd1tdI3E9_p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "roECiA4TLX3C"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "6DFtBFzHKsXO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:orange\">Step 9: Perform Cross-Validation and Calculate Average Metrics</span>\n",
        "\n",
        "- **<span style=\"color:purple\">Purpose:</span>**\n",
        "  - Validate the performance of the DistilBERT model across multiple folds using cross-validation.\n",
        "  - Compute average evaluation metrics to assess the model's robustness and generalization.\n",
        "\n",
        "- **<span style=\"color:purple\">Actions:</span>**\n",
        "  - **<span style=\"color:blue\">Cross-Validation Setup:</span>**\n",
        "    - Define the number of cross-validation runs and initialize an empty list to store results.\n",
        "\n",
        "  - **<span style=\"color:blue\">Dataset Processing:</span>**\n",
        "    - Tokenize and preprocess the dataset for each cross-validation run to ensure consistency and fairness.\n",
        "\n",
        "  - **<span style=\"color:blue\">Model Training and Evaluation:</span>**\n",
        "    - Train the DistilBERT model and evaluate its performance on each fold of the cross-validation using predefined metrics.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Average Metrics Calculation:</span>**\n",
        "    - Calculate average metrics such as accuracy, precision, recall, and F1-score across all cross-validation runs.\n",
        "  \n",
        "  - **<span style=\"color:blue\">Results Saving:</span>**\n",
        "    - Save the cross-validation results to a CSV file for further analysis and comparison.\n",
        "\n",
        "- **<span style=\"color:purple\">Notes:</span>**\n",
        "  - **<span style=\"color:green\">Data Consistency:</span>** Ensure dataset preprocessing and model training procedures are consistent across all cross-validation folds.\n",
        "  \n",
        "  - **<span style=\"color:green\">Metric Interpretation:</span>** Interpret average metrics to gauge the model's performance across various data subsets.\n",
        "  \n",
        "  - **<span style=\"color:green\">Result Analysis:</span>** Use saved results to compare different models or parameter settings and inform future model improvements.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "ZHrWuGmHd8BS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to train and evaluate the model"
      ],
      "metadata": {
        "id": "dEePc-B7KsZL"
      },
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train_and_evaluate(train_dataset, val_dataset):\n",
        "    trainer = Trainer(\n",
        "        model=model,\n",
        "        args=training_args,\n",
        "        train_dataset=train_dataset,\n",
        "        eval_dataset=val_dataset\n",
        "    )\n",
        "\n",
        "    trainer.train()  # Train the model\n",
        "\n",
        "    predictions = trainer.predict(val_dataset)  # Make predictions on validation dataset\n",
        "    preds = predictions.predictions.argmax(-1)  # Get predicted labels\n",
        "    labels = predictions.label_ids  # Get true labels\n",
        "\n",
        "    accuracy = accuracy_score(labels, preds)  # Calculate accuracy\n",
        "    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')  # Calculate precision, recall, F1-score\n",
        "\n",
        "    return accuracy, precision, recall, f1  # Return evaluation metrics"
      ],
      "metadata": {
        "id": "p5bJvBrGKgxl"
      },
      "execution_count": 66,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "eMnD1xksKVet"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to perform cross-validation"
      ],
      "metadata": {
        "id": "sao9hme6MdIw"
      },
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def cross_validate(dataset, num_runs=5):\n",
        "    results = []  # Initialize an empty list to store results\n",
        "\n",
        "    for i in range(num_runs):\n",
        "        seed = 42 + i  # Define seed for reproducibility\n",
        "\n",
        "        # Preprocess datasets for current seed\n",
        "        tokenized_datasets = dataset.map(preprocess, batched=True)  # Tokenize and preprocess dataset\n",
        "\n",
        "        # Split datasets into training and validation sets\n",
        "        train_size = int(0.8 * len(tokenized_datasets))  # Calculate size of training dataset\n",
        "        val_size = len(tokenized_datasets) - train_size  # Calculate size of validation dataset\n",
        "        train_dataset = tokenized_datasets.shuffle(seed=seed).select(range(train_size))  # Shuffle and select training dataset\n",
        "        val_dataset = tokenized_datasets.shuffle(seed=seed).select(range(train_size, train_size + val_size))  # Select validation dataset\n",
        "\n",
        "        # Train and evaluate with current seed\n",
        "        accuracy, precision, recall, f1 = train_and_evaluate(train_dataset, val_dataset)  # Train and evaluate the model\n",
        "\n",
        "        # Collect results\n",
        "        results.append({\n",
        "            'Seed': seed,\n",
        "            'Accuracy': accuracy,\n",
        "            'Precision': precision,\n",
        "            'Recall': recall,\n",
        "            'F1-score': f1\n",
        "        })  # Append results for current run to the list\n",
        "\n",
        "    return results  # Return the collected results after all cross-validation runs"
      ],
      "metadata": {
        "id": "6e_OhJ7xMdMp"
      },
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "O1Eoq1jSMoSU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "XLiWAU-yMoXU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Perform cross-validation"
      ],
      "metadata": {
        "id": "p0mNWTyBMocb"
      },
      "execution_count": 69,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "num_runs = 5  # Number of cross-validation runs"
      ],
      "metadata": {
        "id": "7UCzeshJMdPe"
      },
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "cross_validation_results = cross_validate(dataset, num_runs)  # Execute cross-validation"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 466
        },
        "id": "CSrnhLP4NJ4a",
        "outputId": "590ac965-1ad7-4f65-a1a1-d741dd60e1f9"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='20' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [20/20 06:19, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.686400</td>\n",
              "      <td>0.693180</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='20' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [20/20 05:25, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.684000</td>\n",
              "      <td>0.685068</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='20' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [20/20 05:12, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.677300</td>\n",
              "      <td>0.678845</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='20' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [20/20 05:27, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.679500</td>\n",
              "      <td>0.681347</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='20' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [20/20 05:31, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.687100</td>\n",
              "      <td>0.677510</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "mjG61h8kNJ_E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate average metrics across runs"
      ],
      "metadata": {
        "id": "4qenHmyJNKAq"
      },
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avg_accuracy = np.mean([result['Accuracy'] for result in cross_validation_results])  # Calculate average accuracy"
      ],
      "metadata": {
        "id": "lUZVNqDnNKCc"
      },
      "execution_count": 73,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avg_precision = np.mean([result['Precision'] for result in cross_validation_results])  # Calculate average precision\n"
      ],
      "metadata": {
        "id": "jfPT5AteNKFO"
      },
      "execution_count": 74,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avg_recall = np.mean([result['Recall'] for result in cross_validation_results])  # Calculate average recall"
      ],
      "metadata": {
        "id": "fd-MZYYQNKIJ"
      },
      "execution_count": 75,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avg_f1 = np.mean([result['F1-score'] for result in cross_validation_results])  # Calculate average F1-score"
      ],
      "metadata": {
        "id": "uLmcMpUBNmdm"
      },
      "execution_count": 76,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "arABRb3zNmhb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print average metrics"
      ],
      "metadata": {
        "id": "2kOpt4QiNmlQ"
      },
      "execution_count": 77,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"Average Accuracy: {avg_accuracy:.4f}\")  # Print average accuracy\n",
        "print(f\"Average Precision: {avg_precision:.4f}\")  # Print average precision\n",
        "print(f\"Average Recall: {avg_recall:.4f}\")  # Print average recall\n",
        "print(f\"Average F1-score: {avg_f1:.4f}\")  # Print average F1-score"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LZOJdlq2NmoK",
        "outputId": "dfc946fe-fce9-4be2-d34e-667032d2459d"
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Accuracy: 0.7000\n",
            "Average Precision: 0.6754\n",
            "Average Recall: 0.6624\n",
            "Average F1-score: 0.6600\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "-_GtwNXJNmpz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "KjlrR3DpNtqs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Save average metrics to CSV file"
      ],
      "metadata": {
        "id": "B7iT--XxNtsi"
      },
      "execution_count": 79,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avg_metrics_df = pd.DataFrame({  # Create dataframe for average metrics\n",
        "    'Metric': ['Average Accuracy', 'Average Precision', 'Average Recall', 'Average F1-score'],  # Define metric names\n",
        "    'Score': [avg_accuracy, avg_precision, avg_recall, avg_f1]  # Define corresponding scores\n",
        "})"
      ],
      "metadata": {
        "id": "oRb85nhINtvw"
      },
      "execution_count": 80,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "RrwLSvV3N0lO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avg_metrics_df.to_csv('./average_metrics_cv_updated.csv', index=False)  # Save average metrics to CSV file"
      ],
      "metadata": {
        "id": "N9HC7oRgN1L2"
      },
      "execution_count": 81,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "dIJ-CHtZN1Nc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "jyVuNebmN1kL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div style=\"background-color:#f0f8ff; padding:10px;\">\n",
        "\n",
        "# <span style=\"color:blue\">Project Conclusion and Additional Notes</span>\n",
        "\n",
        "- **<span style=\"color:green\">Project Overview:</span>**\n",
        "  - This project focused on implementing a sentiment analysis model using DistilBERT, a lighter version of BERT, for analyzing sentiment in textual data from the Amazon Reviews dataset.\n",
        "\n",
        "- **<span style=\"color:green\">Key Objectives Achieved:</span>**\n",
        "  - **Implementation:** The project successfully implemented DistilBERT for sentiment analysis using the Hugging Face Transformers library. This involved leveraging pre-trained language models and fine-tuning them for sentiment classification.\n",
        "  \n",
        "  - **Evaluation:** The model's performance was evaluated using standard metrics such as accuracy, precision, recall, and F1-score. This evaluation provided insights into how well the model classified sentiment in the validation dataset.\n",
        "  \n",
        "  - **Cross-Validation:** To ensure the model's robustness and generalization, cross-validation was performed across multiple folds of the dataset. This helped validate that the model's performance was consistent and not overfitted to a specific subset of data.\n",
        "\n",
        "- **<span style=\"color:green\">Implementation Steps Recap:</span>**\n",
        "  - **Step 1: Install Necessary Libraries:** Essential libraries including PyTorch, Transformers, and Datasets were installed to facilitate model development and data handling.\n",
        "  \n",
        "  - **Step 2: Import Statements:** Necessary modules and libraries were imported, such as PyTorch for tensor computations, Hugging Face Transformers for leveraging pre-trained models, and scikit-learn for evaluation metrics.\n",
        "  \n",
        "  - **Step 3: Dataset Loading and Preprocessing:** A subset of the Amazon Reviews dataset was loaded and preprocessed using the DistilBERT tokenizer. This involved tokenizing the text data and preparing it for input into the model.\n",
        "  \n",
        "  - **Step 4: Split Datasets into Training and Validation Sets:** The dataset was split into training and validation sets to facilitate model training and evaluation. This step ensured that the model's performance could be assessed on unseen data.\n",
        "  \n",
        "  - **Step 5: Initialize and Train DistilBERT Model:** The DistilBERT model was initialized for sequence classification and trained using the Trainer class from Hugging Face Transformers. Training parameters such as batch size, number of epochs, and learning rate were configured to optimize model performance.\n",
        "  \n",
        "  - **Step 6: Evaluate the Trained Model:** The trained model was evaluated on the validation dataset using metrics like accuracy, precision, recall, and F1-score. These metrics provided quantitative measures of the model's performance in sentiment analysis tasks.\n",
        "  \n",
        "  - **Step 7: Save Evaluation Metrics and Predictions:** Evaluation metrics (e.g., accuracy, F1-score) and model predictions were saved to CSV files. This allowed for further analysis and comparison of different model configurations or datasets.\n",
        "  \n",
        "  - **Step 8: Save and Load the Trained Model:** After training, the DistilBERT model was saved to disk for future use or deployment. This step ensured that the trained model could be easily loaded for inference tasks without needing to retrain from scratch.\n",
        "  \n",
        "  - **Step 9: Perform Cross-Validation and Calculate Average Metrics:** Cross-validation was implemented to validate the model's performance across multiple folds of the dataset. Average metrics such as accuracy, precision, recall, and F1-score were computed to assess the model's consistency and generalization.\n",
        "\n",
        "- **<span style=\"color:green\">Conclusion:</span>**\n",
        "  - This project demonstrated the effectiveness of DistilBERT, a distilled version of BERT, for sentiment analysis tasks. By following the outlined steps, a robust sentiment analysis model was developed, trained, evaluated, and validated.\n",
        "  \n",
        "  - The project highlighted the importance of leveraging pre-trained models and fine-tuning them for specific tasks like sentiment analysis. This approach not only saves computational resources but also benefits from the extensive knowledge embedded in large-scale language models.\n",
        "  \n",
        "  - The evaluation metrics provided insights into the model's strengths and areas for improvement, helping guide future iterations or refinements of the sentiment analysis pipeline.\n",
        "  \n",
        "- **<span style=\"color:green\">Future Steps:</span>**\n",
        "  - **Model Fine-Tuning:** Consider fine-tuning the DistilBERT model with additional labeled data or adjusting hyperparameters to further improve performance in specific domains or datasets.\n",
        "  \n",
        "  - **Deployment:** Explore deployment options to integrate the trained model into real-world applications for sentiment analysis. This may involve deploying the model as a web service, embedding it into existing workflows, or deploying it on edge devices.\n",
        "  \n",
        "  - **Further Research:** Continue exploring advancements in transformer-based models and their applications in natural language processing tasks. This includes investigating new architectures, exploring multilingual models, or integrating domain-specific knowledge into model training.\n",
        "\n",
        "- **<span style=\"color:green\">Notes:</span>**\n",
        "  - **Dataset Considerations:** Ensure that data preprocessing steps are tailored to specific datasets and tasks to maintain model performance consistency across different domains or datasets.\n",
        "  \n",
        "  - **Model Evaluation:** Interpret evaluation metrics comprehensively to understand the strengths and limitations of the sentiment analysis model. Consider analyzing performance metrics in conjunction with qualitative assessments of model predictions.\n",
        "  \n",
        "  - **Documentation and Reproducibility:** Maintain detailed documentation of code, experimental setups, and results for reproducibility and future reference. This ensures that the project can be easily replicated or extended by other researchers or practitioners in the field.\n",
        "\n",
        "</div>"
      ],
      "metadata": {
        "id": "SvjXFP-veufR"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "tFvBi_wcets7"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}