Pre_processing_Data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "colab": {
      "name": "Pre-processing_Data.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gdg-ml-team/DevFest19/blob/master/Pre_processing_Data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yatqOJxQi4LH",
        "colab_type": "text"
      },
      "source": [
        "<h1>Pre-processing Data or Data Wrangling</h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U00A_Vtk9F1T",
        "colab_type": "text"
      },
      "source": [
        "<h1> Contents </h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q1WjCpkFWH6u",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "<h3>  \n",
        "\n",
        "* Identify and handle missing values   \n",
        "\n",
        "\n",
        "          1. Identify missing values\n",
        "          2. Deal with missing values\n",
        "          3. Correct data format   \n",
        "\n",
        "\n",
        "* Data standardization  \n",
        "\n",
        "* Data Normalization  \n",
        "\n",
        "* Binning  \n",
        "\n",
        "* Turning categorical into quantitative variables   \n",
        "\n",
        "\n",
        "\n",
        "</h3>  \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxoYPpiND6Hu",
        "colab_type": "text"
      },
      "source": [
        "#<h2>What is the purpose of Data Wrangling?</h2>\n",
        "Data Wrangling is the process of converting data from the initial format to a format that may be better for analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdwNYN7jD6Hv",
        "colab_type": "text"
      },
      "source": [
        "#Understand Data\n",
        " <h2>Read Dataset Description</h2>\n",
        "We want to predict students performance in math in secondary education of two high schools. Please read the description of data set before start.\n",
        "\n",
        " [Dataset Description](https://archive.ics.uci.edu/ml/datasets/student+performance)\n",
        "\n",
        "<h3>Import data</h3>\n",
        "<p>\n",
        "here link of raw data set: <a href=\"https://github.com/gdg-ml-team/DevFest19/blob/master/student-mat-raw.csv\">https://github.com/gdg-ml-team/DevFest19/blob/master/student-mat-raw.csv</a>. \n",
        "\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bgbbfukgD6Hw",
        "colab_type": "text"
      },
      "source": [
        "<h4>Import pandas</h4> "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Nv9NIVjD6Hx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pylab as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mye1n1xxD6Hz",
        "colab_type": "text"
      },
      "source": [
        "<h2>Reading the data set from the URL.</h2>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cBd-6REWD6H2",
        "colab_type": "code",
        "outputId": "94bc6676-7549-46fd-cb87-057272c26982",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "source": [
        "#URL of the dataset\n",
        "filename = \"https://raw.githubusercontent.com/gdg-ml-team/DevFest19/master/student-mat-raw.csv\"\n",
        "\n",
        "#Use the Pandas method read_csv() to load the data from the web address.\n",
        "df = pd.read_csv(filename)\n",
        "\n",
        "#Use the method head() to display the first five rows of the dataframe.\n",
        "# To see what the data set looks like, we'll use the head() method.\n",
        "df.head(10)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>school</th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>18.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>at_home</td>\n",
              "      <td>teacher</td>\n",
              "      <td>course</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>?</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>17.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>course</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>?</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>GP</td>\n",
              "      <td>Female</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3.0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>health</td>\n",
              "      <td>services</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>GP</td>\n",
              "      <td>Male</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>reputation</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>10.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>GP</td>\n",
              "      <td>M</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>12</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>other</td>\n",
              "      <td>teacher</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>6.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "      <td>M</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>A</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>18</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "      <td>M</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  school     sex   age address famsize  ... health  absences    G1  G2  G3\n",
              "0     GP       F  18.0       U     GT3  ...      3       NaN   5.0   6   6\n",
              "1     GP       F  17.0       U     GT3  ...      3       NaN   5.0   5   6\n",
              "2     GP  Female  15.0       U     LE3  ...      3      10.0   7.0   8  10\n",
              "3     GP       F  15.0       U     GT3  ...      5       2.0  15.0  14  15\n",
              "4     GP       F  16.0       U     GT3  ...      5       4.0   6.0  10  10\n",
              "5     GP    Male  16.0       U     LE3  ...      5      10.0  15.0  15  15\n",
              "6     GP       M  16.0       U     LE3  ...      3       0.0  12.0  12  11\n",
              "7    NaN     NaN   NaN     NaN     GT3  ...      1       6.0   6.0   5   6\n",
              "8    NaN       M  15.0       U     LE3  ...      1       0.0  16.0  18  19\n",
              "9    NaN       M  15.0       U     GT3  ...      5       0.0  14.0  15  15\n",
              "\n",
              "[10 rows x 33 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SK9iA_JRgPzx",
        "colab_type": "text"
      },
      "source": [
        "#Identify and handle missing values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ySC9kG1oD6H_",
        "colab_type": "text"
      },
      "source": [
        "As we can see, several question marks appeared in the dataframe; those are missing values which may hinder our further analysis. \n",
        "<div>So, how do we identify all those missing values and deal with them?</div> \n",
        "\n",
        "\n",
        "<h1>How to work with missing data?</h1>\n",
        "\n",
        "Steps for working with missing data:\n",
        "<ol>\n",
        "    <li>identify missing data</li>\n",
        "    <li>deal with missing data</li>\n",
        "    <li>correct data format</li>\n",
        "</ol>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ogFAnj85D6IA",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "<h1 id=\"identify_missing_values\">1. Identify missing values</h1>\n",
        "<h4>Convert \"?\" to NaN</h4>\n",
        "In the car dataset, missing data comes with the question mark \"?\".\n",
        "We replace \"?\" with NaN (Not a Number), which is Python's default missing value marker, for reasons of computational speed and convenience. Here we use the function: \n",
        " <pre>.replace(A, B, inplace = True) </pre>\n",
        "to replace A by B"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L7rJtNlLD6IB",
        "colab_type": "code",
        "outputId": "4d0c6a19-6a98-4bb0-8c4a-5c3f3b34b68c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# replace \"?\" to NaN\n",
        "df.replace(\"?\", np.nan, inplace = True)\n",
        "df.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>school</th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>18.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>at_home</td>\n",
              "      <td>teacher</td>\n",
              "      <td>course</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>17.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>course</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>GP</td>\n",
              "      <td>Female</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3.0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>health</td>\n",
              "      <td>services</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>GP</td>\n",
              "      <td>Male</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>reputation</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>10.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>GP</td>\n",
              "      <td>M</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>12</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>other</td>\n",
              "      <td>teacher</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>6.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>NaN</td>\n",
              "      <td>M</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>A</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>18</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>NaN</td>\n",
              "      <td>M</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  school     sex   age address famsize  ... health  absences    G1  G2  G3\n",
              "0     GP       F  18.0       U     GT3  ...      3       NaN   5.0   6   6\n",
              "1     GP       F  17.0       U     GT3  ...      3       NaN   5.0   5   6\n",
              "2     GP  Female  15.0       U     LE3  ...      3      10.0   7.0   8  10\n",
              "3     GP       F  15.0       U     GT3  ...      5       2.0  15.0  14  15\n",
              "4     GP       F  16.0       U     GT3  ...      5       4.0   6.0  10  10\n",
              "5     GP    Male  16.0       U     LE3  ...      5      10.0  15.0  15  15\n",
              "6     GP       M  16.0       U     LE3  ...      3       0.0  12.0  12  11\n",
              "7    NaN     NaN   NaN     NaN     GT3  ...      1       6.0   6.0   5   6\n",
              "8    NaN       M  15.0       U     LE3  ...      1       0.0  16.0  18  19\n",
              "9    NaN       M  15.0       U     GT3  ...      5       0.0  14.0  15  15\n",
              "\n",
              "[10 rows x 33 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ei8zywrFD6IC",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "<h4>Evaluating for Missing Data</h4>\n",
        "\n",
        "The missing values are converted to Python's default. We use Python's built-in functions to identify these missing values. There are two methods to detect missing data:\n",
        "<ol>\n",
        "    <li><b>.isnull()</b></li>\n",
        "    <li><b>.notnull()</b></li>\n",
        "</ol>\n",
        "The output is a boolean value indicating whether the value that is passed into the argument is in fact missing data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Np2Fq25pD6ID",
        "colab_type": "code",
        "outputId": "c00e52ef-44a9-4f7c-f6bd-d117554e9e7b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        }
      },
      "source": [
        "missing_data = df.isnull()\n",
        "missing_data.head(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>school</th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   school    sex    age  address  ...  absences     G1     G2     G3\n",
              "0   False  False  False    False  ...      True  False  False  False\n",
              "1   False  False  False    False  ...      True  False  False  False\n",
              "2   False  False  False    False  ...     False  False  False  False\n",
              "3   False  False  False    False  ...     False  False  False  False\n",
              "4   False  False  False    False  ...     False  False  False  False\n",
              "\n",
              "[5 rows x 33 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_wSAJVYuD6IF",
        "colab_type": "text"
      },
      "source": [
        "\"True\" stands for missing value, while \"False\" stands for not missing value."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aUNXgkoZrFkS",
        "colab_type": "text"
      },
      "source": [
        "<h2>Count missing values in each column</h2>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dlapZ2eeqLdc",
        "colab_type": "code",
        "outputId": "84981e1c-f284-4721-b1e5-49c41abe4ef6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 595
        }
      },
      "source": [
        "df.isnull().sum()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "school        22\n",
              "sex            1\n",
              "age            1\n",
              "address        1\n",
              "famsize        0\n",
              "Pstatus        0\n",
              "Medu           0\n",
              "Fedu           0\n",
              "Mjob           0\n",
              "Fjob           0\n",
              "reason         0\n",
              "guardian       0\n",
              "traveltime     0\n",
              "studytime      0\n",
              "failures       2\n",
              "schoolsup      0\n",
              "famsup         0\n",
              "paid           0\n",
              "activities     0\n",
              "nursery        0\n",
              "higher         0\n",
              "internet       0\n",
              "romantic       0\n",
              "famrel         0\n",
              "freetime       2\n",
              "goout          0\n",
              "Dalc           0\n",
              "Walc           0\n",
              "health         0\n",
              "absences       2\n",
              "G1             2\n",
              "G2             0\n",
              "G3             0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iSGqwmzRD6II",
        "colab_type": "text"
      },
      "source": [
        "<h1 id=\"deal_missing_values\">2. Deal with missing data</h1>\n",
        "<b>How to deal with missing data?</b>\n",
        "\n",
        "<ol>\n",
        "    <li>drop data<br>\n",
        "        a. drop the whole row<br>\n",
        "        b. drop the whole column\n",
        "    </li>\n",
        "    <li>replace data<br>\n",
        "        a. replace it by mean<br>\n",
        "        b. replace it by frequency<br>\n",
        "        c. replace it based on other functions\n",
        "    </li>\n",
        "</ol>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6ICLBEWD6IJ",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "<b>Replace by mean:</b>\n",
        "<ul>\n",
        "    <li>\"G1\": 2 missing data, replace them with mean</li>\n",
        "    <li>\"absences\": 2 missing data, replace them with mean</li>\n",
        "</ul>\n",
        "\n",
        "<b>Replace by frequency:</b>\n",
        "<ul>\n",
        "    <li>\"failures\": 2 missing data, replace them with \"0\". \n",
        "        <ul>\n",
        "    <li>Reason: 0 is most frequent, it is most likely to occur</li>\n",
        "     </ul> </ul>\n",
        "    <ul>\n",
        "            <li>\"freetime\": 2 missing data, replace them with \"3\". \n",
        "        <ul>\n",
        "            <li>Reason: 3 is most frequent, it is most likely to occur</li>\n",
        "        </ul>\n",
        "    </li>\n",
        "</ul>\n",
        "\n",
        "<b>Drop the whole row:</b>\n",
        "<ul>\n",
        "    <li> row #7 contains 4 missing data(school,sex,age,address) simply delete the whole row\n",
        "        <ul>\n",
        "            <li>Reason: school, sex ,age and address have missing value in the same rows</li>\n",
        "        </ul>\n",
        "    </li>\n",
        "</ul>\n",
        "\n",
        "<b>Drop the whole column:</b>\n",
        "<ul>\n",
        "    <li>\"school\": 22 missing data, simply delete the whole column\n",
        "        <ul>\n",
        "            <li>Whole columns should be dropped only if most entries in the column are empty. In our dataset, school has a lot of missing data and drop this column not effect.\n",
        "</li>\n",
        "        </ul>\n",
        "    </li>\n",
        "</ul>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1ekCSLiD6IJ",
        "colab_type": "text"
      },
      "source": [
        "<h4>Calculate the average of the column </h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R_8a7A2aD6IK",
        "colab_type": "code",
        "outputId": "e1615a34-20d7-462d-d325-33f0b598c1ac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "avg_G1 = df[\"G1\"].astype(\"float\").mean(axis=0)\n",
        "print(\"Average of G1:\", int(avg_G1))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Average of G1: 10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Na_mYbs-D6IM",
        "colab_type": "text"
      },
      "source": [
        "<h4>Replace \"NaN\" by mean value in \"G1\" column</h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O3O9ztRED6IM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df[\"G1\"].replace(np.nan, avg_G1, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P3YmBCa_D6IO",
        "colab_type": "text"
      },
      "source": [
        "<h4>Calculate the mean value for 'absences' column</h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tXCJp0LFD6IO",
        "colab_type": "code",
        "outputId": "39c4de3b-898a-4f3a-9ec1-15aaeb1359eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "avg_absences=df['absences'].astype('float').mean(axis=0)\n",
        "print(\"Average of absences:\", int(avg_absences))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Average of absences: 5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0FAj57lsD6IR",
        "colab_type": "text"
      },
      "source": [
        "<h4>Replace NaN by mean value</h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-f0NI6MBD6IT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df[\"absences\"].replace(np.nan, avg_absences, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVEwEgix1BUU",
        "colab_type": "text"
      },
      "source": [
        "<h3>Replace by frequency</h3>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7jBeuLx7D6Ik",
        "colab_type": "text"
      },
      "source": [
        "To see which values are present in a particular column, we can use the \".value_counts()\" method:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Hd4d7gUD6Il",
        "colab_type": "code",
        "outputId": "dddab497-08bb-4057-e94e-90428aa3286e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "df['failures'].value_counts()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0    310\n",
              "1.0     50\n",
              "2.0     17\n",
              "3.0     16\n",
              "Name: failures, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGs9NxFBD6Io",
        "colab_type": "text"
      },
      "source": [
        "We can see that four doors are the most common type. We can also use the \".idxmax()\" method to calculate for us the most common type automatically:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SZMbRmWqD6Ip",
        "colab_type": "code",
        "outputId": "58760f66-d935-4560-b38d-a6128678f3bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "df['failures'].value_counts().idxmax()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hrVGXlgcD6Ir",
        "colab_type": "text"
      },
      "source": [
        "The replacement procedure is very similar to what we have seen previously"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYMNjbgbD6Is",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#replace the missing 'failures' values by the most frequent \n",
        "df[\"failures\"].replace(np.nan, 0, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_U9jdhhh_L8u",
        "colab_type": "code",
        "outputId": "2fe8b82b-02eb-4d8c-e5f4-352c5ee25d39",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "df['freetime'].value_counts()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3    155\n",
              "4    115\n",
              "2     64\n",
              "5     40\n",
              "1     19\n",
              "Name: freetime, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydHdlWta_S9W",
        "colab_type": "code",
        "outputId": "8473154c-7ca2-45be-b4a8-9b0c72ff8c0c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "df['freetime'].value_counts().idxmax()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'3'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FuzYj0Za_Zuu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#replace the missing 'freetime' values by the most frequent \n",
        "df[\"freetime\"].replace(np.nan, 3, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmzy1XLW1bLp",
        "colab_type": "text"
      },
      "source": [
        "<h3>Drop the whole row</h3>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VKZvWXrQD6Iu",
        "colab_type": "text"
      },
      "source": [
        "Finally, let's drop the row that do not have (sex,age,address)  data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RduzPQKpD6Iu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# simply drop whole row with NaN in \"price\" column\n",
        "df.dropna(subset=[\"sex\"], axis=0, inplace=True)\n",
        "\n",
        "# reset index, because we droped two rows\n",
        "df.reset_index(drop=True, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jBFGuj41klT",
        "colab_type": "text"
      },
      "source": [
        "<h3>Drop the whole column</h3>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zfdiPdSa1tuc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# simply drop whole row with NaN in \"price\" column\n",
        "df.drop([\"school\"], axis=1, inplace=True)\n",
        "\n",
        "# reset index, because we droped two rows\n",
        "df.reset_index(drop=True, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0LubZNbdD6Iw",
        "colab_type": "code",
        "outputId": "52342139-3a22-4aaa-af1b-6af9030648a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "source": [
        "df.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>F</td>\n",
              "      <td>18.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>at_home</td>\n",
              "      <td>teacher</td>\n",
              "      <td>course</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>5.712468</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>F</td>\n",
              "      <td>17.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>course</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>5.712468</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Female</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3.0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>F</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>health</td>\n",
              "      <td>services</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>15.0</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>F</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>6.0</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Male</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>reputation</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>15.0</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>M</td>\n",
              "      <td>16.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>12.0</td>\n",
              "      <td>12</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>M</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>A</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>16.0</td>\n",
              "      <td>18</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>M</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>14.0</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>F</td>\n",
              "      <td>15.0</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>teacher</td>\n",
              "      <td>health</td>\n",
              "      <td>reputation</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>10.0</td>\n",
              "      <td>8</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      sex   age address famsize Pstatus  ...  health   absences    G1  G2  G3\n",
              "0       F  18.0       U     GT3       A  ...       3   5.712468   5.0   6   6\n",
              "1       F  17.0       U     GT3       T  ...       3   5.712468   5.0   5   6\n",
              "2  Female  15.0       U     LE3       T  ...       3  10.000000   7.0   8  10\n",
              "3       F  15.0       U     GT3       T  ...       5   2.000000  15.0  14  15\n",
              "4       F  16.0       U     GT3       T  ...       5   4.000000   6.0  10  10\n",
              "5    Male  16.0       U     LE3       T  ...       5  10.000000  15.0  15  15\n",
              "6       M  16.0       U     LE3       T  ...       3   0.000000  12.0  12  11\n",
              "7       M  15.0       U     LE3       A  ...       1   0.000000  16.0  18  19\n",
              "8       M  15.0       U     GT3       T  ...       5   0.000000  14.0  15  15\n",
              "9       F  15.0       U     GT3       T  ...       2   0.000000  10.0   8   9\n",
              "\n",
              "[10 rows x 32 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p0JOgfQ3D6Ix",
        "colab_type": "text"
      },
      "source": [
        "<b>Good!</b> Now, we obtain the dataset with no missing values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gc2rfLbLD6Iy",
        "colab_type": "text"
      },
      "source": [
        "<h1 id=\"correct_data_format\">3. Correct data format</h1>\n",
        "<b>We are almost there!</b>\n",
        "<p>The last step in data cleaning is checking and making sure that all data is in the correct format (int, float, text or other).</p>\n",
        "\n",
        "In Pandas, we use \n",
        "<p><b>.dtype()</b> to check the data type</p>\n",
        "<p><b>.astype()</b> to change the data type</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i9TESRcKD6Iy",
        "colab_type": "text"
      },
      "source": [
        "<h4>Lets list the data types for each column</h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "18o5YwA0D6Iz",
        "colab_type": "code",
        "outputId": "9cdecfcf-8b85-4714-9113-148e5bba56ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        }
      },
      "source": [
        "df.dtypes"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "sex            object\n",
              "age           float64\n",
              "address        object\n",
              "famsize        object\n",
              "Pstatus        object\n",
              "Medu            int64\n",
              "Fedu            int64\n",
              "Mjob           object\n",
              "Fjob           object\n",
              "reason         object\n",
              "guardian       object\n",
              "traveltime      int64\n",
              "studytime       int64\n",
              "failures      float64\n",
              "schoolsup      object\n",
              "famsup         object\n",
              "paid           object\n",
              "activities     object\n",
              "nursery        object\n",
              "higher         object\n",
              "internet       object\n",
              "romantic       object\n",
              "famrel          int64\n",
              "freetime       object\n",
              "goout           int64\n",
              "Dalc            int64\n",
              "Walc            int64\n",
              "health          int64\n",
              "absences      float64\n",
              "G1            float64\n",
              "G2              int64\n",
              "G3              int64\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RrIvbZitD6I0",
        "colab_type": "text"
      },
      "source": [
        "<p>Assume we can see above, some columns are not of the correct data type. Numerical variables should have type 'float' or 'int', and variables with strings such as categories should have type 'object'.</p> "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLwpW32FD6I1",
        "colab_type": "text"
      },
      "source": [
        "<h4>Convert data types to proper format</h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "toe7J74yD6I1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df[[\"age\", \"absences\",\"failures\"]] = df[[\"age\", \"absences\",\"failures\"]].astype(\"int\")\n",
        "df[[\"G2\"]] = df[[\"G2\"]].astype(\"float\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pTBRJ7oqD6I3",
        "colab_type": "text"
      },
      "source": [
        "<h4>Let us list the columns after the conversion</h4>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bxi-Gyu8D6I4",
        "colab_type": "code",
        "outputId": "3236eaed-7276-4eb4-82bc-56107ec23d42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 646
        }
      },
      "source": [
        "df.info()#df.dtypes"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 394 entries, 0 to 393\n",
            "Data columns (total 32 columns):\n",
            "sex           394 non-null object\n",
            "age           394 non-null int64\n",
            "address       394 non-null object\n",
            "famsize       394 non-null object\n",
            "Pstatus       394 non-null object\n",
            "Medu          394 non-null int64\n",
            "Fedu          394 non-null int64\n",
            "Mjob          394 non-null object\n",
            "Fjob          394 non-null object\n",
            "reason        394 non-null object\n",
            "guardian      394 non-null object\n",
            "traveltime    394 non-null int64\n",
            "studytime     394 non-null int64\n",
            "failures      394 non-null int64\n",
            "schoolsup     394 non-null object\n",
            "famsup        394 non-null object\n",
            "paid          394 non-null object\n",
            "activities    394 non-null object\n",
            "nursery       394 non-null object\n",
            "higher        394 non-null object\n",
            "internet      394 non-null object\n",
            "romantic      394 non-null object\n",
            "famrel        394 non-null int64\n",
            "freetime      394 non-null object\n",
            "goout         394 non-null int64\n",
            "Dalc          394 non-null int64\n",
            "Walc          394 non-null int64\n",
            "health        394 non-null int64\n",
            "absences      394 non-null int64\n",
            "G1            394 non-null float64\n",
            "G2            394 non-null float64\n",
            "G3            394 non-null int64\n",
            "dtypes: float64(2), int64(13), object(17)\n",
            "memory usage: 98.6+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h83Vh74UD6I5",
        "colab_type": "text"
      },
      "source": [
        "<b>Wonderful!</b>\n",
        "\n",
        "Now, we finally obtain the cleaned dataset with no missing values and all data in its proper format."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DxACg2frD6I5",
        "colab_type": "text"
      },
      "source": [
        "#Data Standardization\n",
        "<p>\n",
        "Data is usually collected from different agencies with different formats.\n",
        "(Data Standardization is also a term for a particular type of data normalization)\n",
        "</p>\n",
        "    \n",
        "<b>What is Standardization?</b>\n",
        "<p>Standardization is the process of transforming data into a common format which allows the researcher to make the meaningful comparison.\n",
        "</p>\n",
        "\n",
        "<b>Example  </b>\n",
        "<p>Transform using The formula for unit conversion is:</p>\n",
        "<p>1 kilometer equals 0.62137 miles. \n",
        "\n",
        "Kilometer= Miles/ 0.62137\n",
        "\n",
        "df['Kilometer'] = df['Miles'] / 0.62137</p>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uw6kepjhFl8I",
        "colab_type": "code",
        "outputId": "f1ea2418-3774-4727-c23d-fd652434662c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "df['sex'].unique()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['F', 'Female', 'Male', 'M'], dtype=object)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F97HOp_uF5US",
        "colab_type": "code",
        "outputId": "7c46b69f-ade2-4318-a4d5-a26ee8640c80",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "source": [
        "df.replace(\"Female\", 'F', inplace = True)\n",
        "df.replace(\"Male\", 'M', inplace = True)\n",
        "df.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>F</td>\n",
              "      <td>18</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>at_home</td>\n",
              "      <td>teacher</td>\n",
              "      <td>course</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>F</td>\n",
              "      <td>17</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>course</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>F</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>F</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>health</td>\n",
              "      <td>services</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>15.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>F</td>\n",
              "      <td>16</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>6.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>M</td>\n",
              "      <td>16</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>reputation</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>15.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>M</td>\n",
              "      <td>16</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>M</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>A</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>services</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>M</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>F</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>teacher</td>\n",
              "      <td>health</td>\n",
              "      <td>reputation</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  sex  age address famsize Pstatus  ...  health  absences    G1    G2  G3\n",
              "0   F   18       U     GT3       A  ...       3         5   5.0   6.0   6\n",
              "1   F   17       U     GT3       T  ...       3         5   5.0   5.0   6\n",
              "2   F   15       U     LE3       T  ...       3        10   7.0   8.0  10\n",
              "3   F   15       U     GT3       T  ...       5         2  15.0  14.0  15\n",
              "4   F   16       U     GT3       T  ...       5         4   6.0  10.0  10\n",
              "5   M   16       U     LE3       T  ...       5        10  15.0  15.0  15\n",
              "6   M   16       U     LE3       T  ...       3         0  12.0  12.0  11\n",
              "7   M   15       U     LE3       A  ...       1         0  16.0  18.0  19\n",
              "8   M   15       U     GT3       T  ...       5         0  14.0  15.0  15\n",
              "9   F   15       U     GT3       T  ...       2         0  10.0   8.0   9\n",
              "\n",
              "[10 rows x 32 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nz552eKCD6JG",
        "colab_type": "text"
      },
      "source": [
        "#Data Normalization\n",
        "\n",
        "<b>Why normalization?</b>\n",
        "<p>Normalization is the process of transforming values of several variables into a similar range. Typical normalizations include scaling the variable so the variable average is 0, scaling the variable so the variance is 1, or scaling variable so the variable values range from 0 to 1\n",
        "</p>\n",
        "\n",
        "<b>Example</b>\n",
        "<p>To demonstrate normalization, let's say we want to scale the columns \"age\" </p>\n",
        "<p><b>Target:</b>would like to Normalize those variables so their value ranges from 0 to 1.</p>\n",
        "<p><b>Approach:</b> replace original value by (original value)/(maximum value)</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qDv9SSAwD6JG",
        "colab_type": "code",
        "outputId": "92c9237f-430b-4eb1-ee7e-2beebcc5886d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "# replace (original value) by (original value)/(maximum value)\n",
        "df['age'] = df['age']/df['age'].max()\n",
        "df['age'].head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    0.818182\n",
              "1    0.772727\n",
              "2    0.681818\n",
              "3    0.681818\n",
              "4    0.727273\n",
              "Name: age, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1BPTMYrD6JJ",
        "colab_type": "text"
      },
      "source": [
        "Here we can see, we've normalized \"age\" in the range of [0,1]."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "udjnWvV6iD1r",
        "colab_type": "text"
      },
      "source": [
        "#Binning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_GZuEkEnD6JK",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "<b>Why binning?</b>\n",
        "<p>\n",
        "    Binning is a process of transforming continuous numerical variables into discrete categorical 'bins', for grouped analysis.\n",
        "</p>\n",
        "\n",
        "<b>Example: </b>\n",
        "<p>In our dataset, \"G1\" is a real valued variable ranging from 0 to 20. </p>\n",
        "\n",
        "<p>We will use the Pandas method 'cut' to segment the 'G1' column into 3 bins </p>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9UwTxtnD6JK",
        "colab_type": "text"
      },
      "source": [
        "<h3>Example of Binning Data In Pandas</h3>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "351ksPmgD6JL",
        "colab_type": "text"
      },
      "source": [
        " Convert data to correct format "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "io_q-FzwD6JM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df[\"G1\"]=df[\"G1\"].astype(int, copy=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iz0UOHkSD6JO",
        "colab_type": "text"
      },
      "source": [
        "Lets plot the histogram of horspower, to see what the distribution of horsepower looks like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "siklGI1uD6JO",
        "colab_type": "code",
        "outputId": "37934bc8-20a4-4d23-b0cb-07869d97f0fd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        }
      },
      "source": [
        "import matplotlib as plt\n",
        "from matplotlib import pyplot\n",
        "plt.pyplot.hist(df[\"G1\"])\n",
        "\n",
        "# set x/y labels and plot title\n",
        "plt.pyplot.xlabel(\"G1\")\n",
        "plt.pyplot.ylabel(\"count\")\n",
        "plt.pyplot.title(\"G1 bins\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'G1 bins')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFAlJREFUeJzt3X+wX3V95/HnC5BS8AdgrhEJ7KUV\ncdApaO9S1K4roC5dWMJ2LAOjnVTZzfbHuurY2minu7XT3cEfq2Wsq5PhV7RWQArClllXNrI6nSqY\nCFQhuthskGBCIkpRVGj0vX98T/Q2e5P7vUnO99zk83zM3Pmec77nx0vMzSvnd6oKSVK7Dhk6gCRp\nWBaBJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJpLySZTlJJDtvN9+9IcsWkc0l7wyJQM5JcnOSOJI8n\n2dYN/3aSdN+fleT2JH+fZNO+bKuq/ktV/Zv9ElzqmUWgJiR5K3A58B7g2cBS4DeBlwGHd7M9DlwF\n/N4QGaWhWAQ66CV5BvDHwG9X1Q1V9d0auauqXltVTwBU1Z1V9VFg4wJW/4Yk30yyJcnvztrmHyX5\n825452GkFUm+keRbSf5g1rxnJFmX5LEkDyd53/75Xy6NZ87jm9JB5iXAzwA397Dus4CTgZ8DPpPk\n7qr6X7uZ95eBU4DnAXcmubGqNjDaU7m8qj6a5KnAC3vIKe2WewRqwRLgW1W1Y+eEJH+T5NEkP0jy\n8n1Y9zur6vGq+jJwNXDJPPP+oKruAe4BTuum/wPw3CRLqup7VfWFfcgjLZhFoBY8AiyZfYVPVb20\nqo7uvtuX34MHZw0/ADxnD/NunTX8feCp3fCljPYSvprki0nO34c80oJZBGrB54EngOU9rPuEWcMn\nAt9c6Aqq6v6qugR4FvAu4IYkR+2nfNK8LAId9KrqUeCdwH9L8pokT0tySJLTgZ/8hdtNOwJ4ymg0\nRyQ5fDer3ekPkxyZ5AXA64HrFpovyeuSTFXVj4FHu8k/Xuh6pL3lyWI1oareneQh4G3ARxhdKroR\n+H3gb7rZXg7cPmuxHwCfBV6xh1V/Fvg6o39UvbeqPr0X8c4F3pfkSEaHly6uqh/sxXqkvRJfTCNJ\nbfPQkCQ1ziKQpMZZBJLUOItAkhp3QFw1tGTJkpqenh46hiQdUNavX/+tqpqab74Dogimp6dZt27d\n0DEk6YCS5IFx5vPQkCQ1ziKQpMZZBJLUOItAkhpnEUhS4ywCSWqcRSBJjbMIJKlxFoEkNe6AuLNY\nWqymV9062LY3XXbeYNvWwcU9AklqnEUgSY2zCCSpcRaBJDXOk8XabzxxKh2Y3COQpMZZBJLUuN6K\nIMkpSe6e9fNYkjcnOTbJbUnu7z6P6SuDJGl+vRVBVX2tqk6vqtOBXwS+D9wErALWVtXJwNpuXJI0\nkEkdGjoH+LuqegBYDqzppq8BLpxQBknSHCZVBBcDH++Gl1bVlm54K7B0rgWSrEyyLsm67du3TyKj\nJDWp9yJIcjhwAfCJXb+rqgJqruWqanVVzVTVzNTUVM8pJaldk9gj+BXgS1X1cDf+cJLjALrPbRPI\nIEnajUkUwSX89LAQwC3Aim54BXDzBDJIknaj1yJIchTwKuDGWZMvA16V5H7gld24JGkgvT5ioqoe\nB565y7RHGF1FJElaBLyzWJIaZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCkxlkEktQ4i0CSGmcR\nSFLjLAJJapxFIEmNswgkqXG9Pn1UUn+mV906yHY3XXbeINtVf9wjkKTGWQSS1DiLQJIaZxFIUuMs\nAklqXN8vrz86yQ1JvppkQ5KXJDk2yW1J7u8+j+kzgyRpz/reI7gc+FRVPR84DdgArALWVtXJwNpu\nXJI0kN6KIMkzgJcDVwJU1ZNV9SiwHFjTzbYGuLCvDJKk+fW5R3ASsB24OsldSa5IchSwtKq2dPNs\nBZbOtXCSlUnWJVm3ffv2HmNKUtv6LILDgBcDH6qqFwGPs8thoKoqoOZauKpWV9VMVc1MTU31GFOS\n2tZnEWwGNlfVHd34DYyK4eEkxwF0n9t6zCBJmkdvRVBVW4EHk5zSTToHuA+4BVjRTVsB3NxXBknS\n/Pp+6NwbgY8lORzYCLyeUflcn+RS4AHgop4zSJL2oNciqKq7gZk5vjqnz+1KksbnncWS1DiLQJIa\nZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJJapxFIEmNswgkqXEW\ngSQ1ziKQpMZZBJLUOItAkhpnEUhS43p9Z3GSTcB3gR8BO6pqJsmxwHXANLAJuKiqvtNnDh38plfd\nOnQE6YA1iT2Cs6rq9Kra+RL7VcDaqjoZWNuNS5IGMsShoeXAmm54DXDhABkkSZ2+i6CATydZn2Rl\nN21pVW3phrcCS+daMMnKJOuSrNu+fXvPMSWpXb2eIwB+uaoeSvIs4LYkX539ZVVVkpprwapaDawG\nmJmZmXMeSdK+63WPoKoe6j63ATcBZwAPJzkOoPvc1mcGSdKe9VYESY5K8rSdw8Crga8AtwArutlW\nADf3lUGSNL8+Dw0tBW5KsnM7f1FVn0ryReD6JJcCDwAX9ZhBkjSP3oqgqjYCp80x/RHgnL62K0la\nGO8slqTGWQSS1DiLQJIaZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCkxlkEktQ4i0CSGmcRSFLj\nLAJJapxFIEmNG6sIkqwdZ5ok6cCzx/cRJDkCOBJYkuQYIN1XTweO7zmbJGkC5nsxzb8D3gw8B1jP\nT4vgMeDPeswlSZqQPRZBVV0OXJ7kjVX1gQllkiRN0FivqqyqDyR5KTA9e5mq+khPuSRJEzJWEST5\nKPDzwN3Aj7rJBcxbBEkOBdYBD1XV+UlOAq4FnsnocNOvV9WTe5FdkrQfjPvy+hng1KqqvdjGm4AN\njE4wA7wLeH9VXZvkw8ClwIf2Yr2SpP1g3PsIvgI8e6ErT7IMOA+4ohsPcDZwQzfLGuDCha5XkrT/\njLtHsAS4L8mdwBM7J1bVBfMs96fA24CndePPBB6tqh3d+GZ2cxlqkpXASoATTzxxzJiSpIUatwj+\naKErTnI+sK2q1id5xUKXr6rVwGqAmZmZvTkkJUkaw7hXDX12L9b9MuCCJP8SOILROYLLgaOTHNbt\nFSwDHtqLdUuS9pNxHzHx3SSPdT8/TPKjJI/taZmqentVLauqaeBi4DNV9VrgduA13WwrgJv3Ib8k\naR+Nu0ew8xj/zhO+y4Ez93Kbvw9cm+RPgLuAK/dyPZKk/WDBTx+tkU8C/2IBy/zvqjq/G95YVWdU\n1XOr6teq6on5lpck9WfcG8p+ddboIYzuK/hhL4kkSRM17lVD/2rW8A5gE6PDQ5KkA9y45whe33cQ\n7T/Tq24dOoKkA8i4Vw0tS3JTkm3dz192dw1Lkg5w454svhq4hdF7CZ4D/PdumiTpADduEUxV1dVV\ntaP7uQaY6jGXJGlCxi2CR5K8Lsmh3c/rgEf6DCZJmoxxi+ANwEXAVmALozuDf6OnTJKkCRr38tE/\nBlZU1XcAkhwLvJdRQUiSDmDj7hH8ws4SAKiqbwMv6ieSJGmSxi2CQ5Ics3Ok2yMYd29CkrSIjfuX\n+X8FPp/kE934rwH/uZ9IkqRJGvfO4o8kWcfoNZMAv1pV9/UXS5I0KWMf3un+4vcvf0k6yCz4MdSS\npIOLRSBJjbMIJKlxFoEkNc4ikKTG9VYESY5IcmeSe5Lcm+Sd3fSTktyR5OtJrktyeF8ZJEnz63OP\n4Ang7Ko6DTgdODfJmcC7gPdX1XOB7wCX9phBkjSP3oqgRr7XjT6l+ylGN6Xd0E1fA1zYVwZJ0vx6\nfV5QkkOB9cBzgQ8Cfwc8WlU7ulk2A8fvZtmVwEqAE088sc+YvfC9wdL+N9Tv1abLzhtku5PS68ni\nqvpRVZ0OLAPOAJ6/gGVXV9VMVc1MTfkyNEnqy0SuGqqqR4HbgZcARyfZuSeyDHhoEhkkSXPr86qh\nqSRHd8M/C7wK2MCoEF7TzbYCuLmvDJKk+fV5juA4YE13nuAQ4Pqq+qsk9wHXJvkT4C7gyh4zSJLm\n0VsRVNXfMsdbzKpqI6PzBZKkRcA7iyWpcRaBJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJJapxFIEmN\nswgkqXEWgSQ1ziKQpMb1+mIaSQcfX7p08HGPQJIaZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXOIpCk\nxvVWBElOSHJ7kvuS3JvkTd30Y5PcluT+7vOYvjJIkubX5x7BDuCtVXUqcCbwO0lOBVYBa6vqZGBt\nNy5JGkhvRVBVW6rqS93wd4ENwPHAcmBNN9sa4MK+MkiS5jeRcwRJpoEXAXcAS6tqS/fVVmDpbpZZ\nmWRdknXbt2+fRExJalLvRZDkqcBfAm+uqsdmf1dVBdRcy1XV6qqaqaqZqampvmNKUrN6LYIkT2FU\nAh+rqhu7yQ8nOa77/jhgW58ZJEl71udVQwGuBDZU1ftmfXULsKIbXgHc3FcGSdL8+nwM9cuAXwe+\nnOTubto7gMuA65NcCjwAXNRjBknSPHorgqr6ayC7+fqcvrYrSVoY7yyWpMZZBJLUOItAkhpnEUhS\n4ywCSWqcRSBJjbMIJKlxFoEkNc4ikKTGWQSS1DiLQJIaZxFIUuMsAklqnEUgSY2zCCSpcRaBJDXO\nIpCkxlkEktQ4i0CSGtfbO4uTXAWcD2yrqhd2044FrgOmgU3ARVX1nb4ySNL+ML3q1kG2u+my8yay\nnT73CK4Bzt1l2ipgbVWdDKztxiVJA+qtCKrqc8C3d5m8HFjTDa8BLuxr+5Kk8Uz6HMHSqtrSDW8F\nlu5uxiQrk6xLsm779u2TSSdJDRrsZHFVFVB7+H51Vc1U1czU1NQEk0lSWyZdBA8nOQ6g+9w24e1L\nknYx6SK4BVjRDa8Abp7w9iVJu+itCJJ8HPg8cEqSzUkuBS4DXpXkfuCV3bgkaUC93UdQVZfs5qtz\n+tqmJGnhvLNYkhpnEUhS4ywCSWqcRSBJjbMIJKlxFoEkNc4ikKTGWQSS1DiLQJIaZxFIUuMsAklq\nnEUgSY2zCCSpcRaBJDXOIpCkxlkEktS43l5Ms1hMr7p16AiStKi5RyBJjbMIJKlxgxRBknOTfC3J\n15OsGiKDJGlk4kWQ5FDgg8CvAKcClyQ5ddI5JEkjQ+wRnAF8vao2VtWTwLXA8gFySJIY5qqh44EH\nZ41vBn5p15mSrARWdqPfS/K1CWSbzxLgW0OHmIO5FsZcC2OuhdlvufKufV7FPxlnpkV7+WhVrQZW\nD51jtiTrqmpm6By7MtfCmGthzLUwizXXngxxaOgh4IRZ48u6aZKkAQxRBF8ETk5yUpLDgYuBWwbI\nIUligENDVbUjyb8H/idwKHBVVd076Rx7aVEdqprFXAtjroUx18Is1ly7laoaOoMkaUDeWSxJjbMI\nJKlxFsGYkhya5K4kfzV0ltmSHJ3khiRfTbIhyUuGzgSQ5C1J7k3ylSQfT3LEQDmuSrItyVdmTTs2\nyW1J7u8+j1kkud7T/f/4t0luSnL0Ysg167u3JqkkSxZLriRv7P6b3Zvk3YshV5LTk3whyd1J1iU5\nY9K5FsoiGN+bgA1Dh5jD5cCnqur5wGksgoxJjgf+AzBTVS9kdFHAxQPFuQY4d5dpq4C1VXUysLYb\nn7Rr+P9z3Qa8sKp+Afg/wNsnHYq5c5HkBODVwDcmHahzDbvkSnIWo6cSnFZVLwDeuxhyAe8G3llV\npwP/sRtf1CyCMSRZBpwHXDF0ltmSPAN4OXAlQFU9WVWPDpvqJw4DfjbJYcCRwDeHCFFVnwO+vcvk\n5cCabngNcOFEQzF3rqr6dFXt6Ea/wOgem8Fzdd4PvA0Y5OqS3eT6LeCyqnqim2fbIslVwNO74Wcw\n0J/9hbAIxvOnjH4Jfjx0kF2cBGwHru4OW12R5KihQ1XVQ4z+dfYNYAvw91X16WFT/SNLq2pLN7wV\nWDpkmN14A/A/hg4BkGQ58FBV3TN0ll08D/hnSe5I8tkk/3ToQJ03A+9J8iCj34Mh9uwWxCKYR5Lz\ngW1VtX7oLHM4DHgx8KGqehHwOMMc5vhHumPuyxkV1XOAo5K8bthUc6vR9dOL6hrqJH8A7AA+tgiy\nHAm8g9EhjsXmMOBY4Ezg94Drk2TYSMBoT+UtVXUC8Ba6PfbFzCKY38uAC5JsYvSk1LOT/PmwkX5i\nM7C5qu7oxm9gVAxDeyXwf6tqe1X9A3Aj8NKBM832cJLjALrPiR9S2J0kvwGcD7y2FsdNPj/PqNDv\n6X4HlgFfSvLsQVONbAZurJE7Ge2xT/xE9hxWMPozD/AJRk9cXtQsgnlU1durallVTTM64fmZqloU\n/7qtqq3Ag0lO6SadA9w3YKSdvgGcmeTI7l9o57AITmLPcgujX1a6z5sHzPITSc5ldAjygqr6/tB5\nAKrqy1X1rKqa7n4HNgMv7v7sDe2TwFkASZ4HHM7ieBrpN4F/3g2fDdw/YJaxLNqnj2psbwQ+1j23\naSPw+oHzUFV3JLkB+BKjQxx3MdBt90k+DrwCWJJkM/CfgMsYHUa4FHgAuGiR5Ho78DPAbd0Rji9U\n1W8OnauqBj+0sZv/XlcBV3WXbj4JrJj0XtRucv1b4PLuQokf8tPH6S9aPmJCkhrnoSFJapxFIEmN\nswgkqXEWgSQ1ziKQpMZZBNKYkixN8hdJNiZZn+TzSf51kmcmuT3J95L82dA5pYWyCKQxdDfGfRL4\nXFX9XFX9IqMbDJcxulb8D4HfHTCitNcsAmk8ZwNPVtWHd06oqgeq6gNV9XhV/TWjQpAOOBaBNJ4X\nMLpTWjroWATSXkjywST3JPni0FmkfWURSOO5l1lPdq2q32H0ML2pwRJJ+4lFII3nM8ARSX5r1rQj\nhwoj7U8+dE4aU/fugvcDv8TozXCPAx+uquu6Z/U/ndGjkB8FXl1Vi+GR4NK8LAJJapyHhiSpcRaB\nJDXOIpCkxlkEktQ4i0CSGmcRSFLjLAJJatz/A6VkcVzJVLEsAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4W4pUc0nD6JQ",
        "colab_type": "text"
      },
      "source": [
        "<p>We would like 3 bins of equal size bandwidth so we use numpy's <code>linspace(start_value, end_value, numbers_generated</code> function.</p>\n",
        "<p>Since we want to include the minimum value of G1 we want to set start_value=min(df[\"G1\"]).</p>\n",
        "<p>Since we want to include the maximum value of G1 we want to set end_value=max(df[\"G1\"]).</p>\n",
        "<p>Since we are building 3 bins of equal length, there should be 4 dividers, so numbers_generated=4.</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2rfWlqJcD6JS",
        "colab_type": "text"
      },
      "source": [
        "We build a bin array, with a minimum value to a maximum value, with bandwidth calculated above. The bins will be values used to determine when one bin ends and another begins."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XZqwfvyuD6JS",
        "colab_type": "code",
        "outputId": "ab035c29-0b5c-4a5d-9608-36724b456162",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "bins = np.linspace(min(df[\"G1\"]), max(df[\"G1\"]), 4)\n",
        "bins"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 3.        ,  8.33333333, 13.66666667, 19.        ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ajgRneHED6JU",
        "colab_type": "text"
      },
      "source": [
        " We set group  names:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HOiprIFAD6JU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "group_names = ['Low', 'Medium', 'High']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kdqYgTWAD6JY",
        "colab_type": "text"
      },
      "source": [
        " We apply the function \"cut\" the determine what each value of \"df['G1']\" belongs to. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QepdsQMvD6JZ",
        "colab_type": "code",
        "outputId": "7937bcbe-74e5-4d91-81b6-6a12368a5479",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        }
      },
      "source": [
        "df['G1-binned'] = pd.cut(df['G1'], bins, labels=group_names, include_lowest=True )\n",
        "df[['G1','G1-binned']].head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>G1</th>\n",
              "      <th>G1-binned</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5</td>\n",
              "      <td>Low</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5</td>\n",
              "      <td>Low</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>7</td>\n",
              "      <td>Low</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>15</td>\n",
              "      <td>High</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>6</td>\n",
              "      <td>Low</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>15</td>\n",
              "      <td>High</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>12</td>\n",
              "      <td>Medium</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>16</td>\n",
              "      <td>High</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>14</td>\n",
              "      <td>High</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>Medium</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   G1 G1-binned\n",
              "0   5       Low\n",
              "1   5       Low\n",
              "2   7       Low\n",
              "3  15      High\n",
              "4   6       Low\n",
              "5  15      High\n",
              "6  12    Medium\n",
              "7  16      High\n",
              "8  14      High\n",
              "9  10    Medium"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PC1Tbu07D6Jb",
        "colab_type": "text"
      },
      "source": [
        "Lets see the number of vehicles in each bin."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lZjANIl8D6Jc",
        "colab_type": "code",
        "outputId": "abf2d0e3-ad5f-4313-9511-836e9ee5fd94",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "df[\"G1-binned\"].value_counts(sort=False)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Low       110\n",
              "Medium    189\n",
              "High       95\n",
              "Name: G1-binned, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rGiuv0XHD6Jf",
        "colab_type": "text"
      },
      "source": [
        "Lets plot the distribution of each bin."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2gwfUKv4D6Jf",
        "colab_type": "code",
        "outputId": "ccdbfa65-6c23-4fe6-9ae5-b4177037ec4d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        }
      },
      "source": [
        "pyplot.bar(group_names, df[\"G1-binned\"].value_counts(sort=False))\n",
        "\n",
        "# set x/y labels and plot title\n",
        "plt.pyplot.xlabel(\"G1\")\n",
        "plt.pyplot.ylabel(\"count\")\n",
        "plt.pyplot.title(\"G1 bins\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'G1 bins')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE/JJREFUeJzt3X2QZXV95/H3RxCRBxfjtAR52BEX\nrVXWDElLEiMU6sbgw4bVzSqzakDZHUjUMmXcBLUS0Y27RlHKhwRqXEdEBXFFIrthoywqJCWIMzqO\nPC4PGZYhI7QQHkXiDN/9456GS/sbuD3dt0/P9PtVdarP+Z3fOffbdavvp8/vPNxUFZIkzfSEvguQ\nJC1OBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCGmeJVmepJLsuo31707y3xe6Lmm2DAgJSHJsku8k\nuT/J7d387ydJt/7FSb6Z5O4kG+fyWlX1X6vqP85L4dIYGRBa8pL8IfAx4MPALwL7AicBvwHs1nW7\nH1gD/Oc+apT6YEBoSUvyz4D3A79fVV+uqntr4PtV9fqqehCgqq6oqs8BN81i929O8g9JNid559Br\nnpLk89389HDUcUn+X5IfJ3nPUN/Dk6xNck+S25J8dH5+c+nxNcdIpSXk14EnAV8dw75fDBwCHAx8\nI8n6qvo/2+j7IuA5wLOBK5J8paquYXBk87Gq+lySvYBDx1Cn1OQRhJa6ZcCPq2rLdEOSbye5K8kD\nSY6cw77fV1X3V9UPgc8AKx+n7wNV9QPgB8Avde0/A/5FkmVVdV9VXT6HeqRZMSC01N0BLBu+4qiq\nXlhV+3Tr5vI3csvQ/M3AMx6j74+G5n8C7NXNn8DgqOLaJN9N8qo51CPNigGhpe4y4EHgmDHs+8Ch\n+YOAf5jtDqrq+qpaCTwd+HPgy0n2nKf6pMdkQGhJq6q7gPcBf5nkd5LsneQJSVYAD38Qd227A08c\nLGb3JLttY7fT/iTJHkmeB7wJOHe29SV5Q5KJqnoIuKtrfmi2+5G2hyepteRV1YeS3Ar8EXAWg0ta\nbwL+GPh21+1I4JtDmz0AXAIc9Ri7vgS4gcE/YqdW1de3o7yjgY8m2YPBMNWxVfXAduxHmrX4hUGS\npBaHmCRJTQaEJKnJgJAkNRkQkqSmHfoqpmXLltXy5cv7LkOSdijr1q37cVVNPF6/HTogli9fztq1\na/suQ5J2KEluHqWfQ0ySpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqSmHfpO\nai0ty0/+675L2Glt/OAr+y5Bi5BHEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1\nGRCSpCYDQpLUZEBIkprGFhBJ1iS5PcmVQ23nJlnfTRuTrO/alyd5YGjdGeOqS5I0mnE+rO9M4JPA\nWdMNVfW66fkkHwHuHup/Y1WtGGM9kqRZGFtAVNWlSZa31iUJ8FrgJeN6fUnS3PR1DuII4Laqun6o\n7ZlJvp/kkiRHbGvDJKuSrE2ydmpqavyVStIS1VdArATOGVreDBxUVYcB7wDOTvKU1oZVtbqqJqtq\ncmJiYgFKlaSlacEDIsmuwGuAc6fbqurBqrqjm18H3Ag8e6FrkyQ9oo8jiH8NXFtVm6Ybkkwk2aWb\nPxg4BLiph9okSZ1xXuZ6DnAZ8Jwkm5Kc0K06lkcPLwEcCWzoLnv9MnBSVd05rtokSY9vnFcxrdxG\n+/GNtvOA88ZViyRp9ryTWpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRA\nSJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmsb5ndRrktye5MqhtlOS3Jpk\nfTe9Ymjdu5LckOS6JL81rrokSaMZ5xHEmcDRjfbTqmpFN10IkOS5wLHA87pt/jLJLmOsTZL0OMYW\nEFV1KXDniN2PAb5YVQ9W1d8DNwCHj6s2SdLj6+McxFuTbOiGoJ7ate0P3DLUZ1PX9nOSrEqyNsna\nqampcdcqSUvWQgfE6cCzgBXAZuAjs91BVa2uqsmqmpyYmJjv+iRJnQUNiKq6raq2VtVDwKd4ZBjp\nVuDAoa4HdG2SpJ4saEAk2W9o8dXA9BVOFwDHJnlSkmcChwBXLGRtkqRH23VcO05yDnAUsCzJJuC9\nwFFJVgAFbAROBKiqq5J8Cbga2AK8paq2jqs2SdLjG1tAVNXKRvOnH6P/B4APjKseSdLseCe1JKnJ\ngJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwI\nSVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpKaxBUSSNUluT3LlUNuHk1ybZEOS85Ps07UvT/JAkvXd\ndMa46pIkjWacRxBnAkfPaLsIOLSqng/8X+BdQ+turKoV3XTSGOuSJI1gbAFRVZcCd85o+3pVbekW\nLwcOGNfrS5Lmps9zEG8G/vfQ8jOTfD/JJUmO2NZGSVYlWZtk7dTU1PirlKQlqpeASPIeYAvwha5p\nM3BQVR0GvAM4O8lTWttW1eqqmqyqyYmJiYUpWJKWoAUPiCTHA68CXl9VBVBVD1bVHd38OuBG4NkL\nXZsk6RELGhBJjgb+CPjtqvrJUPtEkl26+YOBQ4CbFrI2SdKj7TquHSc5BzgKWJZkE/BeBlctPQm4\nKAnA5d0VS0cC70/yM+Ah4KSqurO5Y0nSghhbQFTVykbzp7fR9zzgvHHVIkmaPe+kliQ1GRCSpCYD\nQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqSmsT1qY0ew/OS/7ruEndbGD76y7xIk\nzdFIRxBJLh6lTZK083jMI4gkuwN7MHgi61OBdKueAuw/5tokST16vCGmE4E/AJ4BrOORgLgH+OQY\n65Ik9ewxA6KqPgZ8LMnbquoTC1STJGkRGOkkdVV9IskLgeXD21TVWWOqS5LUs5ECIsnngGcB64Gt\nXXMBBoQk7aRGvcx1EnhuVdU4i5EkLR6j3ih3JfCLs915kjVJbk9y5VDbLyS5KMn13c+ndu1J8vEk\nNyTZkOSXZ/t6kqT5M2pALAOuTvK1JBdMTyNsdyZw9Iy2k4GLq+oQ4OJuGeDlwCHdtAo4fcTaJElj\nMOoQ0ynbs/OqujTJ8hnNxwBHdfOfBb4F/HHXflY3jHV5kn2S7FdVm7fntSVJczPqVUyXzONr7jv0\nof8jYN9ufn/glqF+m7q2RwVEklUMjjA46KCD5rEsSdKwUR+1cW+Se7rpp0m2Jrlnri/eHS3M6sR3\nVa2uqsmqmpyYmJhrCZKkbRj1CGLv6fkkYTAc9Gvb+Zq3TQ8dJdkPuL1rvxU4cKjfAV2bJKkHs37c\ndw38FfBb2/maFwDHdfPHAV8dav/d7mqmXwPu9vyDJPVn1BvlXjO0+AQG90X8dITtzmFwQnpZkk3A\ne4EPAl9KcgJwM/DarvuFwCuAG4CfAG8a7VeQJI3DqFcx/Zuh+S3ARgbDTI+pqlZuY9VLG30LeMuI\n9UiSxmzUcxD+Ny9p1vxSrvFZiC/lGvUqpgOSnN/dFX17kvOSHDDu4iRJ/Rn1JPVnGJxEfkY3/c+u\nTZK0kxo1ICaq6jNVtaWbzgS8CUGSdmKjBsQdSd6QZJduegNwxzgLkyT1a9SAeDODy1F/xODRF78D\nHD+mmiRJi8Col7m+Hziuqv4RBo/sBk5lEBySpJ3QqEcQz58OB4CquhM4bDwlSZIWg1ED4gnTX+wD\nDx9BjHr0IUnaAY36If8R4LIk/6Nb/vfAB8ZTkiRpMRj1TuqzkqwFXtI1vaaqrh5fWZKkvo08TNQF\ngqEgSUvErB/3LUlaGgwISVKTASFJajIgJElNBoQkqcmAkCQ1Lfjd0EmeA5w71HQw8KfAPsB/Aqa6\n9ndX1YULXJ4kqbPgAVFV1wErAJLsAtwKnA+8CTitqk5d6JokST+v7yGmlwI3VtXNPdchSZqh74A4\nFjhnaPmtSTYkWTP8cEBJ0sLrLSCS7Ab8NjD9AMDTgWcxGH7azOABga3tViVZm2Tt1NRUq4skaR70\neQTxcuB7VXUbQFXdVlVbq+oh4FPA4a2Nqmp1VU1W1eTEhF+LLUnj0mdArGRoeCnJfkPrXg1cueAV\nSZIe1suX/iTZE/hN4MSh5g8lWQEUsHHGOknSAuslIKrqfuBpM9re2EctkqS2vq9ikiQtUgaEJKnJ\ngJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwI\nSVKTASFJajIgJElNBoQkqcmAkCQ19fKd1ABJNgL3AluBLVU1meQXgHOB5cBG4LVV9Y991ShJS1nf\nRxAvrqoVVTXZLZ8MXFxVhwAXd8uSpB70HRAzHQN8tpv/LPBve6xFkpa0PgOigK8nWZdkVde2b1Vt\n7uZ/BOw7c6Mkq5KsTbJ2ampqoWqVpCWnt3MQwIuq6tYkTwcuSnLt8MqqqiQ1c6OqWg2sBpicnPy5\n9ZKk+dHbEURV3dr9vB04HzgcuC3JfgDdz9v7qk+SlrpeAiLJnkn2np4HXgZcCVwAHNd1Ow74ah/1\nSZL6G2LaFzg/yXQNZ1fV3yT5LvClJCcANwOv7ak+SVryegmIqroJ+KVG+x3ASxe+IknSTIvtMldJ\n0iJhQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJ\nTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaFjwgkhyY5JtJrk5yVZK3d+2nJLk1yfpuesVC1yZJekQf\n30m9BfjDqvpekr2BdUku6tadVlWn9lCTJGmGBQ+IqtoMbO7m701yDbD/QtchSXpsvZ6DSLIcOAz4\nTtf01iQbkqxJ8tRtbLMqydoka6emphaoUklaenoLiCR7AecBf1BV9wCnA88CVjA4wvhIa7uqWl1V\nk1U1OTExsWD1StJS00tAJHkig3D4QlV9BaCqbquqrVX1EPAp4PA+apMkDfRxFVOATwPXVNVHh9r3\nG+r2auDKha5NkvSIPq5i+g3gjcAPk6zv2t4NrEyyAihgI3BiD7VJkjp9XMX0d0Aaqy5c6FokSdvm\nndSSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMB\nIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmhZdQCQ5Osl1SW5IcnLf9UjSUrWoAiLJLsBf\nAC8HngusTPLcfquSpKVpUQUEcDhwQ1XdVFX/BHwROKbnmiRpSdq17wJm2B+4ZWh5E/Crwx2SrAJW\ndYv3JblugWrr2zLgx30XMar8ed8VLAo7zHvm+wXsQO8XzPk9++ejdFpsAfG4qmo1sLrvOhZakrVV\nNdl3HRqd79mOxffr5y22IaZbgQOHlg/o2iRJC2yxBcR3gUOSPDPJbsCxwAU91yRJS9KiGmKqqi1J\n3gp8DdgFWFNVV/Vc1mKx5IbVdgK+ZzsW368ZUlV91yBJWoQW2xCTJGmRMCAkSU0GxCKQ5L6+a9BA\nkkry+aHlXZNMJflfs9zPt5JMdvMXJtlnvmvVaGb+fSU5Psknu/mTkvzu42z/cP+lZlGdpJYWgfuB\nQ5M8uaoeAH6TOV5qXVWvmJfKNO+q6oy+a1jMPIJYpJIsT/KNJBuSXJzkoCS7JPn7DOyTZGuSI7v+\nlyY5pO+6dxIXAq/s5lcC50yvSLJnkjVJrkjy/STHdO1PTvLFJNckOR948tA2G5Ms697TK4fa35nk\nlG7+W0lOS7K228cLknwlyfVJ/mwBfuclKckpSd7Zzb+g+3tbn+TDw+8V8Iwkf9O9Hx/qqdwFZ0As\nXp8APltVzwe+AHy8qrYC1zF4kOGLgO8BRyR5EnBgVV3fW7U7ly8CxybZHXg+8J2hde8BvlFVhwMv\nBj6cZE/g94CfVNW/BN4L/Mp2vO4/dXfyngF8FXgLcChwfJKnbfdvoyd3H/rrk6wH3r+Nfp8BTqyq\nFcDWGetWAK8D/hXwuiQHztx4Z2RALF6/DpzdzX+OQSAA/C1wZDf9t679BQxuMtQ8qKoNwHIGRw8X\nzlj9MuDk7oPmW8DuwEEM3o/PD22/YTteevqm0B8CV1XV5qp6ELiJRz9hQLPzQFWtmJ6AP53ZoTtH\ntHdVXdY1nT2jy8VVdXdV/RS4mhGfZbSjMyB2PJcCRzB48u2FwD7AUQyCQ/PnAuBUhoaXOgH+3dAH\nzkFVdc2I+9zCo//mdp+x/sHu50ND89PLni/s1/D7sZUl8n4YEIvXtxk8agTg9TwSAFcALwQe6v6b\nWQ+cyCA4NH/WAO+rqh/OaP8a8LYkAUhyWNd+KfAfurZDGQxNzXQb8PQkT+uGBV81lso1a1V1F3Bv\nkumnRx/7WP2XCgNicdgjyaah6R3A24A3JdkAvBF4O0A35HALcHm37d8CezMYltA8qapNVfXxxqr/\nAjwR2JDkqm4Z4HRgryTXMBjjXtfY58+6dVcAFwHXjqN2bbcTgE91w4d7Anf3XE/vfNSGJAFJ9qqq\n+7r5k4H9qurtPZfVqyUxjiZJI3hlkncx+Fy8GTi+33L65xGEJKnJcxCSpCYDQpLUZEBIkpoMCGmO\nkuyb5OwkNyVZl+SyJK/u7nf4ZpL7lurTQLVjMyCkOehumPsr4NKqOriqfoXBTVYHAD8F/gR4Z48l\nStvNgJDm5iUMHrL38GOjq+rmqvpEVd1fVX/HICikHY4BIc3N8xg8VVfa6RgQ0jxK8hdJfpDEp+tq\nh2dASHNzFfDL0wtV9RbgpcBEbxVJ88SAkObmG8DuSX5vqG2PvoqR5pOP2pDmKMl+wGnArwJTDL7X\n+oyqOjfJRuApwG7AXcDLqurqvmqVZsOAkCQ1OcQkSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRA\nSJKa/j8KYUK42i8daAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GCVvUyk_D6Ji",
        "colab_type": "text"
      },
      "source": [
        "<p>\n",
        "    Check the dataframe above carefully, you will find the last column provides the bins for \"G1\" with 3 categories (\"Low\",\"Medium\" and \"High\"). \n",
        "</p>\n",
        "<p>\n",
        "    We successfully narrow the intervals to 3!\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UanrhrUBD6Ji",
        "colab_type": "text"
      },
      "source": [
        "<h3>Bins visualization</h3>\n",
        "Normally, a histogram is used to visualize the distribution of bins we created above. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ibWu-4pHD6Jj",
        "colab_type": "code",
        "outputId": "04464492-8261-40bc-c00e-530778325300",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        }
      },
      "source": [
        "\n",
        "# draw historgram of attribute \"G1\" with bins = 3\n",
        "plt.pyplot.hist(df[\"G1\"], bins = 3)\n",
        "\n",
        "# set x/y labels and plot title\n",
        "plt.pyplot.xlabel(\"G1\")\n",
        "plt.pyplot.ylabel(\"count\")\n",
        "plt.pyplot.title(\"G1 bins\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'G1 bins')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFG9JREFUeJzt3X2wZHV95/H3RwiyoAhmrog87AAL\nbqmlgztLjEaWh6zBh5VoZVkozaKyGUnUjcbEgFbiQ5VbRFGW1QRr1BEwSjAgym5MAosuVCqCDvIg\nT4aHDDLjwFwhiCKiA9/9o8+V9vIbpi93uk/Dfb+quu45vz7d/ZmZe+dzz6/POZ2qQpKk+Z7UdwBJ\n0nSyICRJTRaEJKnJgpAkNVkQkqQmC0KS1GRBSNtYkuVJKsn2W7j/3Uk+Nelc0kJZEBKQ5Jgklye5\nL8mmbvn3kqS7/7AkX0vygyTrFvNaVfU/quq/bZPg0hhZEFrykrwTOA34MPBMYHfgBOAlwA7dZvcB\na4A/6iOj1AcLQktakqcBHwB+r6rOraof1sCVVfW6qnoAoKq+UVWfBW5dwNO/Kcn3kmxM8odDr/m+\nJH/ZLc9NRx2X5LtJvp/kPUPbHpxkbZJ7k9yZ5KPb5k8ubV1zjlRaQn4VeDLw5TE892HAAcB+wFeT\nXFVV/3cL2/4a8GzgQOAbSb5YVTcw2LM5rao+m+QpwPPGkFNqcg9CS90y4PtVtXluIMk/Jrknyf1J\nDlnEc7+/qu6rqm8DnwGO3cq291fV1cDVwAu68Z8B/ybJsqr6UVVdtog80oJYEFrq7gKWDR9xVFUv\nrqpdu/sW8zNy+9DybcCzHmXbO4aWfww8pVs+nsFexY1JvpnkVYvIIy2IBaGl7uvAA8BRY3juvYeW\n9wG+t9AnqKqbqupY4BnAnwHnJtl5G+WTHpUFoSWtqu4B3g/8RZLfSvLUJE9KsgL4+X/E3diOwC8N\nVrNjkh228LRz/iTJTkmeC7wROGeh+ZK8PslMVT0E3NMNP7TQ55EeC9+k1pJXVR9KsgF4F3AWg0Na\nbwX+GPjHbrNDgK8NPex+4BLg0Ed56kuAmxn8InZKVV34GOIdCXw0yU4MpqmOqar7H8PzSAsWPzBI\nktTiFJMkqcmCkCQ1WRCSpCYLQpLU9Lg+imnZsmW1fPnyvmNI0uPKFVdc8f2qmtnado/rgli+fDlr\n167tO4YkPa4kuW2U7ZxikiQ1WRCSpCYLQpLUZEFIkposCElSkwUhSWqyICRJTRaEJKnJgpAkNT2u\nz6TWE8/yE/+m7wiaZ93Jr+w7gnriHoQkqcmCkCQ1WRCSpCYLQpLUZEFIkposCElSkwUhSWqyICRJ\nTRaEJKnJgpAkNVkQkqSmsRVEkjVJNiW5dmjsnCRXdbd1Sa7qxpcnuX/ovk+MK5ckaTTjvFjfGcDH\ngbPmBqrqv8wtJ/kI8IOh7W+pqhVjzCNJWoCxFURVXZpkeeu+JAGOBg4f1+tLkhanr/cgXgrcWVU3\nDY3tm+TKJJckeemWHphkVZK1SdbOzs6OP6kkLVF9FcSxwNlD6xuBfarqIOAPgM8n2aX1wKpaXVUr\nq2rlzMzMBKJK0tI08YJIsj3wWuCcubGqeqCq7uqWrwBuAQ6cdDZJ0sP62IP4deDGqlo/N5BkJsl2\n3fJ+wAHArT1kkyR1xnmY69nA14FnJ1mf5PjurmP4xeklgEOAa7rDXs8FTqiqu8eVTZK0deM8iunY\nLYy/oTF2HnDeuLJIkhbOM6klSU0WhCSpyYKQJDVZEJKkJgtCktRkQUiSmiwISVKTBSFJarIgJElN\nFoQkqcmCkCQ1WRCSpCYLQpLUZEFIkposCElSkwUhSWqyICRJTRaEJKlpnJ9JvSbJpiTXDo29L8mG\nJFd1t1cM3XdSkpuTfCfJb4wrlyRpNOPcgzgDOLIxfmpVrehuXwFI8hzgGOC53WP+Isl2Y8wmSdqK\nsRVEVV0K3D3i5kcBf1VVD1TVPwM3AwePK5skaev6eA/irUmu6aagduvG9gRuH9pmfTf2CElWJVmb\nZO3s7Oy4s0rSkjXpgjgd2B9YAWwEPrLQJ6iq1VW1sqpWzszMbOt8kqTORAuiqu6sqger6iHgkzw8\njbQB2Hto0726MUlSTyZaEEn2GFp9DTB3hNMFwDFJnpxkX+AA4BuTzCZJ+kXbj+uJk5wNHAosS7Ie\neC9waJIVQAHrgDcDVNV1Sb4AXA9sBt5SVQ+OK5skaevGVhBVdWxj+NOPsv0HgQ+OK48kaWE8k1qS\n1GRBSJKaLAhJUpMFIUlqsiAkSU0WhCSpyYKQJDVZEJKkJgtCktRkQUiSmiwISVKTBSFJarIgJElN\nFoQkqcmCkCQ1WRCSpCYLQpLUZEFIkposCElS09gKIsmaJJuSXDs09uEkNya5Jsn5SXbtxpcnuT/J\nVd3tE+PKJUkazTj3IM4Ajpw3dhHwvKp6PvBPwElD991SVSu62wljzCVJGsHYCqKqLgXunjd2YVVt\n7lYvA/Ya1+tLkhanz/cg3gT87dD6vkmuTHJJkpdu6UFJViVZm2Tt7Ozs+FNK0hLVS0EkeQ+wGfhc\nN7QR2KeqDgL+APh8kl1aj62q1VW1sqpWzszMTCawJC1BEy+IJG8AXgW8rqoKoKoeqKq7uuUrgFuA\nAyedTZL0sIkWRJIjgXcBr66qHw+NzyTZrlveDzgAuHWS2SRJv2j7cT1xkrOBQ4FlSdYD72Vw1NKT\ngYuSAFzWHbF0CPCBJD8DHgJOqKq7m08sSZqIsRVEVR3bGP70FrY9DzhvXFkkSQvnmdSSpCYLQpLU\nZEFIkposCElSkwUhSWqyICRJTRaEJKnJgpAkNVkQkqQmC0KS1DS2S208Hiw/8W/6jiBJU2ukPYgk\nF48yJkl64njUPYgkOwI7Mbgi625Aurt2AfYcczZJUo+2NsX0ZuDtwLOAK3i4IO4FPj7GXJKknj1q\nQVTVacBpSd5WVR+bUCZJ0hQY6U3qqvpYkhcDy4cfU1VnjSmXJKlnIxVEks8C+wNXAQ92wwVYEJL0\nBDXqYa4rgedUVY0zjCRpeox6oty1wDMX+uRJ1iTZlOTaobGnJ7koyU3d19268ST5X0luTnJNkhcu\n9PUkSdvOqAWxDLg+yd8nuWDuNsLjzgCOnDd2InBxVR0AXNytA7wcOKC7rQJOHzGbJGkMRp1iet9j\nefKqujTJ8nnDRwGHdstnAv8P+ONu/KxuGuuyJLsm2aOqNj6W15YkLc6oRzFdsg1fc/eh//TvAHbv\nlvcEbh/abn039gsFkWQVgz0M9tlnn20YS5I0bNRLbfwwyb3d7SdJHkxy72JfvNtbWNAb31W1uqpW\nVtXKmZmZxUaQJG3BqHsQT51bThIG00Eveoyveefc1FGSPYBN3fgGYO+h7fbqxiRJPVjw5b5r4EvA\nbzzG17wAOK5bPg748tD4f+2OZnoR8APff5Ck/ox6otxrh1afxOC8iJ+M8LizGbwhvSzJeuC9wMnA\nF5IcD9wGHN1t/hXgFcDNwI+BN472R5AkjcOoRzH9p6HlzcA6BtNMj6qqjt3CXUc0ti3gLSPmkSSN\n2ajvQfjbvLRE+cFa02ndya8c+2uMehTTXknO786K3pTkvCR7jTucJKk/o75J/RkGbyI/q7v9725M\nkvQENWpBzFTVZ6pqc3c7A/AkBEl6Ahu1IO5K8vok23W31wN3jTOYJKlfoxbEmxgcjnoHg0tf/Bbw\nhjFlkiRNgVEPc/0AcFxV/QsMLtkNnMKgOCRJT0Cj7kE8f64cAKrqbuCg8USSJE2DUQviSXMf7AM/\n34MYde9DkvQ4NOp/8h8Bvp7kr7v1/wx8cDyRJEnTYNQzqc9KshY4vBt6bVVdP75YkqS+jTxN1BWC\npSBJS8SCL/ctSVoaLAhJUpMFIUlqsiAkSU0WhCSpyYKQJDVN/GzoJM8Gzhka2g/4U2BX4HeA2W78\n3VX1lQnHkyR1Jl4QVfUdYAVAku2ADcD5wBuBU6vqlElnkiQ9Ut9TTEcAt1TVbT3nkCTN03dBHAOc\nPbT+1iTXJFkzfHFASdLk9VYQSXYAXg3MXQDwdGB/BtNPGxlcILD1uFVJ1iZZOzs729pEkrQN9LkH\n8XLgW1V1J0BV3VlVD1bVQ8AngYNbD6qq1VW1sqpWzsz4sdiSNC59FsSxDE0vJdlj6L7XANdOPJEk\n6ed6+dCfJDsD/xF489Dwh5KsAApYN+8+SdKE9VIQVXUf8Mvzxn67jyySpLa+j2KSJE0pC0KS1GRB\nSJKaLAhJUpMFIUlqsiAkSU0WhCSpyYKQJDVZEJKkJgtCktRkQUiSmiwISVKTBSFJarIgJElNFoQk\nqcmCkCQ1WRCSpCYLQpLUZEFIkpp6+UxqgCTrgB8CDwKbq2plkqcD5wDLgXXA0VX1L31llKSlrO89\niMOqakVVrezWTwQurqoDgIu7dUlSD/ouiPmOAs7sls8EfrPHLJK0pPVZEAVcmOSKJKu6sd2ramO3\nfAew+/wHJVmVZG2StbOzs5PKKklLTm/vQQC/VlUbkjwDuCjJjcN3VlUlqfkPqqrVwGqAlStXPuJ+\nSdK20dseRFVt6L5uAs4HDgbuTLIHQPd1U1/5JGmp66Ugkuyc5Klzy8DLgGuBC4Djus2OA77cRz5J\nUn9TTLsD5yeZy/D5qvq7JN8EvpDkeOA24Oie8knSktdLQVTVrcALGuN3AUdMPpEkab5pO8xVkjQl\nLAhJUpMFIUlqsiAkSU0WhCSpyYKQJDVZEJKkJgtCktRkQUiSmiwISVKTBSFJarIgJElNFoQkqcmC\nkCQ1WRCSpCYLQpLUZEFIkposCElS08QLIsneSb6W5Pok1yX5/W78fUk2JLmqu71i0tkkSQ/r4zOp\nNwPvrKpvJXkqcEWSi7r7Tq2qU3rIJEmaZ+IFUVUbgY3d8g+T3ADsOekckqRH1+t7EEmWAwcBl3dD\nb01yTZI1SXbbwmNWJVmbZO3s7OyEkkrS0tNbQSR5CnAe8Paquhc4HdgfWMFgD+MjrcdV1eqqWllV\nK2dmZiaWV5KWml4KIskvMSiHz1XVFwGq6s6qerCqHgI+CRzcRzZJ0kAfRzEF+DRwQ1V9dGh8j6HN\nXgNcO+lskqSH9XEU00uA3wa+neSqbuzdwLFJVgAFrAPe3EM2SVKnj6OY/gFI466vTDqLJGnLPJNa\nktRkQUiSmiwISVKTBSFJarIgJElNFoQkqcmCkCQ1WRCSpCYLQpLUZEFIkposCElSkwUhSWqyICRJ\nTRaEJKnJgpAkNVkQkqQmC0KS1GRBSJKaLAhJUtPUFUSSI5N8J8nNSU7sO48kLVVTVRBJtgP+HHg5\n8Bzg2CTP6TeVJC1NU1UQwMHAzVV1a1X9FPgr4KieM0nSkrR93wHm2RO4fWh9PfArwxskWQWs6lZ/\nlOQ7E8q2NcuA7/cdosFcC2OuhTHXwmyzXPmzRT38X4+y0bQVxFZV1Wpgdd855kuytqpW9p1jPnMt\njLkWxlwLM625tmTappg2AHsPre/VjUmSJmzaCuKbwAFJ9k2yA3AMcEHPmSRpSZqqKaaq2pzkrcDf\nA9sBa6rqup5jjWrqpr065loYcy2MuRZmWnM1par6ziBJmkLTNsUkSZoSFoQkqcmC2AaSbJfkyiT/\np+8sc5LsmuTcJDcmuSHJr/adCSDJO5Jcl+TaJGcn2bHHLGuSbEpy7dDY05NclOSm7utuU5Dpw92/\n4zVJzk+y6yQzPVq2ofvemaSSLJuWXEne1v29XZfkQ9OQK8mKJJcluSrJ2iQHTzrXQlgQ28bvAzf0\nHWKe04C/q6p/C7yAKciXZE/gvwMrq+p5DA5EOKbHSGcAR84bOxG4uKoOAC7u1vvOdBHwvKp6PvBP\nwEkTzjTnDB6ZjSR7Ay8DvjvpQJ0zmJcryWEMrsLwgqp6LnDKNOQCPgS8v6pWAH/arU8tC2KRkuwF\nvBL4VN9Z5iR5GnAI8GmAqvppVd3Tb6qf2x74V0m2B3YCvtdXkKq6FLh73vBRwJnd8pnAb/adqaou\nrKrN3eplDM4Pmrgt/H0BnAq8C+jliJct5Ppd4OSqeqDbZtOU5Cpgl275afT4/T8KC2Lx/ieDH46H\n+g4yZF9gFvhMN/X1qSQ79x2qqjYw+E3uu8BG4AdVdWG/qR5h96ra2C3fAezeZ5iGNwF/23eIOUmO\nAjZU1dV9Z5nnQOClSS5PckmSf993oM7bgQ8nuZ3Bz0Jfe4MjsSAWIcmrgE1VdUXfWebZHnghcHpV\nHQTcx+SnSh6hm88/ikGBPQvYOcnr+021ZTU4BnxqjgNP8h5gM/C5vrMAJNkJeDeDqZJpsz3wdOBF\nwB8BX0iSfiMBgz2bd1TV3sA76Pbyp5UFsTgvAV6dZB2DK88enuQv+40EDC5yuL6qLu/Wz2VQGH37\ndeCfq2q2qn4GfBF4cc+Z5rszyR4A3deJT020JHkD8CrgdTU9Jy/tz6Dsr+5+BvYCvpXkmb2mGlgP\nfLEGvsFgD3/ib6A3HMfg+x7grxlcwXpqWRCLUFUnVdVeVbWcwZutX62q3n8jrqo7gNuTPLsbOgK4\nvsdIc74LvCjJTt1vc0cwBW+ez3MBgx9iuq9f7jELMPgQLQbTmK+uqh/3nWdOVX27qp5RVcu7n4H1\nwAu777++fQk4DCDJgcAOTMfVXb8H/Idu+XDgph6zbNVUXWpD29TbgM9117S6FXhjz3moqsuTnAt8\ni8FUyZX0eOmBJGcDhwLLkqwH3guczGA64njgNuDoKch0EvBk4KJuluSyqjphkrm2lK2qep8i2cLf\n2RpgTXeI6U+B4ya957WFXL8DnNYdpPETHv7ogqnkpTYkSU1OMUmSmiwISVKTBSFJarIgJElNFoQk\nqcmCkBYpye5JPp/k1iRXJPl6ktck+eUkX0vyoyQf7zuntFAWhLQI3Ql/XwIurar9qurfMThpci8G\nx7n/CfCHPUaUHjMLQlqcw4GfVtUn5gaq6raq+lhV3VdV/8CgKKTHHQtCWpznMjgzXHrCsSCkbSjJ\nnye5Osk3+84iLZYFIS3OdQxdKbeq3sLgIoQzvSWSthELQlqcrwI7JvndobGd+gojbUterE9apO5z\nI04FfoXBJ/ndB3yiqs7pPidhFwaXm74HeFlVTcOl16WtsiAkSU1OMUmSmiwISVKTBSFJarIgJElN\nFoQkqcmCkCQ1WRCSpKb/D4NYprPa+aP2AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-A1MDvtSD6Jk",
        "colab_type": "text"
      },
      "source": [
        "The plot above shows the binning result for attribute \"G1\". "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ub5zhh91iQQC",
        "colab_type": "text"
      },
      "source": [
        "#Turning categorical into quantitative variables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZQFC0v6ID6Jk",
        "colab_type": "text"
      },
      "source": [
        "##Indicator variable (or dummy variable)\n",
        "<b>What is an indicator variable?</b>\n",
        "<p>\n",
        "    An indicator variable (or dummy variable) is a numerical variable used to label categories. They are called 'dummies' because the numbers themselves don't have inherent meaning. \n",
        "</p>\n",
        "\n",
        "<b>Why we use indicator variables?</b>\n",
        "<p>\n",
        "    So we can use categorical variables for regression analysis in the later modules.\n",
        "</p>\n",
        "<b>Example</b>\n",
        "<p>\n",
        "    We see the column \"address\" has two unique values, \"R\" or \"U\". Regression doesn't understand words, only numbers. To use this attribute in regression analysis, we convert \"address\" into indicator variables.\n",
        "</p>\n",
        "\n",
        "<p>\n",
        "    We will use the panda's method 'get_dummies' to assign numerical values to different categories of address type. \n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3o_A8umpD6Jm",
        "colab_type": "text"
      },
      "source": [
        "get indicator variables and assign it to data frame \"dummy_variable_1\" "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BsWs0aupD6Jn",
        "colab_type": "code",
        "outputId": "a72ed546-aefc-4a49-cd75-6c9b183c8b49",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "#get indicator variables and assign it to data frame \"dummy_variable_1\"\n",
        "dummy_variable_1 = pd.get_dummies(df[\"address\"])\n",
        "dummy_variable_1.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>R</th>\n",
              "      <th>U</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   R  U\n",
              "0  0  1\n",
              "1  0  1\n",
              "2  0  1\n",
              "3  0  1\n",
              "4  0  1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pT-PI__HD6Jp",
        "colab_type": "text"
      },
      "source": [
        "change column names for clarity "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gkxARWAYD6Jq",
        "colab_type": "code",
        "outputId": "4ab6875b-7d5b-41cc-a398-418d4d1f4ec9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "dummy_variable_1.rename(columns={'R':'address_R', 'U':'address_U'}, inplace=True)\n",
        "dummy_variable_1.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>address_R</th>\n",
              "      <th>address_U</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   address_R  address_U\n",
              "0          0          1\n",
              "1          0          1\n",
              "2          0          1\n",
              "3          0          1\n",
              "4          0          1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WrqLSR5tD6Jr",
        "colab_type": "text"
      },
      "source": [
        "We now have the value 0 to represent \"address_R\" and 1 to represent \"address_U\" in the column \"address\". We will now insert this column back into our original dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rrAYrLz-D6Jr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# merge data frame \"df\" and \"dummy_variable_1\" \n",
        "df = pd.concat([df, dummy_variable_1], axis=1)\n",
        "\n",
        "# drop original column \"fuel-type\" from \"df\"\n",
        "df.drop(\"address\", axis = 1, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DDU8wdziD6Js",
        "colab_type": "code",
        "outputId": "cdea1004-8f4f-4216-9669-58b5e3fdfb8c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "      <th>G1-binned</th>\n",
              "      <th>address_R</th>\n",
              "      <th>address_U</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>F</td>\n",
              "      <td>0.818182</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>at_home</td>\n",
              "      <td>teacher</td>\n",
              "      <td>course</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>6.0</td>\n",
              "      <td>6</td>\n",
              "      <td>Low</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>F</td>\n",
              "      <td>0.772727</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>course</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>6</td>\n",
              "      <td>Low</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>F</td>\n",
              "      <td>0.681818</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>7</td>\n",
              "      <td>8.0</td>\n",
              "      <td>10</td>\n",
              "      <td>Low</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>F</td>\n",
              "      <td>0.681818</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>health</td>\n",
              "      <td>services</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>15</td>\n",
              "      <td>14.0</td>\n",
              "      <td>15</td>\n",
              "      <td>High</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>F</td>\n",
              "      <td>0.727273</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>10.0</td>\n",
              "      <td>10</td>\n",
              "      <td>Low</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  sex       age famsize Pstatus  Medu  ...    G2  G3 G1-binned address_R address_U\n",
              "0   F  0.818182     GT3       A     4  ...   6.0   6       Low         0         1\n",
              "1   F  0.772727     GT3       T     1  ...   5.0   6       Low         0         1\n",
              "2   F  0.681818     LE3       T     1  ...   8.0  10       Low         0         1\n",
              "3   F  0.681818     GT3       T     4  ...  14.0  15      High         0         1\n",
              "4   F  0.727273     GT3       T     3  ...  10.0  10       Low         0         1\n",
              "\n",
              "[5 rows x 34 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qkyl8xu0D6Jt",
        "colab_type": "text"
      },
      "source": [
        "The last two columns are now the indicator variable representation of the address variable. It's all 0s and 1s now."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hjB1_ZW7D6Jz",
        "colab_type": "text"
      },
      "source": [
        "<h3>save the new csv </h3>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vmTH7Ts4D6J0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.to_csv('clean_df.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7xFKsK9HJknW",
        "colab_type": "text"
      },
      "source": [
        "##Easy way to Turn categorical into quantitative"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Brn6XpyMImFf",
        "colab_type": "code",
        "outputId": "c9877d7f-82f4-4ac1-9f00-c1993c169c54",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        }
      },
      "source": [
        "csv_file=\"https://raw.githubusercontent.com/gdg-ml-team/DevFest19/master/student-math.csv\"\n",
        "dataf = pd.read_csv(csv_file)\n",
        "dataf.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>school</th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>18</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>A</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>at_home</td>\n",
              "      <td>teacher</td>\n",
              "      <td>course</td>\n",
              "      <td>mother</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>17</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>course</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>LE3</td>\n",
              "      <td>T</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>at_home</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>7</td>\n",
              "      <td>8</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>15</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>health</td>\n",
              "      <td>services</td>\n",
              "      <td>home</td>\n",
              "      <td>mother</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>15</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>GP</td>\n",
              "      <td>F</td>\n",
              "      <td>16</td>\n",
              "      <td>U</td>\n",
              "      <td>GT3</td>\n",
              "      <td>T</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>other</td>\n",
              "      <td>other</td>\n",
              "      <td>home</td>\n",
              "      <td>father</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>no</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  school sex  age address famsize Pstatus  ...  Walc  health absences  G1  G2  G3\n",
              "0     GP   F   18       U     GT3       A  ...     1       3        6   5   6   6\n",
              "1     GP   F   17       U     GT3       T  ...     1       3        4   5   5   6\n",
              "2     GP   F   15       U     LE3       T  ...     3       3       10   7   8  10\n",
              "3     GP   F   15       U     GT3       T  ...     1       5        2  15  14  15\n",
              "4     GP   F   16       U     GT3       T  ...     2       5        4   6  10  10\n",
              "\n",
              "[5 rows x 33 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hM6DuMm4KYN0",
        "colab_type": "code",
        "outputId": "0a6f8bcd-49b7-4002-ca90-075d5aeb0098",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 663
        }
      },
      "source": [
        "dataf.info()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 395 entries, 0 to 394\n",
            "Data columns (total 33 columns):\n",
            "school        395 non-null object\n",
            "sex           395 non-null object\n",
            "age           395 non-null int64\n",
            "address       395 non-null object\n",
            "famsize       395 non-null object\n",
            "Pstatus       395 non-null object\n",
            "Medu          395 non-null int64\n",
            "Fedu          395 non-null int64\n",
            "Mjob          395 non-null object\n",
            "Fjob          395 non-null object\n",
            "reason        395 non-null object\n",
            "guardian      395 non-null object\n",
            "traveltime    395 non-null int64\n",
            "studytime     395 non-null int64\n",
            "failures      395 non-null int64\n",
            "schoolsup     395 non-null object\n",
            "famsup        395 non-null object\n",
            "paid          395 non-null object\n",
            "activities    395 non-null object\n",
            "nursery       395 non-null object\n",
            "higher        395 non-null object\n",
            "internet      395 non-null object\n",
            "romantic      395 non-null object\n",
            "famrel        395 non-null int64\n",
            "freetime      395 non-null int64\n",
            "goout         395 non-null int64\n",
            "Dalc          395 non-null int64\n",
            "Walc          395 non-null int64\n",
            "health        395 non-null int64\n",
            "absences      395 non-null int64\n",
            "G1            395 non-null int64\n",
            "G2            395 non-null int64\n",
            "G3            395 non-null int64\n",
            "dtypes: int64(16), object(17)\n",
            "memory usage: 101.9+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9QZAlMV0Kfgb",
        "colab_type": "code",
        "outputId": "b6f311b5-ce68-4ed3-9940-299e17222cfb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "columns  = dataf.columns;\n",
        "for coloumn in columns :\n",
        "    if (str(dataf[coloumn].dtype).startswith(\"object\")):\n",
        "            values = dataf[coloumn].unique()\n",
        "            convertor = dict(zip(values,range(len(values))))\n",
        "            print(coloumn,convertor)\n",
        "            dataf[coloumn] = [convertor[item] for item in dataf[coloumn]]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "school {'GP': 0, 'MS': 1}\n",
            "sex {'F': 0, 'M': 1}\n",
            "address {'U': 0, 'R': 1}\n",
            "famsize {'GT3': 0, 'LE3': 1}\n",
            "Pstatus {'A': 0, 'T': 1}\n",
            "Mjob {'at_home': 0, 'health': 1, 'other': 2, 'services': 3, 'teacher': 4}\n",
            "Fjob {'teacher': 0, 'other': 1, 'services': 2, 'health': 3, 'at_home': 4}\n",
            "reason {'course': 0, 'other': 1, 'home': 2, 'reputation': 3}\n",
            "guardian {'mother': 0, 'father': 1, 'other': 2}\n",
            "schoolsup {'yes': 0, 'no': 1}\n",
            "famsup {'no': 0, 'yes': 1}\n",
            "paid {'no': 0, 'yes': 1}\n",
            "activities {'no': 0, 'yes': 1}\n",
            "nursery {'yes': 0, 'no': 1}\n",
            "higher {'yes': 0, 'no': 1}\n",
            "internet {'no': 0, 'yes': 1}\n",
            "romantic {'no': 0, 'yes': 1}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vFuIal3sMQ4O",
        "colab_type": "code",
        "outputId": "2b168bc0-e891-453a-c6b5-f829732db76e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        }
      },
      "source": [
        "dataf.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>school</th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>address</th>\n",
              "      <th>famsize</th>\n",
              "      <th>Pstatus</th>\n",
              "      <th>Medu</th>\n",
              "      <th>Fedu</th>\n",
              "      <th>Mjob</th>\n",
              "      <th>Fjob</th>\n",
              "      <th>reason</th>\n",
              "      <th>guardian</th>\n",
              "      <th>traveltime</th>\n",
              "      <th>studytime</th>\n",
              "      <th>failures</th>\n",
              "      <th>schoolsup</th>\n",
              "      <th>famsup</th>\n",
              "      <th>paid</th>\n",
              "      <th>activities</th>\n",
              "      <th>nursery</th>\n",
              "      <th>higher</th>\n",
              "      <th>internet</th>\n",
              "      <th>romantic</th>\n",
              "      <th>famrel</th>\n",
              "      <th>freetime</th>\n",
              "      <th>goout</th>\n",
              "      <th>Dalc</th>\n",
              "      <th>Walc</th>\n",
              "      <th>health</th>\n",
              "      <th>absences</th>\n",
              "      <th>G1</th>\n",
              "      <th>G2</th>\n",
              "      <th>G3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>18</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>17</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>15</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>7</td>\n",
              "      <td>8</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>15</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>15</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>16</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   school  sex  age  address  famsize  ...  health  absences  G1  G2  G3\n",
              "0       0    0   18        0        0  ...       3         6   5   6   6\n",
              "1       0    0   17        0        0  ...       3         4   5   5   6\n",
              "2       0    0   15        0        1  ...       3        10   7   8  10\n",
              "3       0    0   15        0        0  ...       5         2  15  14  15\n",
              "4       0    0   16        0        0  ...       5         4   6  10  10\n",
              "\n",
              "[5 rows x 33 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vd5EzTlQMVBS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dataf.to_csv('clean_dataf.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}