CISC_873_DM_F22_A2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNR+cIooKFkbAc6+0O8vIqZ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Mostafa3zazi/CISC-873-DM-Data-Mining/blob/main/CISC_873_DM_F22_A2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#CISC-873-DM-F22-A2: Whether a first date will lead to a relationship"
      ],
      "metadata": {
        "id": "r0xMAFacM9qB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Download data form kaggel"
      ],
      "metadata": {
        "id": "gqRLGJ_RMujS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BE_3pLlyC0yP",
        "outputId": "b9bdf278-21b7-423c-fc20-15232b5901cf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# !pip install kaggle --upgrade"
      ],
      "metadata": {
        "id": "Y3_sUwWTCsxA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!mkdir ~/.kaggle\n",
        "!cp '/content/drive/MyDrive/Colab Notebooks/kaggle.json' ~/.kaggle/"
      ],
      "metadata": {
        "id": "MQTtGUT4D2b_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!chmod 600 ~/.kaggle/kaggle.json"
      ],
      "metadata": {
        "id": "kY_9YEADDREz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!kaggle --version"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MPaeJaLbDb9_",
        "outputId": "abc0ae05-0379-4303-e920-0f4db9d14334"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Kaggle API 1.5.4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mhisc8Z2Ce5l",
        "outputId": "8dab3939-ce46-41ce-aee3-c8e76c96c211"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.12 / client 1.5.4)\n",
            "Downloading Speed%20Dating%20Data%20Description.pdf to /content\n",
            "  0% 0.00/202k [00:00<?, ?B/s]\n",
            "100% 202k/202k [00:00<00:00, 77.1MB/s]\n",
            "Downloading train.csv.zip to /content\n",
            "  0% 0.00/942k [00:00<?, ?B/s]\n",
            "100% 942k/942k [00:00<00:00, 62.4MB/s]\n",
            "Downloading test.csv.zip to /content\n",
            "  0% 0.00/391k [00:00<?, ?B/s]\n",
            "100% 391k/391k [00:00<00:00, 127MB/s]\n"
          ]
        }
      ],
      "source": [
        "!kaggle competitions download -c cisc-873-dm-f22-a2"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!unzip train.csv.zip\n",
        "!unzip test.csv.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xi3G6t4GGN9k",
        "outputId": "cb0c9d10-59f3-4d38-b869-48999c7e023b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  train.csv.zip\n",
            "  inflating: train.csv               \n",
            "Archive:  test.csv.zip\n",
            "  inflating: test.csv                \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#install required packages"
      ],
      "metadata": {
        "id": "OJVdPE2RE8bg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install scikit-optimize"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Awz8zWqA1NWQ",
        "outputId": "5fde4a89-8556-4759-c5f5-bea9574de47b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting scikit-optimize\n",
            "  Downloading scikit_optimize-0.9.0-py2.py3-none-any.whl (100 kB)\n",
            "\u001b[?25l\r\u001b[K     |███▎                            | 10 kB 3.4 MB/s eta 0:00:01\r\u001b[K     |██████▌                         | 20 kB 4.9 MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 30 kB 2.8 MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 40 kB 3.7 MB/s eta 0:00:01\r\u001b[K     |████████████████▍               | 51 kB 4.5 MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 61 kB 5.3 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▉         | 71 kB 4.9 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▏     | 81 kB 3.9 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▍  | 92 kB 4.4 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 100 kB 3.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.21.5)\n",
            "Collecting pyaml>=16.9\n",
            "  Downloading pyaml-21.10.1-py2.py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.4.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.1.0)\n",
            "Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.0.2)\n",
            "Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from pyaml>=16.9->scikit-optimize) (3.13)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->scikit-optimize) (3.1.0)\n",
            "Installing collected packages: pyaml, scikit-optimize\n",
            "Successfully installed pyaml-21.10.1 scikit-optimize-0.9.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#inspecting training data"
      ],
      "metadata": {
        "id": "5TY6OLr3NBH4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#import libraries for data exploration and processing\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from matplotlib import pyplot as plt"
      ],
      "metadata": {
        "id": "FcaS7og4EJcQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#read train and test files\n",
        "df_train_original = pd.read_csv('train.csv')\n",
        "df_test = pd.read_csv('test.csv')"
      ],
      "metadata": {
        "id": "FFKv-yf5FGLX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_train_original.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        },
        "id": "KxNEUFtdFQsX",
        "outputId": "99bb6018-55e6-465b-c504-7a6846eb5e77"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-45fe919d-751f-4f04-bd38-f9ae4642fecc\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gender</th>\n",
              "      <th>idg</th>\n",
              "      <th>condtn</th>\n",
              "      <th>wave</th>\n",
              "      <th>round</th>\n",
              "      <th>position</th>\n",
              "      <th>positin1</th>\n",
              "      <th>order</th>\n",
              "      <th>partner</th>\n",
              "      <th>pid</th>\n",
              "      <th>...</th>\n",
              "      <th>sinc3_3</th>\n",
              "      <th>intel3_3</th>\n",
              "      <th>fun3_3</th>\n",
              "      <th>amb3_3</th>\n",
              "      <th>attr5_3</th>\n",
              "      <th>sinc5_3</th>\n",
              "      <th>intel5_3</th>\n",
              "      <th>fun5_3</th>\n",
              "      <th>amb5_3</th>\n",
              "      <th>id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>14</td>\n",
              "      <td>18</td>\n",
              "      <td>2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>14</td>\n",
              "      <td>12</td>\n",
              "      <td>372.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2583</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>14</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>63.0</td>\n",
              "      <td>...</td>\n",
              "      <td>8.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6830</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>14</td>\n",
              "      <td>1</td>\n",
              "      <td>13</td>\n",
              "      <td>10</td>\n",
              "      <td>8</td>\n",
              "      <td>8.0</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>331.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>38</td>\n",
              "      <td>2</td>\n",
              "      <td>9</td>\n",
              "      <td>20</td>\n",
              "      <td>18</td>\n",
              "      <td>13.0</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>200.0</td>\n",
              "      <td>...</td>\n",
              "      <td>9.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5508</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>24</td>\n",
              "      <td>2</td>\n",
              "      <td>14</td>\n",
              "      <td>20</td>\n",
              "      <td>6</td>\n",
              "      <td>6.0</td>\n",
              "      <td>20</td>\n",
              "      <td>17</td>\n",
              "      <td>357.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4828</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 192 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-45fe919d-751f-4f04-bd38-f9ae4642fecc')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-45fe919d-751f-4f04-bd38-f9ae4642fecc button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-45fe919d-751f-4f04-bd38-f9ae4642fecc');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   gender  idg  condtn  wave  round  position  positin1  order  partner  \\\n",
              "0       0    3       2    14     18         2       2.0     14       12   \n",
              "1       1   14       1     3     10         2       NaN      8        8   \n",
              "2       1   14       1    13     10         8       8.0     10       10   \n",
              "3       1   38       2     9     20        18      13.0      6        7   \n",
              "4       1   24       2    14     20         6       6.0     20       17   \n",
              "\n",
              "     pid  ...  sinc3_3  intel3_3  fun3_3  amb3_3  attr5_3  sinc5_3  intel5_3  \\\n",
              "0  372.0  ...      NaN       NaN     NaN     NaN      NaN      NaN       NaN   \n",
              "1   63.0  ...      8.0       8.0     7.0     8.0      NaN      NaN       NaN   \n",
              "2  331.0  ...      NaN       NaN     NaN     NaN      NaN      NaN       NaN   \n",
              "3  200.0  ...      9.0       8.0     8.0     6.0      NaN      NaN       NaN   \n",
              "4  357.0  ...      NaN       NaN     NaN     NaN      NaN      NaN       NaN   \n",
              "\n",
              "   fun5_3  amb5_3    id  \n",
              "0     NaN     NaN  2583  \n",
              "1     NaN     NaN  6830  \n",
              "2     NaN     NaN  4840  \n",
              "3     NaN     NaN  5508  \n",
              "4     NaN     NaN  4828  \n",
              "\n",
              "[5 rows x 192 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "I have read the speed dating describtion PDF and came up with these findings:\n",
        "\n",
        "1. there are some columns that are not relivant to our taks and should be dropped : [ 'id', 'idg', 'partner', 'pid', 'zipcode' ]\n",
        "2. redundant columns:\n",
        "[ 'field', 'career' ] (we can encode career_c and field_cd instade)"
      ],
      "metadata": {
        "id": "WapjUnbxWKyb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train = df_train_original.copy()"
      ],
      "metadata": {
        "id": "ar-02dCMkZz6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "non_relivant_columns =  [ 'id', 'idg', 'partner', 'pid', 'zipcode' ]\n",
        "redundant_columns = [ 'field', 'career' ]\n",
        "df_train.drop(non_relivant_columns + redundant_columns , inplace = True,axis = 1)"
      ],
      "metadata": {
        "id": "Q1kMqfXtH9hc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_train.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YpWaFtj8IY7b",
        "outputId": "106898c6-1615-4383-9dbb-a537ec69fd4b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5909 entries, 0 to 5908\n",
            "Columns: 185 entries, gender to amb5_3\n",
            "dtypes: float64(172), int64(8), object(5)\n",
            "memory usage: 8.3+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "there are 5 catigorical data with string (object) values ... let's find them and see how to preprocess them."
      ],
      "metadata": {
        "id": "ugbyVX8xIdM4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "objcet_columns = df_train.select_dtypes(include=['object'], exclude=None).columns\n",
        "objcet_columns"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HwLfvsnHIsX-",
        "outputId": "6ca20ecc-9d1b-4606-c43f-f2268fff0f68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['undergra', 'mn_sat', 'tuition', 'from', 'income'], dtype='object')"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_train[objcet_columns]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "kEFgY_12J3My",
        "outputId": "721be8ef-3be8-43aa-de6b-49629a0c624c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-be33ed51-6fbb-4241-9e06-426fb9df8693\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>undergra</th>\n",
              "      <th>mn_sat</th>\n",
              "      <th>tuition</th>\n",
              "      <th>from</th>\n",
              "      <th>income</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>University of Michigan-Ann Arbor</td>\n",
              "      <td>1,290.00</td>\n",
              "      <td>21,645.00</td>\n",
              "      <td>Palo Alto, CA</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Boston, MA</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Rizvi College of Architecture, Bombay University</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Bombay, India</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Washington, DC</td>\n",
              "      <td>45,300.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Harvard College</td>\n",
              "      <td>1,400.00</td>\n",
              "      <td>26,019.00</td>\n",
              "      <td>Midwest USA</td>\n",
              "      <td>46,138.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5904</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>New York</td>\n",
              "      <td>65,708.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5905</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Colombia</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5906</th>\n",
              "      <td>University of Washington</td>\n",
              "      <td>1,155.00</td>\n",
              "      <td>13,258.00</td>\n",
              "      <td>Seattle</td>\n",
              "      <td>37,881.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5907</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Canada</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5908</th>\n",
              "      <td>Hamilton College</td>\n",
              "      <td>1,280.00</td>\n",
              "      <td>27,350.00</td>\n",
              "      <td>Cambridge, MA</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5909 rows × 5 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-be33ed51-6fbb-4241-9e06-426fb9df8693')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-be33ed51-6fbb-4241-9e06-426fb9df8693 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-be33ed51-6fbb-4241-9e06-426fb9df8693');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                              undergra    mn_sat    tuition  \\\n",
              "0                     University of Michigan-Ann Arbor  1,290.00  21,645.00   \n",
              "1                                                  NaN       NaN        NaN   \n",
              "2     Rizvi College of Architecture, Bombay University       NaN        NaN   \n",
              "3                                                  NaN       NaN        NaN   \n",
              "4                                      Harvard College  1,400.00  26,019.00   \n",
              "...                                                ...       ...        ...   \n",
              "5904                                               NaN       NaN        NaN   \n",
              "5905                                               NaN       NaN        NaN   \n",
              "5906                          University of Washington  1,155.00  13,258.00   \n",
              "5907                                               NaN       NaN        NaN   \n",
              "5908                                  Hamilton College  1,280.00  27,350.00   \n",
              "\n",
              "                from     income  \n",
              "0      Palo Alto, CA        NaN  \n",
              "1         Boston, MA        NaN  \n",
              "2      Bombay, India        NaN  \n",
              "3     Washington, DC  45,300.00  \n",
              "4        Midwest USA  46,138.00  \n",
              "...              ...        ...  \n",
              "5904        New York  65,708.00  \n",
              "5905        Colombia        NaN  \n",
              "5906         Seattle  37,881.00  \n",
              "5907          Canada        NaN  \n",
              "5908   Cambridge, MA        NaN  \n",
              "\n",
              "[5909 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "there are 3 columns that can be converted to numeric values [ 'mn_sat', 'tuition', 'income' ]"
      ],
      "metadata": {
        "id": "kRmaROg9Qoa2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "obj_num_col = [ 'mn_sat', 'tuition', 'income' ]\n",
        "df_train[obj_num_col] = df_train[obj_num_col].apply(\n",
        "    lambda x: x.str.replace(',', '').astype(float),\n",
        "    axis = 1)\n",
        "df_train[obj_num_col]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "bd5Zs48pKyIn",
        "outputId": "556fdc52-9ddf-452e-dff3-00385a7e77f9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-3453ea45-94c2-47f7-916f-803bebcc8685\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mn_sat</th>\n",
              "      <th>tuition</th>\n",
              "      <th>income</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1290.0</td>\n",
              "      <td>21645.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>45300.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1400.0</td>\n",
              "      <td>26019.0</td>\n",
              "      <td>46138.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5904</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65708.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5905</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5906</th>\n",
              "      <td>1155.0</td>\n",
              "      <td>13258.0</td>\n",
              "      <td>37881.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5907</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5908</th>\n",
              "      <td>1280.0</td>\n",
              "      <td>27350.0</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5909 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3453ea45-94c2-47f7-916f-803bebcc8685')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3453ea45-94c2-47f7-916f-803bebcc8685 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3453ea45-94c2-47f7-916f-803bebcc8685');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "      mn_sat  tuition   income\n",
              "0     1290.0  21645.0      NaN\n",
              "1        NaN      NaN      NaN\n",
              "2        NaN      NaN      NaN\n",
              "3        NaN      NaN  45300.0\n",
              "4     1400.0  26019.0  46138.0\n",
              "...      ...      ...      ...\n",
              "5904     NaN      NaN  65708.0\n",
              "5905     NaN      NaN      NaN\n",
              "5906  1155.0  13258.0  37881.0\n",
              "5907     NaN      NaN      NaN\n",
              "5908  1280.0  27350.0      NaN\n",
              "\n",
              "[5909 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "check if we can encode the rest columns [ 'undergra', 'from' ]"
      ],
      "metadata": {
        "id": "Di_OPMDZVZgR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train['undergra'].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EBuCRsQ0PZlb",
        "outputId": "996970e3-971a-4614-e1a4-482c7ec6ecfd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Harvard                               82\n",
              "UC Berkeley                           78\n",
              "Columbia                              72\n",
              "Yale                                  60\n",
              "NYU                                   51\n",
              "                                      ..\n",
              "National University Of Singapore       4\n",
              "oberlin                                4\n",
              "University of Rochester                3\n",
              "Universidad Catolica de Chile          3\n",
              "Fu Jen Catholic University, Taiwan     3\n",
              "Name: undergra, Length: 241, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_train['from'].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fXRwSMnjV9jE",
        "outputId": "27bb6ca9-db60-4d72-eff1-fed9232d3fc9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "New York            368\n",
              "New Jersey          246\n",
              "California          211\n",
              "China               104\n",
              "New York City        94\n",
              "                   ... \n",
              "china                 4\n",
              "sofia, bg             3\n",
              "Upstate New York      3\n",
              "Greenwich, CT         2\n",
              "Europe                2\n",
              "Name: from, Length: 269, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "So many unique values, is we uesed onehot encoding the result set will contain huge number of columns so we will drop them."
      ],
      "metadata": {
        "id": "hX4cIpTKWAlG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_train.drop(['undergra','from'] , inplace = True,axis = 1)"
      ],
      "metadata": {
        "id": "G9FUvU99WYT0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#check for nan values\n",
        "df_train.isna().sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3AKqOcOycF8O",
        "outputId": "e3186a25-f78d-46c7-8a67-2b27ba24a652"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "gender         0\n",
              "condtn         0\n",
              "wave           0\n",
              "round          0\n",
              "position       0\n",
              "            ... \n",
              "attr5_3     4496\n",
              "sinc5_3     4496\n",
              "intel5_3    4496\n",
              "fun5_3      4496\n",
              "amb5_3      4496\n",
              "Length: 183, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_train.isna().sum().hist();"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        },
        "id": "GKYppdNtOpbL",
        "outputId": "ea9e0eee-9631-4522-9f80-2148c2e450a6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD6CAYAAAC4RRw1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUI0lEQVR4nO3dbYxcZ3mH8esmJiRkqZ0XOrKcqA4iAqVxCXiUBgVVuwmhkCCSDxENjahDU63UloqWVMUpUmmlVjVt05amqLBqUF3JsAkhka3w1tRki5BKwAsB54XUJnVo3OAtxDZdGpWa3v0wz5plPfGMZ3dm9pleP2m15zxzzpz7Xp/8c/aZMzuRmUiS6vOCYRcgSeqNAS5JlTLAJalSBrgkVcoAl6RKGeCSVKmuAjwifisiHo2IRyLiYxFxRkRcGBEPRcT+iLgrIk7vd7GSpB+JTveBR8QG4AvAxZn5XETcDXwKuAa4NzOnI+JDwNcy829O9lznnXdebty4sadCv//973PWWWf1tG8N7K9uo9zfKPcGdfQ3Ozv7ncx86dLxNV3uvwY4MyL+B3gx8AxwJfCL5fHtwO8DJw3wjRs3smfPnm5r/jEzMzOMj4/3tG8N7K9uo9zfKPcGdfQXEU+1G+84hZKZB4E/A75FK7iPArPAkcw8VjZ7GtiwMqVKkrrRzRTK2cAngF8AjgAfB+4Bfj8zX162uQD4dGZe0mb/SWASoNFobJ6enu6p0Pn5ecbGxnratwb2V7dR7m+Ue4M6+puYmJjNzObS8W6mUF4P/Gtm/gdARNwLXAGsi4g15Sr8fOBgu50zcwqYAmg2m9nrryo1/JqzHPZXt1Hub5R7g7r76+YulG8Bl0fEiyMigKuAx4AHgRvKNluAnf0pUZLUTjdz4A/RmjL5CrC37DMFvAd4d0TsB84F7uxjnZKkJbq6CyUz3we8b8nwk8BlK16RJKkrvhNTkiplgEtSpQxwSapUt+/EHLq9B49y89ZPDvy4B7ZdO/BjSlI3vAKXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcoAl6RKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZXqGOAR8YqIeHjR1/ci4jcj4pyIeCAi9pXvZw+iYElSSzcfavxEZl6amZcCm4H/Au4DtgK7M/MiYHdZlyQNyKlOoVwFfDMznwKuA7aX8e3A9StZmCTp5E41wG8EPlaWG5n5TFn+NtBYsaokSR1FZna3YcTpwL8DP52ZhyLiSGauW/T44cw8YR48IiaBSYBGo7F5enq6p0Lnnj3Koed62nVZNm1YO5DjzM/PMzY2NpBjDYP91WuUe4M6+puYmJjNzObS8VP5TMw3AV/JzENl/VBErM/MZyJiPTDXbqfMnAKmAJrNZo6Pj59a5cUdO3Zy+97Bf4TngZvGB3KcmZkZev3Z1MD+6jXKvUHd/Z3KFMrb+NH0CcAuYEtZ3gLsXKmiJEmddRXgEXEWcDVw76LhbcDVEbEPeH1ZlyQNSFdzEpn5feDcJWPfpXVXiiRpCHwnpiRVygCXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcoAl6RKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekSnX7mZjrIuKeiPhGRDweEa+NiHMi4oGI2Fe+n93vYiVJP9LtFfgHgM9k5iuBVwGPA1uB3Zl5EbC7rEuSBqRjgEfEWuDngDsBMvMHmXkEuA7YXjbbDlzfryIlSSeKzDz5BhGXAlPAY7SuvmeBdwEHM3Nd2SaAwwvrS/afBCYBGo3G5unp6Z4KnXv2KIee62nXZdm0Ye1AjjM/P8/Y2NhAjjUM9levUe4N6uhvYmJiNjObS8e7CfAm8EXgisx8KCI+AHwP+I3FgR0RhzPzpPPgzWYz9+zZ01MDd+zYye171/S073Ic2HbtQI4zMzPD+Pj4QI41DPZXr1HuDeroLyLaBng3c+BPA09n5kNl/R7gNcChiFhfnnw9MLdSxUqSOusY4Jn5beDfIuIVZegqWtMpu4AtZWwLsLMvFUqS2up2TuI3gB0RcTrwJPAOWuF/d0TcAjwFvLU/JUqS2ukqwDPzYeCE+RdaV+OSpCHwnZiSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUKQNckirV1UeqRcQB4D+BHwLHMrMZEecAdwEbgQPAWzPzcH/KlCQtdSpX4BOZeWlmLnw25lZgd2ZeBOwu65KkAVnOFMp1wPayvB24fvnlSJK6FZnZeaOIfwUOAwl8ODOnIuJIZq4rjwdweGF9yb6TwCRAo9HYPD093VOhc88e5dBzPe26LJs2rB3Icebn5xkbGxvIsYbB/uo1yr1BHf1NTEzMLpr9OK6rOXDgdZl5MCJ+EnggIr6x+MHMzIho+3+CzJwCpgCazWaOj4+fWuXFHTt2cvvebstdOQduGh/IcWZmZuj1Z1MD+6vXKPcGdffX1RRKZh4s3+eA+4DLgEMRsR6gfJ/rV5GSpBN1DPCIOCsiXrKwDLwBeATYBWwpm20BdvarSEnSibqZk2gA97WmuVkDfDQzPxMRXwbujohbgKeAt/avTEnSUh0DPDOfBF7VZvy7wFX9KEqS1JnvxJSkShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcoAl6RKGeCSVKmuAzwiTouIr0bE/WX9woh4KCL2R8RdEXF6/8qUJC11Klfg7wIeX7T+fuAvMvPlwGHglpUsTJJ0cl0FeEScD1wL/G1ZD+BK4J6yyXbg+n4UKElqLzKz80YR9wB/DLwE+G3gZuCL5eqbiLgA+HRmXtJm30lgEqDRaGyenp7uqdC5Z49y6Lmedl2WTRvWDuQ48/PzjI2NDeRYw2B/9Rrl3qCO/iYmJmYzs7l0fE2nHSPizcBcZs5GxPipHjgzp4ApgGazmePjp/wUANyxYye37+1Y7oo7cNP4QI4zMzNDrz+bGthfvUa5N6i7v24S8QrgLRFxDXAG8BPAB4B1EbEmM48B5wMH+1emJGmpjnPgmXlbZp6fmRuBG4HPZeZNwIPADWWzLcDOvlUpSTrBcu4Dfw/w7ojYD5wL3LkyJUmSunFKk8qZOQPMlOUngctWviRJUjd8J6YkVcoAl6RKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEp1/Ei1iDgD+DzworL9PZn5voi4EJim9XmYs8DbM/MH/SxW6peNWz+5rP1v3XSMm3t8jgPbrl3WsfX/VzdX4P8NXJmZrwIuBd4YEZcD7wf+IjNfDhwGbulfmZKkpToGeLbMl9UXlq8ErgTuKePbgev7UqEkqa3IzM4bRZxGa5rk5cAHgT8FvliuvomIC4BPZ+YlbfadBCYBGo3G5unp6Z4KnXv2KIee62nXZdm0Ye1AjjM/P8/Y2NhAjjUMq72/vQePLmv/xpn0fH4O6hzr1Wr/t1uuGvqbmJiYzczm0vGOc+AAmflD4NKIWAfcB7yy2wNn5hQwBdBsNnN8fLzbXX/MHTt2cvverspdUQduGh/IcWZmZuj1Z1OD1d5fr/PXC27ddKzn83NQ51ivVvu/3XLV3N8p3YWSmUeAB4HXAusiYuGMPR84uMK1SZJOomOAR8RLy5U3EXEmcDXwOK0gv6FstgXY2a8iJUkn6uZ3vvXA9jIP/gLg7sy8PyIeA6Yj4g+BrwJ39rFOSdISHQM8M78OvLrN+JPAZf0oSpLUme/ElKRKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUqW4+1PiCiHgwIh6LiEcj4l1l/JyIeCAi9pXvZ/e/XEnSgm6uwI8Bt2bmxcDlwK9HxMXAVmB3Zl4E7C7rkqQB6RjgmflMZn6lLP8n8DiwAbgO2F422w5c368iJUkniszsfuOIjcDngUuAb2XmujIewOGF9SX7TAKTAI1GY/P09HRPhc49e5RDz/W067Js2rB2IMeZn59nbGxsIMcahtXe396DR5e1f+NMej4/B3WO9aof/3bL/Xn3qt3PerWfmwATExOzmdlcOt51gEfEGPBPwB9l5r0RcWRxYEfE4cw86Tx4s9nMPXv2nGLpLXfs2Mnte9f0tO9yHNh27UCOMzMzw/j4+ECONQyrvb+NWz+5rP1v3XSs5/NzUOdYr/rxb7fcn3ev2v2sV/u5CRARbQO8q7tQIuKFwCeAHZl5bxk+FBHry+PrgbmVKlaS1Fk3d6EEcCfweGb++aKHdgFbyvIWYOfKlydJej7d/M53BfB2YG9EPFzGfhfYBtwdEbcATwFv7U+JkqR2OgZ4Zn4BiOd5+KqVLUeS1C3fiSlJlTLAJalSg78vrzKDut3p1k3HuHnJsVb77WWShssrcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEr51wh1gn78BcZ2f22xHf8Co9S9bj4T8yMRMRcRjywaOyciHoiIfeX7ST+NXpK08rq5Av874K+Bv180thXYnZnbImJrWX/PypcnqV+6/U2r29+eNHgdr8Az8/PAs0uGrwO2l+XtwPUrXJckqYNeX8RsZOYzZfnbQGOF6pEkdSkys/NGERuB+zPzkrJ+JDPXLXr8cGa2nQePiElgEqDRaGyenp7uqdC5Z49y6Lmedq1C40xO6G/ThrVDqWXvwaMr/pzt+mun1p677a+d1d7zcnpbbdr9rOfn5xkbGxtCNd2bmJiYzczm0vFe70I5FBHrM/OZiFgPzD3fhpk5BUwBNJvNHB8f7+mAd+zYye17R/emmVs3HTuhvwM3jQ+lln7Md7brr51ae+62v3ZWe8/L6W21afeznpmZoddcGrZep1B2AVvK8hZg58qUI0nqVje3EX4M+GfgFRHxdETcAmwDro6IfcDry7okaYA6/l6UmW97noeuWuFaJEmnwLfSS1KlDHBJqpQBLkmVGo17g0ZUP/6olKTR4RW4JFXKAJekShngklQpA1ySKuWLmFpVfOFW6p5X4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcr7wKUh89539corcEmq1LICPCLeGBFPRMT+iNi6UkVJkjrreQolIk4DPghcDTwNfDkidmXmYytVnCStpHbTVbduOsbNfZ7GOrDt2r4873KuwC8D9mfmk5n5A2AauG5lypIkdbKcAN8A/Nui9afLmCRpACIze9sx4gbgjZn5K2X97cDPZuY7l2w3CUyW1VcAT/RY63nAd3rctwb2V7dR7m+Ue4M6+vupzHzp0sHl3EZ4ELhg0fr5ZezHZOYUMLWM4wAQEXsys7nc51mt7K9uo9zfKPcGdfe3nCmULwMXRcSFEXE6cCOwa2XKkiR10vMVeGYei4h3Ap8FTgM+kpmPrlhlkqSTWtY7MTPzU8CnVqiWTpY9DbPK2V/dRrm/Ue4NKu6v5xcxJUnD5VvpJalSVQR4jW/Zj4iPRMRcRDyyaOyciHggIvaV72eX8YiIvyr9fT0iXrNony1l+30RsWUYvbQTERdExIMR8VhEPBoR7yrjI9FjRJwREV+KiK+V/v6gjF8YEQ+VPu4qL+ATES8q6/vL4xsXPddtZfyJiPj54XR0oog4LSK+GhH3l/WR6Q0gIg5ExN6IeDgi9pSxkTg/j8vMVf1F6wXSbwIvA04HvgZcPOy6uqj754DXAI8sGvsTYGtZ3gq8vyxfA3waCOBy4KEyfg7wZPl+dlk+e9i9ldrWA68pyy8B/gW4eFR6LHWOleUXAg+Vuu8GbizjHwJ+tSz/GvChsnwjcFdZvricsy8CLizn8mnD7q/U9m7go8D9ZX1keiv1HQDOWzI2Eufn8X6GXUAX/wivBT67aP024LZh19Vl7RuXBPgTwPqyvB54oix/GHjb0u2AtwEfXjT+Y9utpi9gJ62/izNyPQIvBr4C/CytN3ysKePHz01ad2O9tiyvKdvF0vN18XZD7ul8YDdwJXB/qXUkeltUT7sAH6nzs4YplFF6y34jM58py98GGmX5+XqsovfyK/WraV2ljkyPZYrhYWAOeIDWFeaRzDxWNllc6/E+yuNHgXNZvf39JfA7wP+W9XMZnd4WJPAPETFb3hEOI3R+gh/oMDSZmRFR/S1AETEGfAL4zcz8XkQcf6z2HjPzh8ClEbEOuA945ZBLWhER8WZgLjNnI2J82PX00esy82BE/CTwQER8Y/GDtZ+fUMeLmF29Zb8ShyJiPUD5PlfGn6/HVd17RLyQVnjvyMx7y/BI9QiQmUeAB2lNK6yLiIULn8W1Hu+jPL4W+C6rs78rgLdExAFaf0X0SuADjEZvx2XmwfJ9jtb/gC9jxM7PGgJ8lN6yvwtYeBV7C61544XxXyqvhF8OHC2/5n0WeENEnF1eLX9DGRu6aF1q3wk8npl/vuihkegxIl5arryJiDNpze8/TivIbyibLe1voe8bgM9la9J0F3BjuZPjQuAi4EuD6aK9zLwtM8/PzI20/nv6XGbexAj0tiAizoqIlyws0zqvHmFEzs/jhj0J3+WLEdfQusvhm8B7h11PlzV/DHgG+B9a82a30Jo33A3sA/4ROKdsG7Q+HOObwF6gueh5fhnYX77eMey+FtX1OlpzjF8HHi5f14xKj8DPAF8t/T0C/F4ZfxmtkNoPfBx4URk/o6zvL4+/bNFzvbf0/QTwpmH3tqTPcX50F8rI9FZ6+Vr5enQhN0bl/Fz48p2YklSpGqZQJEltGOCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXq/wDvQkW4qsfJnQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "so many nan values"
      ],
      "metadata": {
        "id": "4qKz-SRDOsg8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#check duplicated\n",
        "df_train.duplicated().sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q2tTFewbPFek",
        "outputId": "fee56ab2-b0b4-4d95-c7d4-1be093ed427f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Problem Formulation\n",
        "the objective is to predict the outcome of a specific speed dating session based on the profile of two people, so we can implement a recommendation system to better match people in speed dating events.\n",
        "\n",
        "the input is: a data sample (information about the dating session)\n",
        "\n",
        "the output is:  predict the probability (0-1, float) that the dating session will lead to a successful match.\n",
        "\n",
        "this is a binary Classification in which we predict a probability using ROCAUC as the evaluation metric. The main challenge is that the data contains missing values, catigorical, and nominal data. we can handel the missing values by dropping them (if we do so we will drop more than half of our data) so we need to find the best way to fill these value (using mean or average). Three different search algorithms will be used for hyperparameter optimization (Grid Search, Random Search, and Bayesian search). and different models will be used (xgboost , SVM , decission tree). the ideal solution would be fining the best stratigy to replace null vlaues and the optimal hyperparameters for the suitable model. the impact is that we will have a powerfull model to be used in future dating session so that we can match the best partners together.\n",
        "\n"
      ],
      "metadata": {
        "id": "dUum1rUBI0r-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Preprocessing Data pipeline"
      ],
      "metadata": {
        "id": "PTizB7XAULWk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.svm import SVC\n",
        "from sklearn.model_selection import cross_val_score, GridSearchCV, StratifiedKFold, RandomizedSearchCV\n",
        "from skopt import BayesSearchCV\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn import preprocessing\n",
        "from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.base import TransformerMixin,BaseEstimator\n",
        "from skopt.space import Real, Categorical, Integer"
      ],
      "metadata": {
        "id": "buhKz8y6dqzu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# #nominal columns\n",
        "# nominal_columns = ['wave' , 'position' , 'positin1' , 'order' , 'race_o' , 'field_cd' , \n",
        "#                    'race' , 'goal' , 'date' , 'go_out' , 'career_c' , 'length' , 'numdat_2','round']\n",
        "# df_train = pd.get_dummies(df_train, prefix = nominal_columns, columns = nominal_columns)"
      ],
      "metadata": {
        "id": "IOCDQxPwWfN1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# df_train_trial0 = df_train.copy()"
      ],
      "metadata": {
        "id": "EWr911yybRsX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class pandas_fill_na(BaseEstimator, TransformerMixin):\n",
        "  def __init__(self, fill_type):\n",
        "      self.fill_type = fill_type\n",
        "  def fit(self, X, y=None):\n",
        "      return self\n",
        "  def transform(self, X):\n",
        "      assert isinstance(X, pd.DataFrame)\n",
        "      if self.fill_type is 'mean':\n",
        "        return X.fillna(X.mean())\n",
        "      if self.fill_type is 'median':\n",
        "        return X.fillna(X.median())\n",
        "      # if self.fill_type is 'mode':\n",
        "      #   return X.fillna(X.mode())"
      ],
      "metadata": {
        "id": "u7_TGH8BZH33"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# x.fillna(x.mode()).isna().sum().sum()"
      ],
      "metadata": {
        "id": "QFrHsfb5xyjZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class pandas_get_dummies(BaseEstimator, TransformerMixin):\n",
        "  def __init__(self, nominal_columns):\n",
        "      self.nominal_columns = nominal_columns\n",
        "  def fit(self, X, y=None):\n",
        "      return self\n",
        "  def transform(self, X):\n",
        "      return pd.get_dummies(X, prefix = self.nominal_columns, columns = self.nominal_columns)"
      ],
      "metadata": {
        "id": "IR7HcF8piUVv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# df_train.fillna(df_train.median(),inplace = True)"
      ],
      "metadata": {
        "id": "Nyt_4snhcAb6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#nominal columns\n",
        "nominal_columns = ['wave' , 'position' , 'positin1' , 'order' , 'race_o' , 'field_cd' , \n",
        "                   'race' , 'goal' , 'date' , 'go_out' , 'career_c' , 'length' , 'numdat_2','round']\n",
        "preprocess_data = Pipeline(\n",
        "    steps = [\n",
        "    (\"fillna\", pandas_fill_na(fill_type = 'mean')),\n",
        "    (\"onehotencode\", pandas_get_dummies(nominal_columns = nominal_columns)),\n",
        "    (\"scaler\" ,preprocessing.StandardScaler()) ])"
      ],
      "metadata": {
        "id": "-8uZwybgan2s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def x_y_split(df,y_lables = 'target'):\n",
        "  return df.copy().iloc[:, df.columns != y_lables] , df[y_lables]"
      ],
      "metadata": {
        "id": "VLDa7XCGck7-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x , y = x_y_split(df = df_train, y_lables = 'match')"
      ],
      "metadata": {
        "id": "DT73KbfYcnEA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# import warnings filter\n",
        "from warnings import simplefilter\n",
        "# ignore all future warnings\n",
        "simplefilter(action='ignore', category=FutureWarning)"
      ],
      "metadata": {
        "id": "LZBomNeIvF9E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Trial 0\n",
        "in this trial we will use simlple svm model (with default parameters) and try grid search using only the fill_type hyperparameter (try with mean and median)"
      ],
      "metadata": {
        "id": "S9GIRPyDnLGI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail0_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"svc\",SVC())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "GcbQEs3Anq03"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "param = {\n",
        "    'preprocess_data__fillna__fill_type': ['mean','median'],\n",
        "}"
      ],
      "metadata": {
        "id": "3oerplMhoLf9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search = GridSearchCV(\n",
        "    trail0_pipeline, param, cv=3, verbose=1,refit=True, scoring='roc_auc')\n",
        "grid_search.get_params().keys()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pz5d-5yxocAl",
        "outputId": "cac90bd8-6751-4421-cda7-52467d5d6e06"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['cv', 'error_score', 'estimator__memory', 'estimator__steps', 'estimator__verbose', 'estimator__preprocess_data', 'estimator__svc', 'estimator__preprocess_data__memory', 'estimator__preprocess_data__steps', 'estimator__preprocess_data__verbose', 'estimator__preprocess_data__fillna', 'estimator__preprocess_data__onehotencode', 'estimator__preprocess_data__scaler', 'estimator__preprocess_data__fillna__fill_type', 'estimator__preprocess_data__onehotencode__nominal_columns', 'estimator__preprocess_data__scaler__copy', 'estimator__preprocess_data__scaler__with_mean', 'estimator__preprocess_data__scaler__with_std', 'estimator__svc__C', 'estimator__svc__break_ties', 'estimator__svc__cache_size', 'estimator__svc__class_weight', 'estimator__svc__coef0', 'estimator__svc__decision_function_shape', 'estimator__svc__degree', 'estimator__svc__gamma', 'estimator__svc__kernel', 'estimator__svc__max_iter', 'estimator__svc__probability', 'estimator__svc__random_state', 'estimator__svc__shrinking', 'estimator__svc__tol', 'estimator__svc__verbose', 'estimator', 'n_jobs', 'param_grid', 'pre_dispatch', 'refit', 'return_train_score', 'scoring', 'verbose'])"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.fit(x, y);"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iqD8AeOWpCne",
        "outputId": "05f995fa-fe6c-4865-bda5-e336294cf38b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 2 candidates, totalling 6 fits\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "results = pd.DataFrame(grid_search.cv_results_)\n",
        "results\n",
        "# GBC_best = [grid_search.best_score_ , grid_search.best_params_]\n",
        "# GBC_resutls[GBC_resutls['rank_test_score'] < 5]\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 210
        },
        "id": "NWz6ebBhr52O",
        "outputId": "4c22a995-9643-40fd-eac8-dd638413f7ab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-d51e47ef-6062-4a38-b1f4-44f080f16be1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.672043</td>\n",
              "      <td>0.234928</td>\n",
              "      <td>1.309494</td>\n",
              "      <td>0.309270</td>\n",
              "      <td>mean</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'mean'}</td>\n",
              "      <td>0.853320</td>\n",
              "      <td>0.843716</td>\n",
              "      <td>0.844388</td>\n",
              "      <td>0.847141</td>\n",
              "      <td>0.004378</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.868165</td>\n",
              "      <td>0.094893</td>\n",
              "      <td>1.570251</td>\n",
              "      <td>0.125753</td>\n",
              "      <td>median</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median'}</td>\n",
              "      <td>0.856059</td>\n",
              "      <td>0.847086</td>\n",
              "      <td>0.845305</td>\n",
              "      <td>0.849484</td>\n",
              "      <td>0.004706</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d51e47ef-6062-4a38-b1f4-44f080f16be1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d51e47ef-6062-4a38-b1f4-44f080f16be1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d51e47ef-6062-4a38-b1f4-44f080f16be1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0       1.672043      0.234928         1.309494        0.309270   \n",
              "1       1.868165      0.094893         1.570251        0.125753   \n",
              "\n",
              "  param_preprocess_data__fillna__fill_type  \\\n",
              "0                                     mean   \n",
              "1                                   median   \n",
              "\n",
              "                                             params  split0_test_score  \\\n",
              "0    {'preprocess_data__fillna__fill_type': 'mean'}           0.853320   \n",
              "1  {'preprocess_data__fillna__fill_type': 'median'}           0.856059   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.843716           0.844388         0.847141        0.004378   \n",
              "1           0.847086           0.845305         0.849484        0.004706   \n",
              "\n",
              "   rank_test_score  \n",
              "0                2  \n",
              "1                1  "
            ]
          },
          "metadata": {},
          "execution_count": 400
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.best_params_ , grid_search.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gJ9UvjoDyxwV",
        "outputId": "a8b20a43-875b-4e8e-9750-b1e92d0da12a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'preprocess_data__fillna__fill_type': 'median'}, 0.8494836133626312)"
            ]
          },
          "metadata": {},
          "execution_count": 401
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Trial 1\n",
        "form trail zero we can see that the median gave use better accuracy\n",
        "\n",
        "let's used the median and turne other parameters useing:\n",
        "* grid search\n",
        "* random search"
      ],
      "metadata": {
        "id": "3WXJHkZnv38r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail1_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"svc\",SVC())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "gpdp1dsMzZA4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "param = {\n",
        "    'preprocess_data__fillna__fill_type': ['median'],\n",
        "    'svc__gamma': ['scale','auto'],\n",
        "    'svc__C' : [0.01 , 0.1 ,1 ,100 ,100000]\n",
        "}"
      ],
      "metadata": {
        "id": "U6BWWBX1r1bW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search = GridSearchCV(\n",
        "    trail1_pipeline, param, cv=3, verbose=1,refit=True)\n",
        "grid_search.get_params().keys()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dLhfATepyr7z",
        "outputId": "6c2babc4-96f1-4a00-b8b5-0d1c291c0239"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['cv', 'error_score', 'estimator__memory', 'estimator__steps', 'estimator__verbose', 'estimator__preprocess_data', 'estimator__svc', 'estimator__preprocess_data__memory', 'estimator__preprocess_data__steps', 'estimator__preprocess_data__verbose', 'estimator__preprocess_data__fillna', 'estimator__preprocess_data__onehotencode', 'estimator__preprocess_data__scaler', 'estimator__preprocess_data__fillna__fill_type', 'estimator__preprocess_data__onehotencode__nominal_columns', 'estimator__preprocess_data__scaler__copy', 'estimator__preprocess_data__scaler__with_mean', 'estimator__preprocess_data__scaler__with_std', 'estimator__svc__C', 'estimator__svc__break_ties', 'estimator__svc__cache_size', 'estimator__svc__class_weight', 'estimator__svc__coef0', 'estimator__svc__decision_function_shape', 'estimator__svc__degree', 'estimator__svc__gamma', 'estimator__svc__kernel', 'estimator__svc__max_iter', 'estimator__svc__probability', 'estimator__svc__random_state', 'estimator__svc__shrinking', 'estimator__svc__tol', 'estimator__svc__verbose', 'estimator', 'n_jobs', 'param_grid', 'pre_dispatch', 'refit', 'return_train_score', 'scoring', 'verbose'])"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.fit(x, y);"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TS25AUc6yu5q",
        "outputId": "10ae01e7-224e-48ad-83bb-fbb535c83767"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "results = pd.DataFrame(grid_search.cv_results_)\n",
        "results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 600
        },
        "id": "TcYUP7vgz1_j",
        "outputId": "c736b865-9f14-4517-8b49-0488f8e4a240"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-d6a09b75-61aa-4380-9495-9d4db5e845fa\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>param_svc__C</th>\n",
              "      <th>param_svc__gamma</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.273134</td>\n",
              "      <td>0.007412</td>\n",
              "      <td>0.970797</td>\n",
              "      <td>0.019507</td>\n",
              "      <td>median</td>\n",
              "      <td>0.01</td>\n",
              "      <td>scale</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.832995</td>\n",
              "      <td>0.832487</td>\n",
              "      <td>0.832910</td>\n",
              "      <td>0.832797</td>\n",
              "      <td>0.000222</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.249317</td>\n",
              "      <td>0.016585</td>\n",
              "      <td>0.956454</td>\n",
              "      <td>0.005707</td>\n",
              "      <td>median</td>\n",
              "      <td>0.01</td>\n",
              "      <td>auto</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.832995</td>\n",
              "      <td>0.832487</td>\n",
              "      <td>0.832910</td>\n",
              "      <td>0.832797</td>\n",
              "      <td>0.000222</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.765318</td>\n",
              "      <td>0.068521</td>\n",
              "      <td>1.900324</td>\n",
              "      <td>0.057404</td>\n",
              "      <td>median</td>\n",
              "      <td>0.1</td>\n",
              "      <td>scale</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.832995</td>\n",
              "      <td>0.832487</td>\n",
              "      <td>0.832910</td>\n",
              "      <td>0.832797</td>\n",
              "      <td>0.000222</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.527338</td>\n",
              "      <td>0.254744</td>\n",
              "      <td>1.227284</td>\n",
              "      <td>0.328857</td>\n",
              "      <td>median</td>\n",
              "      <td>0.1</td>\n",
              "      <td>auto</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.832995</td>\n",
              "      <td>0.832487</td>\n",
              "      <td>0.832910</td>\n",
              "      <td>0.832797</td>\n",
              "      <td>0.000222</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.348423</td>\n",
              "      <td>0.011442</td>\n",
              "      <td>1.015957</td>\n",
              "      <td>0.017997</td>\n",
              "      <td>median</td>\n",
              "      <td>1</td>\n",
              "      <td>scale</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.851269</td>\n",
              "      <td>0.848731</td>\n",
              "      <td>0.849162</td>\n",
              "      <td>0.849721</td>\n",
              "      <td>0.001109</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1.354392</td>\n",
              "      <td>0.011382</td>\n",
              "      <td>1.006000</td>\n",
              "      <td>0.001943</td>\n",
              "      <td>median</td>\n",
              "      <td>1</td>\n",
              "      <td>auto</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.851269</td>\n",
              "      <td>0.848731</td>\n",
              "      <td>0.849162</td>\n",
              "      <td>0.849721</td>\n",
              "      <td>0.001109</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>2.243268</td>\n",
              "      <td>0.418010</td>\n",
              "      <td>1.552314</td>\n",
              "      <td>0.282084</td>\n",
              "      <td>median</td>\n",
              "      <td>100</td>\n",
              "      <td>scale</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.861421</td>\n",
              "      <td>0.853299</td>\n",
              "      <td>0.852717</td>\n",
              "      <td>0.855813</td>\n",
              "      <td>0.003973</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1.873899</td>\n",
              "      <td>0.294345</td>\n",
              "      <td>1.154009</td>\n",
              "      <td>0.012589</td>\n",
              "      <td>median</td>\n",
              "      <td>100</td>\n",
              "      <td>auto</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.861421</td>\n",
              "      <td>0.853299</td>\n",
              "      <td>0.852717</td>\n",
              "      <td>0.855813</td>\n",
              "      <td>0.003973</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1.869826</td>\n",
              "      <td>0.314828</td>\n",
              "      <td>1.150411</td>\n",
              "      <td>0.018079</td>\n",
              "      <td>median</td>\n",
              "      <td>100000</td>\n",
              "      <td>scale</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.861421</td>\n",
              "      <td>0.853299</td>\n",
              "      <td>0.852717</td>\n",
              "      <td>0.855813</td>\n",
              "      <td>0.003973</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1.855017</td>\n",
              "      <td>0.294378</td>\n",
              "      <td>1.143076</td>\n",
              "      <td>0.008951</td>\n",
              "      <td>median</td>\n",
              "      <td>100000</td>\n",
              "      <td>auto</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.861421</td>\n",
              "      <td>0.853299</td>\n",
              "      <td>0.852717</td>\n",
              "      <td>0.855813</td>\n",
              "      <td>0.003973</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d6a09b75-61aa-4380-9495-9d4db5e845fa')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d6a09b75-61aa-4380-9495-9d4db5e845fa button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d6a09b75-61aa-4380-9495-9d4db5e845fa');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0       1.273134      0.007412         0.970797        0.019507   \n",
              "1       1.249317      0.016585         0.956454        0.005707   \n",
              "2       1.765318      0.068521         1.900324        0.057404   \n",
              "3       1.527338      0.254744         1.227284        0.328857   \n",
              "4       1.348423      0.011442         1.015957        0.017997   \n",
              "5       1.354392      0.011382         1.006000        0.001943   \n",
              "6       2.243268      0.418010         1.552314        0.282084   \n",
              "7       1.873899      0.294345         1.154009        0.012589   \n",
              "8       1.869826      0.314828         1.150411        0.018079   \n",
              "9       1.855017      0.294378         1.143076        0.008951   \n",
              "\n",
              "  param_preprocess_data__fillna__fill_type param_svc__C param_svc__gamma  \\\n",
              "0                                   median         0.01            scale   \n",
              "1                                   median         0.01             auto   \n",
              "2                                   median          0.1            scale   \n",
              "3                                   median          0.1             auto   \n",
              "4                                   median            1            scale   \n",
              "5                                   median            1             auto   \n",
              "6                                   median          100            scale   \n",
              "7                                   median          100             auto   \n",
              "8                                   median       100000            scale   \n",
              "9                                   median       100000             auto   \n",
              "\n",
              "                                              params  split0_test_score  \\\n",
              "0  {'preprocess_data__fillna__fill_type': 'median...           0.832995   \n",
              "1  {'preprocess_data__fillna__fill_type': 'median...           0.832995   \n",
              "2  {'preprocess_data__fillna__fill_type': 'median...           0.832995   \n",
              "3  {'preprocess_data__fillna__fill_type': 'median...           0.832995   \n",
              "4  {'preprocess_data__fillna__fill_type': 'median...           0.851269   \n",
              "5  {'preprocess_data__fillna__fill_type': 'median...           0.851269   \n",
              "6  {'preprocess_data__fillna__fill_type': 'median...           0.861421   \n",
              "7  {'preprocess_data__fillna__fill_type': 'median...           0.861421   \n",
              "8  {'preprocess_data__fillna__fill_type': 'median...           0.861421   \n",
              "9  {'preprocess_data__fillna__fill_type': 'median...           0.861421   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.832487           0.832910         0.832797        0.000222   \n",
              "1           0.832487           0.832910         0.832797        0.000222   \n",
              "2           0.832487           0.832910         0.832797        0.000222   \n",
              "3           0.832487           0.832910         0.832797        0.000222   \n",
              "4           0.848731           0.849162         0.849721        0.001109   \n",
              "5           0.848731           0.849162         0.849721        0.001109   \n",
              "6           0.853299           0.852717         0.855813        0.003973   \n",
              "7           0.853299           0.852717         0.855813        0.003973   \n",
              "8           0.853299           0.852717         0.855813        0.003973   \n",
              "9           0.853299           0.852717         0.855813        0.003973   \n",
              "\n",
              "   rank_test_score  \n",
              "0                7  \n",
              "1                7  \n",
              "2                7  \n",
              "3                7  \n",
              "4                5  \n",
              "5                5  \n",
              "6                1  \n",
              "7                1  \n",
              "8                1  \n",
              "9                1  "
            ]
          },
          "metadata": {},
          "execution_count": 459
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.best_params_ , grid_search.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dh9gypf3z3L7",
        "outputId": "1270bb21-e71e-4a53-ec01-1679cbaee130"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'preprocess_data__fillna__fill_type': 'median',\n",
              "  'svc__C': 100,\n",
              "  'svc__gamma': 'scale'},\n",
              " 0.8558126424898963)"
            ]
          },
          "metadata": {},
          "execution_count": 423
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gMiUWlEA1wAd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "SVC_poly_parametres = {\n",
        "    'preprocess_data__fillna__fill_type': ['median'],\n",
        "    'svc__C': [0.01, 0.1, 1, 10, 100, 1000,1e5],\n",
        "    'svc__degree': np.arange(4,20),\n",
        "    'svc__kernel': ['poly']\n",
        "}\n",
        "\n",
        "param_list = [SVC_poly_parametres]\n"
      ],
      "metadata": {
        "id": "MnXADEfK1pba"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rnds = RandomizedSearchCV(trail1_pipeline, param_list , cv=3, n_iter=6, verbose=3,scoring='roc_auc',error_score='raise')"
      ],
      "metadata": {
        "id": "sO8FEDIez3IW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rnds.fit(x,y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SiiZJGAH2-eN",
        "outputId": "046e064c-42a8-4669-8208-fe64cafd2beb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 6 candidates, totalling 18 fits\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=1000, svc__degree=13, svc__kernel=poly;, score=0.640 total time=   5.6s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=1000, svc__degree=13, svc__kernel=poly;, score=0.627 total time=   4.0s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=1000, svc__degree=13, svc__kernel=poly;, score=0.648 total time=   4.0s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=1000, svc__degree=5, svc__kernel=poly;, score=0.708 total time=   3.6s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=1000, svc__degree=5, svc__kernel=poly;, score=0.697 total time=   3.7s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=1000, svc__degree=5, svc__kernel=poly;, score=0.714 total time=   3.7s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=4, svc__kernel=poly;, score=0.723 total time=   3.7s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=4, svc__kernel=poly;, score=0.716 total time=   3.9s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=4, svc__kernel=poly;, score=0.723 total time=   5.6s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=0.01, svc__degree=12, svc__kernel=poly;, score=0.616 total time=   7.4s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=0.01, svc__degree=12, svc__kernel=poly;, score=0.610 total time=   6.1s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=0.01, svc__degree=12, svc__kernel=poly;, score=0.618 total time=   6.8s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=100, svc__degree=9, svc__kernel=poly;, score=0.653 total time=   6.5s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=100, svc__degree=9, svc__kernel=poly;, score=0.644 total time=   5.5s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=100, svc__degree=9, svc__kernel=poly;, score=0.664 total time=   5.5s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=0.1, svc__degree=11, svc__kernel=poly;, score=0.622 total time=   5.0s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=0.1, svc__degree=11, svc__kernel=poly;, score=0.614 total time=   3.6s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=0.1, svc__degree=11, svc__kernel=poly;, score=0.626 total time=   3.5s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomizedSearchCV(cv=3, error_score='raise',\n",
              "                   estimator=Pipeline(steps=[('preprocess_data',\n",
              "                                              Pipeline(steps=[('fillna',\n",
              "                                                               pandas_fill_na(fill_type='mean')),\n",
              "                                                              ('onehotencode',\n",
              "                                                               pandas_get_dummies(nominal_columns=['wave',\n",
              "                                                                                                   'position',\n",
              "                                                                                                   'positin1',\n",
              "                                                                                                   'order',\n",
              "                                                                                                   'race_o',\n",
              "                                                                                                   'field_cd',\n",
              "                                                                                                   'race',\n",
              "                                                                                                   'goal',\n",
              "                                                                                                   'date',\n",
              "                                                                                                   'go_out',\n",
              "                                                                                                   'career_c',\n",
              "                                                                                                   'length',\n",
              "                                                                                                   'numdat_2',\n",
              "                                                                                                   'round'])),\n",
              "                                                              ('scaler',\n",
              "                                                               StandardScaler())])),\n",
              "                                             ('svc', SVC())]),\n",
              "                   n_iter=6,\n",
              "                   param_distributions=[{'preprocess_data__fillna__fill_type': ['median'],\n",
              "                                         'svc__C': [0.01, 0.1, 1, 10, 100, 1000,\n",
              "                                                    100000.0],\n",
              "                                         'svc__degree': array([ 4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]),\n",
              "                                         'svc__kernel': ['poly']}],\n",
              "                   scoring='roc_auc', verbose=3)"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(rnds.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 614
        },
        "id": "kWpGO4MH3Iic",
        "outputId": "cf8b4019-03b8-45f2-ee05-18dc7c1520b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-28aad738-1988-481c-b159-82a01e7f188e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_svc__kernel</th>\n",
              "      <th>param_svc__degree</th>\n",
              "      <th>param_svc__C</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3.450940</td>\n",
              "      <td>0.736348</td>\n",
              "      <td>1.074968</td>\n",
              "      <td>0.014984</td>\n",
              "      <td>poly</td>\n",
              "      <td>13</td>\n",
              "      <td>1000</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 13, 'sv...</td>\n",
              "      <td>0.640158</td>\n",
              "      <td>0.627023</td>\n",
              "      <td>0.648391</td>\n",
              "      <td>0.638524</td>\n",
              "      <td>0.008800</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.650423</td>\n",
              "      <td>0.036462</td>\n",
              "      <td>1.033792</td>\n",
              "      <td>0.029334</td>\n",
              "      <td>poly</td>\n",
              "      <td>5</td>\n",
              "      <td>1000</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 5, 'svc...</td>\n",
              "      <td>0.707975</td>\n",
              "      <td>0.696835</td>\n",
              "      <td>0.713635</td>\n",
              "      <td>0.706148</td>\n",
              "      <td>0.006979</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3.098666</td>\n",
              "      <td>0.620441</td>\n",
              "      <td>1.277885</td>\n",
              "      <td>0.239910</td>\n",
              "      <td>poly</td>\n",
              "      <td>4</td>\n",
              "      <td>100000.0</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 4, 'svc...</td>\n",
              "      <td>0.722965</td>\n",
              "      <td>0.715944</td>\n",
              "      <td>0.722789</td>\n",
              "      <td>0.720566</td>\n",
              "      <td>0.003269</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4.671092</td>\n",
              "      <td>0.513849</td>\n",
              "      <td>2.101970</td>\n",
              "      <td>0.471168</td>\n",
              "      <td>poly</td>\n",
              "      <td>12</td>\n",
              "      <td>0.01</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 12, 'sv...</td>\n",
              "      <td>0.615984</td>\n",
              "      <td>0.609885</td>\n",
              "      <td>0.618181</td>\n",
              "      <td>0.614684</td>\n",
              "      <td>0.003509</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.254656</td>\n",
              "      <td>0.436263</td>\n",
              "      <td>1.556841</td>\n",
              "      <td>0.053702</td>\n",
              "      <td>poly</td>\n",
              "      <td>9</td>\n",
              "      <td>100</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 9, 'svc...</td>\n",
              "      <td>0.652627</td>\n",
              "      <td>0.644220</td>\n",
              "      <td>0.664217</td>\n",
              "      <td>0.653688</td>\n",
              "      <td>0.008198</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>2.898624</td>\n",
              "      <td>0.578740</td>\n",
              "      <td>1.131533</td>\n",
              "      <td>0.082094</td>\n",
              "      <td>poly</td>\n",
              "      <td>11</td>\n",
              "      <td>0.1</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 11, 'sv...</td>\n",
              "      <td>0.622196</td>\n",
              "      <td>0.614167</td>\n",
              "      <td>0.626114</td>\n",
              "      <td>0.620826</td>\n",
              "      <td>0.004973</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-28aad738-1988-481c-b159-82a01e7f188e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-28aad738-1988-481c-b159-82a01e7f188e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-28aad738-1988-481c-b159-82a01e7f188e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0       3.450940      0.736348         1.074968        0.014984   \n",
              "1       2.650423      0.036462         1.033792        0.029334   \n",
              "2       3.098666      0.620441         1.277885        0.239910   \n",
              "3       4.671092      0.513849         2.101970        0.471168   \n",
              "4       4.254656      0.436263         1.556841        0.053702   \n",
              "5       2.898624      0.578740         1.131533        0.082094   \n",
              "\n",
              "  param_svc__kernel param_svc__degree param_svc__C  \\\n",
              "0              poly                13         1000   \n",
              "1              poly                 5         1000   \n",
              "2              poly                 4     100000.0   \n",
              "3              poly                12         0.01   \n",
              "4              poly                 9          100   \n",
              "5              poly                11          0.1   \n",
              "\n",
              "  param_preprocess_data__fillna__fill_type  \\\n",
              "0                                   median   \n",
              "1                                   median   \n",
              "2                                   median   \n",
              "3                                   median   \n",
              "4                                   median   \n",
              "5                                   median   \n",
              "\n",
              "                                              params  split0_test_score  \\\n",
              "0  {'svc__kernel': 'poly', 'svc__degree': 13, 'sv...           0.640158   \n",
              "1  {'svc__kernel': 'poly', 'svc__degree': 5, 'svc...           0.707975   \n",
              "2  {'svc__kernel': 'poly', 'svc__degree': 4, 'svc...           0.722965   \n",
              "3  {'svc__kernel': 'poly', 'svc__degree': 12, 'sv...           0.615984   \n",
              "4  {'svc__kernel': 'poly', 'svc__degree': 9, 'svc...           0.652627   \n",
              "5  {'svc__kernel': 'poly', 'svc__degree': 11, 'sv...           0.622196   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.627023           0.648391         0.638524        0.008800   \n",
              "1           0.696835           0.713635         0.706148        0.006979   \n",
              "2           0.715944           0.722789         0.720566        0.003269   \n",
              "3           0.609885           0.618181         0.614684        0.003509   \n",
              "4           0.644220           0.664217         0.653688        0.008198   \n",
              "5           0.614167           0.626114         0.620826        0.004973   \n",
              "\n",
              "   rank_test_score  \n",
              "0                4  \n",
              "1                2  \n",
              "2                1  \n",
              "3                6  \n",
              "4                3  \n",
              "5                5  "
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rnds.best_params_ , rnds.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eC-i_Rr_BWYr",
        "outputId": "69303dd3-cfca-4a02-8319-f2d8b2934179"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'preprocess_data__fillna__fill_type': 'median',\n",
              "  'svc__C': 100000.0,\n",
              "  'svc__degree': 4,\n",
              "  'svc__kernel': 'poly'},\n",
              " 0.7205661378031997)"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "the best score we got was \n",
        "({'preprocess_data__fillna__fill_type': 'median',\n",
        "  'svc__C': 100,\n",
        "  'svc__gamma': 'scale'},\n",
        " 0.8558126424898963)\n",
        "\n",
        " now we try another model"
      ],
      "metadata": {
        "id": "BytKT2XWKNoj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Trail 2\n",
        "usign decisiong Decision Tree\n",
        "* using grid search to know which fill type to be used and with criterion (entropy or gini)"
      ],
      "metadata": {
        "id": "zzs6mzM8_u3q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail2_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"dt\",DecisionTreeClassifier())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "AWziymKnASwg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#DecisionTreeClassifier()\n",
        "dtreeparam_grid = {\n",
        "    'preprocess_data__fillna__fill_type': ['mean','median'],\n",
        "    'dt__criterion': ['gini','entropy']\n",
        "}"
      ],
      "metadata": {
        "id": "ZVuYXqdPAIHP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search = GridSearchCV(\n",
        "    trail2_pipeline, dtreeparam_grid, cv=3, verbose=1,refit=True)"
      ],
      "metadata": {
        "id": "UncLzB8TA6kR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.fit(x, y);"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QLVeWSc_BEtw",
        "outputId": "67bb87b9-f9ce-4fec-8a49-84a2ba395fd5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(grid_search.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "id": "v79HOHTyBMC8",
        "outputId": "0b72e03d-1aaf-4127-80c9-161181337077"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-035e7602-d540-448d-8bf5-dfb407e25f42\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_dt__criterion</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.382038</td>\n",
              "      <td>0.010594</td>\n",
              "      <td>0.062745</td>\n",
              "      <td>0.001044</td>\n",
              "      <td>gini</td>\n",
              "      <td>mean</td>\n",
              "      <td>{'dt__criterion': 'gini', 'preprocess_data__fi...</td>\n",
              "      <td>0.811168</td>\n",
              "      <td>0.791878</td>\n",
              "      <td>0.800406</td>\n",
              "      <td>0.801151</td>\n",
              "      <td>0.007892</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.378911</td>\n",
              "      <td>0.021955</td>\n",
              "      <td>0.067065</td>\n",
              "      <td>0.005257</td>\n",
              "      <td>gini</td>\n",
              "      <td>median</td>\n",
              "      <td>{'dt__criterion': 'gini', 'preprocess_data__fi...</td>\n",
              "      <td>0.805584</td>\n",
              "      <td>0.803046</td>\n",
              "      <td>0.800914</td>\n",
              "      <td>0.803181</td>\n",
              "      <td>0.001909</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.318529</td>\n",
              "      <td>0.006158</td>\n",
              "      <td>0.081651</td>\n",
              "      <td>0.029773</td>\n",
              "      <td>entropy</td>\n",
              "      <td>mean</td>\n",
              "      <td>{'dt__criterion': 'entropy', 'preprocess_data_...</td>\n",
              "      <td>0.816244</td>\n",
              "      <td>0.799492</td>\n",
              "      <td>0.798375</td>\n",
              "      <td>0.804704</td>\n",
              "      <td>0.008173</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.315381</td>\n",
              "      <td>0.006850</td>\n",
              "      <td>0.065066</td>\n",
              "      <td>0.002914</td>\n",
              "      <td>entropy</td>\n",
              "      <td>median</td>\n",
              "      <td>{'dt__criterion': 'entropy', 'preprocess_data_...</td>\n",
              "      <td>0.806091</td>\n",
              "      <td>0.793909</td>\n",
              "      <td>0.805993</td>\n",
              "      <td>0.801998</td>\n",
              "      <td>0.005720</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-035e7602-d540-448d-8bf5-dfb407e25f42')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-035e7602-d540-448d-8bf5-dfb407e25f42 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-035e7602-d540-448d-8bf5-dfb407e25f42');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0       0.382038      0.010594         0.062745        0.001044   \n",
              "1       0.378911      0.021955         0.067065        0.005257   \n",
              "2       0.318529      0.006158         0.081651        0.029773   \n",
              "3       0.315381      0.006850         0.065066        0.002914   \n",
              "\n",
              "  param_dt__criterion param_preprocess_data__fillna__fill_type  \\\n",
              "0                gini                                     mean   \n",
              "1                gini                                   median   \n",
              "2             entropy                                     mean   \n",
              "3             entropy                                   median   \n",
              "\n",
              "                                              params  split0_test_score  \\\n",
              "0  {'dt__criterion': 'gini', 'preprocess_data__fi...           0.811168   \n",
              "1  {'dt__criterion': 'gini', 'preprocess_data__fi...           0.805584   \n",
              "2  {'dt__criterion': 'entropy', 'preprocess_data_...           0.816244   \n",
              "3  {'dt__criterion': 'entropy', 'preprocess_data_...           0.806091   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.791878           0.800406         0.801151        0.007892   \n",
              "1           0.803046           0.800914         0.803181        0.001909   \n",
              "2           0.799492           0.798375         0.804704        0.008173   \n",
              "3           0.793909           0.805993         0.801998        0.005720   \n",
              "\n",
              "   rank_test_score  \n",
              "0                4  \n",
              "1                2  \n",
              "2                1  \n",
              "3                3  "
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Trial 3\n",
        "continue with decisiong Decision Tree \n",
        "optimize other hyperparameters\n",
        "* it looks like using mean and entroby gave us the highest score\n",
        "* use them and tune other hyperparameters"
      ],
      "metadata": {
        "id": "MG8KWotUBgqv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail3_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"dt\",DecisionTreeClassifier())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "B5yAEDkXBtQt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#DecisionTreeClassifier()\n",
        "dtreeparam_grid = {\n",
        "    'preprocess_data__fillna__fill_type': ['mean'],\n",
        "    'dt__criterion': ['entropy'],\n",
        "    'dt__max_depth': np.arange(5,40),\n",
        "    'min_samples_leaf': np.arange(2,10),\n",
        "    'dt__max_leaf_nodes': np.arange(20,80)\n",
        "}"
      ],
      "metadata": {
        "id": "gcl_nSwIBzQq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rnds = RandomizedSearchCV(trail1_pipeline, param_list , cv=3, n_iter=10, verbose=3,scoring='roc_auc',error_score='raise')"
      ],
      "metadata": {
        "id": "VspSiKpmCQLn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rnds.fit(x,y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WuuwCkRzCaN4",
        "outputId": "a05fb1f5-a6d4-46dd-aa2e-33eda8fc2f7d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=13, svc__kernel=poly;, score=0.644 total time=   5.1s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=13, svc__kernel=poly;, score=0.626 total time=   6.0s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=13, svc__kernel=poly;, score=0.655 total time=   6.4s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=11, svc__kernel=poly;, score=0.650 total time=   4.2s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=11, svc__kernel=poly;, score=0.634 total time=   4.0s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=11, svc__kernel=poly;, score=0.662 total time=   6.0s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=1, svc__degree=14, svc__kernel=poly;, score=0.619 total time=   4.2s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=1, svc__degree=14, svc__kernel=poly;, score=0.613 total time=   5.2s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=1, svc__degree=14, svc__kernel=poly;, score=0.624 total time=   5.0s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=10, svc__degree=9, svc__kernel=poly;, score=0.643 total time=   5.8s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=10, svc__degree=9, svc__kernel=poly;, score=0.637 total time=   4.6s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=10, svc__degree=9, svc__kernel=poly;, score=0.656 total time=   5.1s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=0.01, svc__degree=12, svc__kernel=poly;, score=0.616 total time=   5.0s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=0.01, svc__degree=12, svc__kernel=poly;, score=0.610 total time=   4.2s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=0.01, svc__degree=12, svc__kernel=poly;, score=0.618 total time=   4.7s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=100, svc__degree=12, svc__kernel=poly;, score=0.639 total time=   5.5s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=100, svc__degree=12, svc__kernel=poly;, score=0.630 total time=   4.2s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=100, svc__degree=12, svc__kernel=poly;, score=0.647 total time=   4.5s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=10, svc__degree=5, svc__kernel=poly;, score=0.705 total time=   6.1s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=10, svc__degree=5, svc__kernel=poly;, score=0.695 total time=   6.1s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=10, svc__degree=5, svc__kernel=poly;, score=0.704 total time=   5.9s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=12, svc__kernel=poly;, score=0.646 total time=   4.5s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=12, svc__kernel=poly;, score=0.630 total time=   4.0s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=100000.0, svc__degree=12, svc__kernel=poly;, score=0.658 total time=   3.9s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=0.1, svc__degree=16, svc__kernel=poly;, score=0.612 total time=   5.1s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=0.1, svc__degree=16, svc__kernel=poly;, score=0.606 total time=   5.7s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=0.1, svc__degree=16, svc__kernel=poly;, score=0.615 total time=   3.8s\n",
            "[CV 1/3] END preprocess_data__fillna__fill_type=median, svc__C=1, svc__degree=7, svc__kernel=poly;, score=0.646 total time=   3.8s\n",
            "[CV 2/3] END preprocess_data__fillna__fill_type=median, svc__C=1, svc__degree=7, svc__kernel=poly;, score=0.641 total time=   5.5s\n",
            "[CV 3/3] END preprocess_data__fillna__fill_type=median, svc__C=1, svc__degree=7, svc__kernel=poly;, score=0.653 total time=   4.2s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomizedSearchCV(cv=3, error_score='raise',\n",
              "                   estimator=Pipeline(steps=[('preprocess_data',\n",
              "                                              Pipeline(steps=[('fillna',\n",
              "                                                               pandas_fill_na(fill_type='mean')),\n",
              "                                                              ('onehotencode',\n",
              "                                                               pandas_get_dummies(nominal_columns=['wave',\n",
              "                                                                                                   'position',\n",
              "                                                                                                   'positin1',\n",
              "                                                                                                   'order',\n",
              "                                                                                                   'race_o',\n",
              "                                                                                                   'field_cd',\n",
              "                                                                                                   'race',\n",
              "                                                                                                   'goal',\n",
              "                                                                                                   'date',\n",
              "                                                                                                   'go_out',\n",
              "                                                                                                   'career_c',\n",
              "                                                                                                   'length',\n",
              "                                                                                                   'numdat_2',\n",
              "                                                                                                   'round'])),\n",
              "                                                              ('scaler',\n",
              "                                                               StandardScaler())])),\n",
              "                                             ('svc', SVC())]),\n",
              "                   param_distributions=[{'preprocess_data__fillna__fill_type': ['median'],\n",
              "                                         'svc__C': [0.01, 0.1, 1, 10, 100, 1000,\n",
              "                                                    100000.0],\n",
              "                                         'svc__degree': array([ 4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]),\n",
              "                                         'svc__kernel': ['poly']}],\n",
              "                   scoring='roc_auc', verbose=3)"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(rnds.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 947
        },
        "id": "zB2C_2d5Cc62",
        "outputId": "ff77dcef-04a1-485e-c943-65a2f49e5a44"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-2f683ba0-c1bf-495c-a2ba-27917a5775a8\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_svc__kernel</th>\n",
              "      <th>param_svc__degree</th>\n",
              "      <th>param_svc__C</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4.378679</td>\n",
              "      <td>0.406136</td>\n",
              "      <td>1.459933</td>\n",
              "      <td>0.301288</td>\n",
              "      <td>poly</td>\n",
              "      <td>13</td>\n",
              "      <td>100000.0</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 13, 'sv...</td>\n",
              "      <td>0.643577</td>\n",
              "      <td>0.626234</td>\n",
              "      <td>0.654691</td>\n",
              "      <td>0.641501</td>\n",
              "      <td>0.011710</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3.488199</td>\n",
              "      <td>0.668735</td>\n",
              "      <td>1.232126</td>\n",
              "      <td>0.215795</td>\n",
              "      <td>poly</td>\n",
              "      <td>11</td>\n",
              "      <td>100000.0</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 11, 'sv...</td>\n",
              "      <td>0.649643</td>\n",
              "      <td>0.633670</td>\n",
              "      <td>0.662008</td>\n",
              "      <td>0.648440</td>\n",
              "      <td>0.011600</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3.391711</td>\n",
              "      <td>0.270149</td>\n",
              "      <td>1.406418</td>\n",
              "      <td>0.186173</td>\n",
              "      <td>poly</td>\n",
              "      <td>14</td>\n",
              "      <td>1</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 14, 'sv...</td>\n",
              "      <td>0.618744</td>\n",
              "      <td>0.613017</td>\n",
              "      <td>0.624409</td>\n",
              "      <td>0.618723</td>\n",
              "      <td>0.004651</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.717693</td>\n",
              "      <td>0.456404</td>\n",
              "      <td>1.450402</td>\n",
              "      <td>0.074697</td>\n",
              "      <td>poly</td>\n",
              "      <td>9</td>\n",
              "      <td>10</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 9, 'svc...</td>\n",
              "      <td>0.642934</td>\n",
              "      <td>0.636672</td>\n",
              "      <td>0.655510</td>\n",
              "      <td>0.645039</td>\n",
              "      <td>0.007833</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3.374257</td>\n",
              "      <td>0.272576</td>\n",
              "      <td>1.265279</td>\n",
              "      <td>0.097001</td>\n",
              "      <td>poly</td>\n",
              "      <td>12</td>\n",
              "      <td>0.01</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 12, 'sv...</td>\n",
              "      <td>0.615984</td>\n",
              "      <td>0.609885</td>\n",
              "      <td>0.618181</td>\n",
              "      <td>0.614684</td>\n",
              "      <td>0.003509</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>3.359650</td>\n",
              "      <td>0.469154</td>\n",
              "      <td>1.370684</td>\n",
              "      <td>0.174289</td>\n",
              "      <td>poly</td>\n",
              "      <td>12</td>\n",
              "      <td>100</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 12, 'sv...</td>\n",
              "      <td>0.639191</td>\n",
              "      <td>0.629599</td>\n",
              "      <td>0.646755</td>\n",
              "      <td>0.638515</td>\n",
              "      <td>0.007020</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>4.388936</td>\n",
              "      <td>0.048085</td>\n",
              "      <td>1.648346</td>\n",
              "      <td>0.067124</td>\n",
              "      <td>poly</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 5, 'svc...</td>\n",
              "      <td>0.704847</td>\n",
              "      <td>0.695061</td>\n",
              "      <td>0.704361</td>\n",
              "      <td>0.701423</td>\n",
              "      <td>0.004503</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>3.059193</td>\n",
              "      <td>0.268037</td>\n",
              "      <td>1.074131</td>\n",
              "      <td>0.013537</td>\n",
              "      <td>poly</td>\n",
              "      <td>12</td>\n",
              "      <td>100000.0</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 12, 'sv...</td>\n",
              "      <td>0.646114</td>\n",
              "      <td>0.629623</td>\n",
              "      <td>0.657680</td>\n",
              "      <td>0.644473</td>\n",
              "      <td>0.011513</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>3.515474</td>\n",
              "      <td>0.591054</td>\n",
              "      <td>1.363143</td>\n",
              "      <td>0.202673</td>\n",
              "      <td>poly</td>\n",
              "      <td>16</td>\n",
              "      <td>0.1</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 16, 'sv...</td>\n",
              "      <td>0.611896</td>\n",
              "      <td>0.605706</td>\n",
              "      <td>0.615057</td>\n",
              "      <td>0.610886</td>\n",
              "      <td>0.003884</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>3.157745</td>\n",
              "      <td>0.398605</td>\n",
              "      <td>1.343314</td>\n",
              "      <td>0.370758</td>\n",
              "      <td>poly</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "      <td>median</td>\n",
              "      <td>{'svc__kernel': 'poly', 'svc__degree': 7, 'svc...</td>\n",
              "      <td>0.645974</td>\n",
              "      <td>0.641480</td>\n",
              "      <td>0.652522</td>\n",
              "      <td>0.646659</td>\n",
              "      <td>0.004534</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2f683ba0-c1bf-495c-a2ba-27917a5775a8')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2f683ba0-c1bf-495c-a2ba-27917a5775a8 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2f683ba0-c1bf-495c-a2ba-27917a5775a8');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0       4.378679      0.406136         1.459933        0.301288   \n",
              "1       3.488199      0.668735         1.232126        0.215795   \n",
              "2       3.391711      0.270149         1.406418        0.186173   \n",
              "3       3.717693      0.456404         1.450402        0.074697   \n",
              "4       3.374257      0.272576         1.265279        0.097001   \n",
              "5       3.359650      0.469154         1.370684        0.174289   \n",
              "6       4.388936      0.048085         1.648346        0.067124   \n",
              "7       3.059193      0.268037         1.074131        0.013537   \n",
              "8       3.515474      0.591054         1.363143        0.202673   \n",
              "9       3.157745      0.398605         1.343314        0.370758   \n",
              "\n",
              "  param_svc__kernel param_svc__degree param_svc__C  \\\n",
              "0              poly                13     100000.0   \n",
              "1              poly                11     100000.0   \n",
              "2              poly                14            1   \n",
              "3              poly                 9           10   \n",
              "4              poly                12         0.01   \n",
              "5              poly                12          100   \n",
              "6              poly                 5           10   \n",
              "7              poly                12     100000.0   \n",
              "8              poly                16          0.1   \n",
              "9              poly                 7            1   \n",
              "\n",
              "  param_preprocess_data__fillna__fill_type  \\\n",
              "0                                   median   \n",
              "1                                   median   \n",
              "2                                   median   \n",
              "3                                   median   \n",
              "4                                   median   \n",
              "5                                   median   \n",
              "6                                   median   \n",
              "7                                   median   \n",
              "8                                   median   \n",
              "9                                   median   \n",
              "\n",
              "                                              params  split0_test_score  \\\n",
              "0  {'svc__kernel': 'poly', 'svc__degree': 13, 'sv...           0.643577   \n",
              "1  {'svc__kernel': 'poly', 'svc__degree': 11, 'sv...           0.649643   \n",
              "2  {'svc__kernel': 'poly', 'svc__degree': 14, 'sv...           0.618744   \n",
              "3  {'svc__kernel': 'poly', 'svc__degree': 9, 'svc...           0.642934   \n",
              "4  {'svc__kernel': 'poly', 'svc__degree': 12, 'sv...           0.615984   \n",
              "5  {'svc__kernel': 'poly', 'svc__degree': 12, 'sv...           0.639191   \n",
              "6  {'svc__kernel': 'poly', 'svc__degree': 5, 'svc...           0.704847   \n",
              "7  {'svc__kernel': 'poly', 'svc__degree': 12, 'sv...           0.646114   \n",
              "8  {'svc__kernel': 'poly', 'svc__degree': 16, 'sv...           0.611896   \n",
              "9  {'svc__kernel': 'poly', 'svc__degree': 7, 'svc...           0.645974   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.626234           0.654691         0.641501        0.011710   \n",
              "1           0.633670           0.662008         0.648440        0.011600   \n",
              "2           0.613017           0.624409         0.618723        0.004651   \n",
              "3           0.636672           0.655510         0.645039        0.007833   \n",
              "4           0.609885           0.618181         0.614684        0.003509   \n",
              "5           0.629599           0.646755         0.638515        0.007020   \n",
              "6           0.695061           0.704361         0.701423        0.004503   \n",
              "7           0.629623           0.657680         0.644473        0.011513   \n",
              "8           0.605706           0.615057         0.610886        0.003884   \n",
              "9           0.641480           0.652522         0.646659        0.004534   \n",
              "\n",
              "   rank_test_score  \n",
              "0                6  \n",
              "1                2  \n",
              "2                8  \n",
              "3                4  \n",
              "4                9  \n",
              "5                7  \n",
              "6                1  \n",
              "7                5  \n",
              "8               10  \n",
              "9                3  "
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rnds.best_params_ , rnds.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BO0T7973Cee5",
        "outputId": "9dd6c764-b644-4339-d7f7-ebcf601668a2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'preprocess_data__fillna__fill_type': 'median',\n",
              "  'svc__C': 10,\n",
              "  'svc__degree': 5,\n",
              "  'svc__kernel': 'poly'},\n",
              " 0.7014228911356922)"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "the best score we got is 0.7014228911356922\n",
        "\n",
        "svc still has better performance"
      ],
      "metadata": {
        "id": "ZndLEN06Kln_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Trail 4\n",
        "using XGBOOT\n",
        "* start with random search to find a start point\n",
        "* then in the upcomming 2 trials we will use random and baysian search to find the best parameters"
      ],
      "metadata": {
        "id": "BxPdYnZ9CiSL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail4_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"GBC\", GradientBoostingClassifier())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "tiPTbme4EYWL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# GradientBoostingClassifier()\n",
        "GBC = {\n",
        "    'preprocess_data__fillna__fill_type': ['mean','median'],\n",
        "    'GBC__n_estimators' : np.arange(70,800),\n",
        "    'GBC__learning_rate' : [0.1,0.1,2],\n",
        "    'GBC__loss': ['deviance','exponential'],\n",
        "    'GBC__criterion' : ['friedman_mse','squared_error'],\n",
        "    'GBC__min_samples_leaf': np.arange(2,10)\n",
        "}"
      ],
      "metadata": {
        "id": "VrNk65_BCll6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rnds = RandomizedSearchCV(trail4_pipeline, GBC , cv=3, n_iter=5, verbose=3,scoring='roc_auc',error_score='raise')"
      ],
      "metadata": {
        "id": "ULlthJlRDdL3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rnds.fit(x,y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "40Lbzp1dFLSS",
        "outputId": "3ed00e57-90dd-48c6-aedd-d819e302f470"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n",
            "[CV 1/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=deviance, GBC__min_samples_leaf=2, GBC__n_estimators=567, preprocess_data__fillna__fill_type=median;, score=0.886 total time=  28.8s\n",
            "[CV 2/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=deviance, GBC__min_samples_leaf=2, GBC__n_estimators=567, preprocess_data__fillna__fill_type=median;, score=0.869 total time=  27.3s\n",
            "[CV 3/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=deviance, GBC__min_samples_leaf=2, GBC__n_estimators=567, preprocess_data__fillna__fill_type=median;, score=0.866 total time=  27.4s\n",
            "[CV 1/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=exponential, GBC__min_samples_leaf=3, GBC__n_estimators=240, preprocess_data__fillna__fill_type=mean;, score=0.891 total time=  12.4s\n",
            "[CV 2/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=exponential, GBC__min_samples_leaf=3, GBC__n_estimators=240, preprocess_data__fillna__fill_type=mean;, score=0.880 total time=  13.2s\n",
            "[CV 3/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=exponential, GBC__min_samples_leaf=3, GBC__n_estimators=240, preprocess_data__fillna__fill_type=mean;, score=0.874 total time=  15.4s\n",
            "[CV 1/3] END GBC__criterion=squared_error, GBC__learning_rate=2, GBC__loss=exponential, GBC__min_samples_leaf=8, GBC__n_estimators=80, preprocess_data__fillna__fill_type=median;, score=0.810 total time=   4.0s\n",
            "[CV 2/3] END GBC__criterion=squared_error, GBC__learning_rate=2, GBC__loss=exponential, GBC__min_samples_leaf=8, GBC__n_estimators=80, preprocess_data__fillna__fill_type=median;, score=0.818 total time=   4.1s\n",
            "[CV 3/3] END GBC__criterion=squared_error, GBC__learning_rate=2, GBC__loss=exponential, GBC__min_samples_leaf=8, GBC__n_estimators=80, preprocess_data__fillna__fill_type=median;, score=0.806 total time=   4.0s\n",
            "[CV 1/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=exponential, GBC__min_samples_leaf=5, GBC__n_estimators=98, preprocess_data__fillna__fill_type=median;, score=0.887 total time=   4.9s\n",
            "[CV 2/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=exponential, GBC__min_samples_leaf=5, GBC__n_estimators=98, preprocess_data__fillna__fill_type=median;, score=0.878 total time=   4.9s\n",
            "[CV 3/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=exponential, GBC__min_samples_leaf=5, GBC__n_estimators=98, preprocess_data__fillna__fill_type=median;, score=0.866 total time=   4.9s\n",
            "[CV 1/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=deviance, GBC__min_samples_leaf=5, GBC__n_estimators=447, preprocess_data__fillna__fill_type=mean;, score=0.889 total time=  22.8s\n",
            "[CV 2/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=deviance, GBC__min_samples_leaf=5, GBC__n_estimators=447, preprocess_data__fillna__fill_type=mean;, score=0.874 total time=  23.0s\n",
            "[CV 3/3] END GBC__criterion=friedman_mse, GBC__learning_rate=0.1, GBC__loss=deviance, GBC__min_samples_leaf=5, GBC__n_estimators=447, preprocess_data__fillna__fill_type=mean;, score=0.872 total time=  22.9s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomizedSearchCV(cv=3, error_score='raise',\n",
              "                   estimator=Pipeline(steps=[('preprocess_data',\n",
              "                                              Pipeline(steps=[('fillna',\n",
              "                                                               pandas_fill_na(fill_type='mean')),\n",
              "                                                              ('onehotencode',\n",
              "                                                               pandas_get_dummies(nominal_columns=['wave',\n",
              "                                                                                                   'position',\n",
              "                                                                                                   'positin1',\n",
              "                                                                                                   'order',\n",
              "                                                                                                   'race_o',\n",
              "                                                                                                   'field_cd',\n",
              "                                                                                                   'race',\n",
              "                                                                                                   'goal',\n",
              "                                                                                                   'date',\n",
              "                                                                                                   'go_out',\n",
              "                                                                                                   'career_c',\n",
              "                                                                                                   'length',\n",
              "                                                                                                   'numdat_2',\n",
              "                                                                                                   'round'])),\n",
              "                                                              ('scaler',\n",
              "                                                               StandardSc...\n",
              "       733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745,\n",
              "       746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758,\n",
              "       759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771,\n",
              "       772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784,\n",
              "       785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797,\n",
              "       798, 799]),\n",
              "                                        'preprocess_data__fillna__fill_type': ['mean',\n",
              "                                                                               'median']},\n",
              "                   scoring='roc_auc', verbose=3)"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(rnds.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "id": "ZnCkU7t-EvpC",
        "outputId": "c685045f-c2ab-430b-a2f8-aa5b79cd1bab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-17b84482-f423-4761-95ce-a7af113ca5a4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>param_GBC__n_estimators</th>\n",
              "      <th>param_GBC__min_samples_leaf</th>\n",
              "      <th>param_GBC__loss</th>\n",
              "      <th>param_GBC__learning_rate</th>\n",
              "      <th>param_GBC__criterion</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>27.709937</td>\n",
              "      <td>0.681052</td>\n",
              "      <td>0.082093</td>\n",
              "      <td>0.001230</td>\n",
              "      <td>median</td>\n",
              "      <td>567</td>\n",
              "      <td>2</td>\n",
              "      <td>deviance</td>\n",
              "      <td>0.1</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.885910</td>\n",
              "      <td>0.868729</td>\n",
              "      <td>0.865731</td>\n",
              "      <td>0.873457</td>\n",
              "      <td>0.008890</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>13.585792</td>\n",
              "      <td>1.289262</td>\n",
              "      <td>0.093219</td>\n",
              "      <td>0.031881</td>\n",
              "      <td>mean</td>\n",
              "      <td>240</td>\n",
              "      <td>3</td>\n",
              "      <td>exponential</td>\n",
              "      <td>0.1</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'mean',...</td>\n",
              "      <td>0.891033</td>\n",
              "      <td>0.880401</td>\n",
              "      <td>0.874422</td>\n",
              "      <td>0.881952</td>\n",
              "      <td>0.006870</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3.942741</td>\n",
              "      <td>0.031436</td>\n",
              "      <td>0.067257</td>\n",
              "      <td>0.000625</td>\n",
              "      <td>median</td>\n",
              "      <td>80</td>\n",
              "      <td>8</td>\n",
              "      <td>exponential</td>\n",
              "      <td>2</td>\n",
              "      <td>squared_error</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.810342</td>\n",
              "      <td>0.817860</td>\n",
              "      <td>0.805944</td>\n",
              "      <td>0.811382</td>\n",
              "      <td>0.004920</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4.829808</td>\n",
              "      <td>0.032215</td>\n",
              "      <td>0.070943</td>\n",
              "      <td>0.002096</td>\n",
              "      <td>median</td>\n",
              "      <td>98</td>\n",
              "      <td>5</td>\n",
              "      <td>exponential</td>\n",
              "      <td>0.1</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'median...</td>\n",
              "      <td>0.887201</td>\n",
              "      <td>0.878367</td>\n",
              "      <td>0.865820</td>\n",
              "      <td>0.877129</td>\n",
              "      <td>0.008772</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>22.823807</td>\n",
              "      <td>0.078415</td>\n",
              "      <td>0.078395</td>\n",
              "      <td>0.001477</td>\n",
              "      <td>mean</td>\n",
              "      <td>447</td>\n",
              "      <td>5</td>\n",
              "      <td>deviance</td>\n",
              "      <td>0.1</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>{'preprocess_data__fillna__fill_type': 'mean',...</td>\n",
              "      <td>0.888781</td>\n",
              "      <td>0.873538</td>\n",
              "      <td>0.872403</td>\n",
              "      <td>0.878241</td>\n",
              "      <td>0.007467</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-17b84482-f423-4761-95ce-a7af113ca5a4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-17b84482-f423-4761-95ce-a7af113ca5a4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-17b84482-f423-4761-95ce-a7af113ca5a4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0      27.709937      0.681052         0.082093        0.001230   \n",
              "1      13.585792      1.289262         0.093219        0.031881   \n",
              "2       3.942741      0.031436         0.067257        0.000625   \n",
              "3       4.829808      0.032215         0.070943        0.002096   \n",
              "4      22.823807      0.078415         0.078395        0.001477   \n",
              "\n",
              "  param_preprocess_data__fillna__fill_type param_GBC__n_estimators  \\\n",
              "0                                   median                     567   \n",
              "1                                     mean                     240   \n",
              "2                                   median                      80   \n",
              "3                                   median                      98   \n",
              "4                                     mean                     447   \n",
              "\n",
              "  param_GBC__min_samples_leaf param_GBC__loss param_GBC__learning_rate  \\\n",
              "0                           2        deviance                      0.1   \n",
              "1                           3     exponential                      0.1   \n",
              "2                           8     exponential                        2   \n",
              "3                           5     exponential                      0.1   \n",
              "4                           5        deviance                      0.1   \n",
              "\n",
              "  param_GBC__criterion                                             params  \\\n",
              "0         friedman_mse  {'preprocess_data__fillna__fill_type': 'median...   \n",
              "1         friedman_mse  {'preprocess_data__fillna__fill_type': 'mean',...   \n",
              "2        squared_error  {'preprocess_data__fillna__fill_type': 'median...   \n",
              "3         friedman_mse  {'preprocess_data__fillna__fill_type': 'median...   \n",
              "4         friedman_mse  {'preprocess_data__fillna__fill_type': 'mean',...   \n",
              "\n",
              "   split0_test_score  split1_test_score  split2_test_score  mean_test_score  \\\n",
              "0           0.885910           0.868729           0.865731         0.873457   \n",
              "1           0.891033           0.880401           0.874422         0.881952   \n",
              "2           0.810342           0.817860           0.805944         0.811382   \n",
              "3           0.887201           0.878367           0.865820         0.877129   \n",
              "4           0.888781           0.873538           0.872403         0.878241   \n",
              "\n",
              "   std_test_score  rank_test_score  \n",
              "0        0.008890                4  \n",
              "1        0.006870                1  \n",
              "2        0.004920                5  \n",
              "3        0.008772                3  \n",
              "4        0.007467                2  "
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rnds.best_params_ , rnds.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A_ct7v_zEwFK",
        "outputId": "22aa7b9d-41f8-4b62-d3fd-f7adfce74b5f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'GBC__criterion': 'friedman_mse',\n",
              "  'GBC__learning_rate': 0.1,\n",
              "  'GBC__loss': 'exponential',\n",
              "  'GBC__min_samples_leaf': 3,\n",
              "  'GBC__n_estimators': 240,\n",
              "  'preprocess_data__fillna__fill_type': 'mean'},\n",
              " 0.8819519562332582)"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "greate reults\n",
        "\n",
        "better than svc\n",
        "\n",
        "let's try baysian search"
      ],
      "metadata": {
        "id": "150_diVoK6Xc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Trail 5\n",
        "xgboot using baysian search"
      ],
      "metadata": {
        "id": "wQeWTrHPHWyv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail5_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"GBC\", GradientBoostingClassifier())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "Ki_sl-XCHtDs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# GradientBoostingClassifier()\n",
        "GBC = {\n",
        "    'preprocess_data__fillna__fill_type': Categorical(['mean','median']),\n",
        "    'GBC__n_estimators' : Integer(70,300),\n",
        "    'GBC__learning_rate' : [0.1],\n",
        "    'GBC__loss': Categorical(['deviance','exponential']),\n",
        "    'GBC__criterion' : Categorical(['friedman_mse','squared_error']),\n",
        "    'GBC__min_samples_leaf': Integer(2,10)\n",
        "}"
      ],
      "metadata": {
        "id": "N8SIWZfXHtAK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bys = BayesSearchCV(trail5_pipeline, GBC, cv=3,n_iter=4,verbose=3,scoring='roc_auc')"
      ],
      "metadata": {
        "id": "mgDaDHz8Hs-G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bys.fit(x,y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NcD3EvXUISW8",
        "outputId": "0a77315b-db26-47a5-9429-c6f39b826b72"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "[CV 1/3] END GBC__min_samples_leaf=4, GBC__n_estimators=672;, score=0.886 total time=  36.0s\n",
            "[CV 2/3] END GBC__min_samples_leaf=4, GBC__n_estimators=672;, score=0.867 total time=  43.1s\n",
            "[CV 3/3] END GBC__min_samples_leaf=4, GBC__n_estimators=672;, score=0.866 total time=  34.6s\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "[CV 1/3] END GBC__min_samples_leaf=4, GBC__n_estimators=419;, score=0.889 total time=  23.7s\n",
            "[CV 2/3] END GBC__min_samples_leaf=4, GBC__n_estimators=419;, score=0.871 total time=  28.9s\n",
            "[CV 3/3] END GBC__min_samples_leaf=4, GBC__n_estimators=419;, score=0.867 total time=  27.3s\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "[CV 1/3] END GBC__min_samples_leaf=3, GBC__n_estimators=636;, score=0.884 total time=  32.4s\n",
            "[CV 2/3] END GBC__min_samples_leaf=3, GBC__n_estimators=636;, score=0.870 total time=  32.8s\n",
            "[CV 3/3] END GBC__min_samples_leaf=3, GBC__n_estimators=636;, score=0.868 total time=  32.5s\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "[CV 1/3] END GBC__min_samples_leaf=2, GBC__n_estimators=224;, score=0.890 total time=  11.5s\n",
            "[CV 2/3] END GBC__min_samples_leaf=2, GBC__n_estimators=224;, score=0.876 total time=  15.2s\n",
            "[CV 3/3] END GBC__min_samples_leaf=2, GBC__n_estimators=224;, score=0.865 total time=  12.9s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BayesSearchCV(cv=3,\n",
              "              estimator=Pipeline(steps=[('preprocess_data',\n",
              "                                         Pipeline(steps=[('fillna',\n",
              "                                                          pandas_fill_na(fill_type='mean')),\n",
              "                                                         ('onehotencode',\n",
              "                                                          pandas_get_dummies(nominal_columns=['wave',\n",
              "                                                                                              'position',\n",
              "                                                                                              'positin1',\n",
              "                                                                                              'order',\n",
              "                                                                                              'race_o',\n",
              "                                                                                              'field_cd',\n",
              "                                                                                              'race',\n",
              "                                                                                              'goal',\n",
              "                                                                                              'date',\n",
              "                                                                                              'go_out',\n",
              "                                                                                              'career_c',\n",
              "                                                                                              'length',\n",
              "                                                                                              'numdat_2',\n",
              "                                                                                              'round'])),\n",
              "                                                         ('scaler',\n",
              "                                                          StandardScaler())])),\n",
              "                                        ('GBC', Gradien...\n",
              "       707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719,\n",
              "       720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732,\n",
              "       733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745,\n",
              "       746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758,\n",
              "       759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771,\n",
              "       772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784,\n",
              "       785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797,\n",
              "       798, 799])},\n",
              "              verbose=3)"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(bys.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 308
        },
        "id": "VHhsGec0IpKG",
        "outputId": "68402b1d-9824-4b62-9668-54c73303e9aa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-e185d89b-eaad-4fcd-8d90-b890f9d9c034\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_GBC__min_samples_leaf</th>\n",
              "      <th>param_GBC__n_estimators</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>37.782498</td>\n",
              "      <td>3.721068</td>\n",
              "      <td>0.109047</td>\n",
              "      <td>0.030078</td>\n",
              "      <td>4</td>\n",
              "      <td>672</td>\n",
              "      <td>{'GBC__min_samples_leaf': 4, 'GBC__n_estimator...</td>\n",
              "      <td>0.886115</td>\n",
              "      <td>0.867358</td>\n",
              "      <td>0.866432</td>\n",
              "      <td>0.873302</td>\n",
              "      <td>0.009069</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>26.541570</td>\n",
              "      <td>2.170389</td>\n",
              "      <td>0.099775</td>\n",
              "      <td>0.032449</td>\n",
              "      <td>4</td>\n",
              "      <td>419</td>\n",
              "      <td>{'GBC__min_samples_leaf': 4, 'GBC__n_estimator...</td>\n",
              "      <td>0.889372</td>\n",
              "      <td>0.870859</td>\n",
              "      <td>0.867316</td>\n",
              "      <td>0.875849</td>\n",
              "      <td>0.009671</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>32.459167</td>\n",
              "      <td>0.191587</td>\n",
              "      <td>0.082248</td>\n",
              "      <td>0.001314</td>\n",
              "      <td>3</td>\n",
              "      <td>636</td>\n",
              "      <td>{'GBC__min_samples_leaf': 3, 'GBC__n_estimator...</td>\n",
              "      <td>0.883952</td>\n",
              "      <td>0.869808</td>\n",
              "      <td>0.867976</td>\n",
              "      <td>0.873912</td>\n",
              "      <td>0.007139</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>13.149777</td>\n",
              "      <td>1.494736</td>\n",
              "      <td>0.092627</td>\n",
              "      <td>0.033291</td>\n",
              "      <td>2</td>\n",
              "      <td>224</td>\n",
              "      <td>{'GBC__min_samples_leaf': 2, 'GBC__n_estimator...</td>\n",
              "      <td>0.890031</td>\n",
              "      <td>0.875658</td>\n",
              "      <td>0.865033</td>\n",
              "      <td>0.876907</td>\n",
              "      <td>0.010244</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e185d89b-eaad-4fcd-8d90-b890f9d9c034')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e185d89b-eaad-4fcd-8d90-b890f9d9c034 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e185d89b-eaad-4fcd-8d90-b890f9d9c034');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0      37.782498      3.721068         0.109047        0.030078   \n",
              "1      26.541570      2.170389         0.099775        0.032449   \n",
              "2      32.459167      0.191587         0.082248        0.001314   \n",
              "3      13.149777      1.494736         0.092627        0.033291   \n",
              "\n",
              "  param_GBC__min_samples_leaf param_GBC__n_estimators  \\\n",
              "0                           4                     672   \n",
              "1                           4                     419   \n",
              "2                           3                     636   \n",
              "3                           2                     224   \n",
              "\n",
              "                                              params  split0_test_score  \\\n",
              "0  {'GBC__min_samples_leaf': 4, 'GBC__n_estimator...           0.886115   \n",
              "1  {'GBC__min_samples_leaf': 4, 'GBC__n_estimator...           0.889372   \n",
              "2  {'GBC__min_samples_leaf': 3, 'GBC__n_estimator...           0.883952   \n",
              "3  {'GBC__min_samples_leaf': 2, 'GBC__n_estimator...           0.890031   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.867358           0.866432         0.873302        0.009069   \n",
              "1           0.870859           0.867316         0.875849        0.009671   \n",
              "2           0.869808           0.867976         0.873912        0.007139   \n",
              "3           0.875658           0.865033         0.876907        0.010244   \n",
              "\n",
              "   rank_test_score  \n",
              "0                4  \n",
              "1                2  \n",
              "2                3  \n",
              "3                1  "
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "bys.best_params_ , bys.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_e9xMoawJMlz",
        "outputId": "21a470a9-dc79-4b8f-bf60-e0b7b31767cc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(OrderedDict([('GBC__min_samples_leaf', 2), ('GBC__n_estimators', 224)]),\n",
              " 0.8769071718811347)"
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "random search gave us better score"
      ],
      "metadata": {
        "id": "OFbXSkR9LaK2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#trail 6\n",
        "usign xgboot but with grid search (using ranges from the previous trail)"
      ],
      "metadata": {
        "id": "kBa1hu_9GdO6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "trail6_pipeline = Pipeline(\n",
        "    steps = [\n",
        "             (\"preprocess_data\",preprocess_data),\n",
        "             (\"GBC\", GradientBoostingClassifier())\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "kefuDqWyGca9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# GradientBoostingClassifier()\n",
        "GBC = {\n",
        "    'preprocess_data__fillna__fill_type': ['mean'],\n",
        "    'GBC__n_estimators' : [220,240,260],\n",
        "    'GBC__learning_rate' : [0.1],\n",
        "    'GBC__loss': ['exponential'],\n",
        "    'GBC__criterion' : ['friedman_mse'],\n",
        "    'GBC__min_samples_leaf': [3]\n",
        "}"
      ],
      "metadata": {
        "id": "RUJGJ81FGnnA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search = GridSearchCV(\n",
        "    trail6_pipeline, GBC, cv=3, verbose=1,refit=True,scoring='roc_auc',error_score='raise')"
      ],
      "metadata": {
        "id": "shIwTdpFGnjc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.fit(x, y);"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2ARV4pf0GnhW",
        "outputId": "ef54b433-85ab-4803-9358-54f77e1bcc87"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 3 candidates, totalling 9 fits\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(grid_search.cv_results_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 311
        },
        "id": "TIn1squ8HODW",
        "outputId": "b45c1468-710b-4bf2-8944-61c0172995a9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-84913509-6246-41cd-9ff8-3dcae9654640\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_fit_time</th>\n",
              "      <th>std_fit_time</th>\n",
              "      <th>mean_score_time</th>\n",
              "      <th>std_score_time</th>\n",
              "      <th>param_GBC__criterion</th>\n",
              "      <th>param_GBC__learning_rate</th>\n",
              "      <th>param_GBC__loss</th>\n",
              "      <th>param_GBC__min_samples_leaf</th>\n",
              "      <th>param_GBC__n_estimators</th>\n",
              "      <th>param_preprocess_data__fillna__fill_type</th>\n",
              "      <th>params</th>\n",
              "      <th>split0_test_score</th>\n",
              "      <th>split1_test_score</th>\n",
              "      <th>split2_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>std_test_score</th>\n",
              "      <th>rank_test_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>12.650253</td>\n",
              "      <td>1.724315</td>\n",
              "      <td>0.073036</td>\n",
              "      <td>0.004219</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>0.1</td>\n",
              "      <td>exponential</td>\n",
              "      <td>3</td>\n",
              "      <td>220</td>\n",
              "      <td>mean</td>\n",
              "      <td>{'GBC__criterion': 'friedman_mse', 'GBC__learn...</td>\n",
              "      <td>0.891222</td>\n",
              "      <td>0.880528</td>\n",
              "      <td>0.874405</td>\n",
              "      <td>0.882052</td>\n",
              "      <td>0.006950</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>12.949864</td>\n",
              "      <td>0.861234</td>\n",
              "      <td>0.087001</td>\n",
              "      <td>0.025439</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>0.1</td>\n",
              "      <td>exponential</td>\n",
              "      <td>3</td>\n",
              "      <td>240</td>\n",
              "      <td>mean</td>\n",
              "      <td>{'GBC__criterion': 'friedman_mse', 'GBC__learn...</td>\n",
              "      <td>0.891005</td>\n",
              "      <td>0.880327</td>\n",
              "      <td>0.874400</td>\n",
              "      <td>0.881911</td>\n",
              "      <td>0.006871</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>15.424506</td>\n",
              "      <td>1.389144</td>\n",
              "      <td>0.075675</td>\n",
              "      <td>0.004576</td>\n",
              "      <td>friedman_mse</td>\n",
              "      <td>0.1</td>\n",
              "      <td>exponential</td>\n",
              "      <td>3</td>\n",
              "      <td>260</td>\n",
              "      <td>mean</td>\n",
              "      <td>{'GBC__criterion': 'friedman_mse', 'GBC__learn...</td>\n",
              "      <td>0.890502</td>\n",
              "      <td>0.880140</td>\n",
              "      <td>0.873701</td>\n",
              "      <td>0.881448</td>\n",
              "      <td>0.006921</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-84913509-6246-41cd-9ff8-3dcae9654640')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-84913509-6246-41cd-9ff8-3dcae9654640 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-84913509-6246-41cd-9ff8-3dcae9654640');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
              "0      12.650253      1.724315         0.073036        0.004219   \n",
              "1      12.949864      0.861234         0.087001        0.025439   \n",
              "2      15.424506      1.389144         0.075675        0.004576   \n",
              "\n",
              "  param_GBC__criterion param_GBC__learning_rate param_GBC__loss  \\\n",
              "0         friedman_mse                      0.1     exponential   \n",
              "1         friedman_mse                      0.1     exponential   \n",
              "2         friedman_mse                      0.1     exponential   \n",
              "\n",
              "  param_GBC__min_samples_leaf param_GBC__n_estimators  \\\n",
              "0                           3                     220   \n",
              "1                           3                     240   \n",
              "2                           3                     260   \n",
              "\n",
              "  param_preprocess_data__fillna__fill_type  \\\n",
              "0                                     mean   \n",
              "1                                     mean   \n",
              "2                                     mean   \n",
              "\n",
              "                                              params  split0_test_score  \\\n",
              "0  {'GBC__criterion': 'friedman_mse', 'GBC__learn...           0.891222   \n",
              "1  {'GBC__criterion': 'friedman_mse', 'GBC__learn...           0.891005   \n",
              "2  {'GBC__criterion': 'friedman_mse', 'GBC__learn...           0.890502   \n",
              "\n",
              "   split1_test_score  split2_test_score  mean_test_score  std_test_score  \\\n",
              "0           0.880528           0.874405         0.882052        0.006950   \n",
              "1           0.880327           0.874400         0.881911        0.006871   \n",
              "2           0.880140           0.873701         0.881448        0.006921   \n",
              "\n",
              "   rank_test_score  \n",
              "0                1  \n",
              "1                2  \n",
              "2                3  "
            ]
          },
          "metadata": {},
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "grid_search.best_params_ , grid_search.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "udxXn7cJHSX7",
        "outputId": "ec254cb1-144e-4c28-d406-d4c48dc43cb7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "({'GBC__criterion': 'friedman_mse',\n",
              "  'GBC__learning_rate': 0.1,\n",
              "  'GBC__loss': 'exponential',\n",
              "  'GBC__min_samples_leaf': 3,\n",
              "  'GBC__n_estimators': 220,\n",
              "  'preprocess_data__fillna__fill_type': 'mean'},\n",
              " 0.8820518702075271)"
            ]
          },
          "metadata": {},
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "the best acc is 0.8820518702075271\n",
        "with \n",
        "\n",
        "{'GBC__criterion': 'friedman_mse',\n",
        "  'GBC__learning_rate': 0.1,\n",
        "  'GBC__loss': 'exponential',\n",
        "  'GBC__min_samples_leaf': 3,\n",
        "  'GBC__n_estimators': 220,\n",
        "  'preprocess_data__fillna__fill_type': 'mean'}\n",
        "\n",
        "  and this is the model i used for submission"
      ],
      "metadata": {
        "id": "M8ymYRiMMMHz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Questions\n"
      ],
      "metadata": {
        "id": "tLOyek3_8gvF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        " Why a simple linear regression model (without any activation function) is not good for classification task, compared to Perceptron/Logistic regression?\n",
        " * In most cases, data is not linearly separable. Simple linear regression is only suitable for separating two-dimensional data, and usses only a line for the decision boundary. This does not work for higher dimensional data, which requires a more complex classifier such as SVM."
      ],
      "metadata": {
        "id": "pQD1cthw4x9y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "What's a decision tree and how it is different to a logistic regression model?\n",
        "* Decision trees split data into smaller subsets based on the values of certain features. This process reduces the full dataset until all data points belonging to a subset within one of the leaf nodes has the same answer. Logistic regression is different in that the predicted output is based on a function, and works well with continuous data. Both classifiers are similar as they are discriminative."
      ],
      "metadata": {
        "id": "NOIAe59EE1x7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "What's the difference between grid search and random search?\n",
        "*  Grid Search, we try every combination of a preset list of values of the hyper-parameters and evaluate the model for each combination. The pattern followed here is similar to the grid, where all the values are placed in the form of a matrix. Each set of parameters is taken into consideration and the accuracy is noted. Once all the combinations are evaluated, the model with the set of parameters which give the top accuracy is considered to be the best.\n",
        "* Random search is a technique where random combinations of the hyperparameters are used to find the best solution for the built model. It tries random combinations of a range of values. To optimise with random search, the function is evaluated at some number of random configurations in the parameter space.\n",
        "\n",
        "ref https://medium.com/@senapati.dipak97/grid-search-vs-random-search-d34c92946318"
      ],
      "metadata": {
        "id": "I3HyK56SE-Oh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "What's the difference between bayesian search and random search?\n",
        "* Random search is a technique where random combinations of the hyperparameters are used to find the best solution for the built model. It tries random combinations of a range of values. To optimise with random search, the function is evaluated at some number of random configurations in the parameter space.\n",
        "* Bayesian optimization is a sequential model-based optimization (SMBO) algorithm that uses the results from the previous iteration to decide the next hyperparameter value candidates.\n",
        "So instead of blindly searching the hyperparameter space (like in grid search and random search), this method advocates the usage of intelligence to pick the next set of hyperparameters which will improve the model performance. We iteratively repeat this process until we converge to an optimum.\n",
        "\n",
        "ref: https://medium.com/analytics-vidhya/comparison-of-hyperparameter-tuning-algorithms-grid-search-random-search-bayesian-optimization-5326aaef1bd1"
      ],
      "metadata": {
        "id": "D4nTz71OFvyo"
      }
    }
  ]
}