NB6 Comparison between different classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "b1daa992-4a28-496f-b325-6e8be1a7844c"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Automate the ML process using pipelines \n",
    "\n",
    "There are standard workflows in a machine learning project that can be automated. In Python scikit-learn, Pipelines help to clearly define and automate these workflows.\n",
    "* Pipelines help overcome common problems like data leakage in your test harness. \n",
    "* Python scikit-learn provides a Pipeline utility to help automate machine learning workflows.\n",
    "* Pipelines work by allowing for a linear sequence of data transforms to be chained together culminating in a modeling process that can be evaluated.\n",
    "\n",
    "### Data Preparation and Modeling Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create a pipeline that standardizes the data then creates a model\n",
    "#Load libraries for data processing\n",
    "import pandas as pd #data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import numpy as np\n",
    "from scipy.stats import norm\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.cross_validation import cross_val_score, KFold\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "# visualization\n",
    "import seaborn as sns \n",
    "plt.style.use('fivethirtyeight')\n",
    "sns.set_style(\"white\")\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (8,4) \n",
    "#plt.rcParams['axes.titlesize'] = 'large'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Some Algorithms\n",
    "Now it is time to create some models of the data and estimate their accuracy on unseen data. Here is what we are going to cover in this step:\n",
    "1. Separate out a validation dataset.\n",
    "2. Setup the test harness to use 10-fold cross validation.\n",
    "3. Build 5 different models  \n",
    "4. Select the best model\n",
    "\n",
    "## 1.0 Validation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load data\n",
    "data = pd.read_csv('data/clean-data.csv', index_col=False)\n",
    "data.drop('Unnamed: 0',axis=1, inplace=True)\n",
    "\n",
    "# Split-out validation dataset\n",
    "array = data.values\n",
    "X = array[:,1:31]\n",
    "y = array[:,0]\n",
    "\n",
    "# Divide records in training and testing sets.\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)\n",
    "\n",
    "#transform the class labels from their original string representation (M and B) into integers\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Evaluate Algorithms: Baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR: 0.944808 (0.029070)\n",
      "LDA: 0.957372 (0.029611)\n",
      "KNN: 0.929872 (0.036675)\n",
      "CART: 0.914615 (0.049182)\n",
      "NB: 0.934679 (0.044038)\n",
      "SVM: 0.605769 (0.073092)\n",
      "-> 10-Fold cross-validation accurcay score for the training data for six classifiers\n"
     ]
    }
   ],
   "source": [
    "# Spot-Check Algorithms\n",
    "models = []\n",
    "models.append(( 'LR' , LogisticRegression()))\n",
    "models.append(( 'LDA' , LinearDiscriminantAnalysis()))\n",
    "models.append(( 'KNN' , KNeighborsClassifier()))\n",
    "models.append(( 'CART' , DecisionTreeClassifier()))\n",
    "models.append(( 'NB' , GaussianNB()))\n",
    "models.append(( 'SVM' , SVC()))\n",
    "\n",
    "# Test options and evaluation metric\n",
    "num_folds = 10\n",
    "num_instances = len(X_train)\n",
    "seed = 7 \n",
    "scoring =  'accuracy'\n",
    "\n",
    "# Test options and evaluation metric\n",
    "num_folds = 10\n",
    "num_instances = len(X_train)\n",
    "seed = 7 \n",
    "scoring =  'accuracy'\n",
    "results = []\n",
    "names = []\n",
    "for name, model in models:\n",
    " kfold = KFold(n=num_instances, n_folds=num_folds, random_state=seed)\n",
    " cv_results = cross_val_score(model, X_train, y_train, cv=kfold, scoring=scoring)\n",
    " results.append(cv_results)\n",
    " names.append(name)\n",
    " msg = \"%s: %f (%f)\" % (name, cv_results.mean(), cv_results.std())\n",
    " print(msg)\n",
    "print('-> 10-Fold cross-validation accurcay score for the training data for six classifiers') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Observation\n",
    "> The results suggest That both Logistic Regression and LDA may be worth further study. These are just mean accuracy values. It is always wise to look at the distribution of accuracy values calculated across cross validation folds. We can do that graphically using box and whisker plots.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAEkCAYAAABt1EhbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X1UFHX///HXsoSpoEKSZXkXuXiz+UO0tNJULslOKqKo\nYEJmdp/WpdZlduN9iGZXncwsukw8ZoaiKVheXRGmXWaWFimanu4OWVphoAbkhcD8/vC43zbSxfrA\nsvB8nMM5OzOfmXnPrLiv/cxnBptlWZYAAAAM8vN2AQAAoP4hYAAAAOMIGAAAwDgCBgAAMI6AAQAA\njCNgAHUYN3kB8FUEDDRojz76qMLDw7V48eIqy5KSkjRmzJhaq+X3+0tPT9eCBQtc0+vXr1d4eLjy\n8/NrrSZJKisr04oVKzRq1Cj17NlTPXv21KhRo7R27VpVVFTUai01qbbfb6C+8/d2AYC3lJSUaPPm\nzXI4HFq7dq3uu+8+2e12r9Uzc+ZMt+klS5bouuuu81I1pxUWFuqOO+7Q999/r8TERE2aNEmWZen9\n99/XzJkztWPHDj311FNePW+m/P78A/hrCBhosDZv3qxTp05p5syZGjt2rHJychQdHe21eq688kqv\n7ftspk+frsOHDys9PV3t27d3ze/Xr5/at2+vuXPn6oYbblBsbKz3ijSkLp5/wJdxiQQN1rp163T1\n1VerZ8+e6tSpk1avXn3O9qdOndKiRYt0ww03qFu3bpowYYI2bNig8PBwfffdd652H3zwgcaOHase\nPXrommuu0ZQpU3T48GHX8vXr16tLly7KyMhQnz59dPXVVysvL8+tiz4qKko//vij3njjjSrb37t3\nr8aOHaurrrpKffr00TPPPON2qSI8PFyrVq3S9OnT1aNHD1199dWaN2+eTp48qaeeekq9e/fWNddc\no+nTp+vkyZNnPd6DBw/qvffe04QJE9zCxRljxoxRUlKSWrRo4Zr3008/6dFHH1X//v3VrVs3xcXF\nKTs72229P1tfeHi40tLS9NBDD6l79+667rrrXOudUVFRodTUVA0ZMkTdunVTRESEEhIStGPHjvM6\n/2fex9GjR6t79+7q0aOHJkyYoD179rgdS3Xf6z179uiWW25Rt27d1KdPHz399NP16vIS8EcIGGiQ\nvv76a33yySeub94jRozQBx98cM7xDTNmzNCKFSuUmJioJUuWqGXLlnriiSfc2mzYsEHjx49XaGio\n/vnPf+rRRx9Vbm6uRo8erYKCAle7iooKvfzyy5o7d66mT5+uLl26uG3n+eefV0hIiPr166f09HRd\nfPHFrmUzZ85UdHS0UlNTFRUVpRdffFHp6elu6z/99NOy2+16/vnnNWLECK1cuVKxsbE6dOiQFi1a\npFtvvVXr16/XihUrznq8W7dulXQ67PwRu92uxx9/XP3795ckHT16VCNHjtTOnTv197//XYsXL9bl\nl1+u+++/X2+88YaR+p5//nkdO3ZMzz77rO644w6tWbNGDz/8sGv5okWLtGTJEsXHx+tf//qX5s2b\np2PHjumBBx5QSUlJtc//oUOHdN9996lLly564YUXtHDhQhUVFWnChAkqLi6WVP33urKyUpMmTdLA\ngQOVmpqqm266SampqVXeM6DesYAGaOHChVZkZKRVWlpqWZZlFRYWWl27drVSUlJcbRITE62EhATL\nsiwrPz/fCg8Pt5YtW+a2ndtvv91yOBzWoUOHrIqKCuv666+3xo0b59YmPz/f6tq1q5WcnGxZlmWt\nW7fOcjgc1rp169za/XZ/lmVZffv2taZNm+aaPrPeypUrXfPO7HPixImueQ6Hwxo+fLhrury83IqI\niLD69+9vlZWVueYPGTLEuuuuu856jmbNmmU5HA7r5MmTZ23zWwsXLrS6du1qffvtt27zx40bZ/Xu\n3ds6derUX6rP4XBYN954o2s7lmVZy5cvtxwOh3Xw4EHLsixrypQp1iuvvOK2/7fffttyOBzWxx9/\nbFlW9c7/m2++aTkcDuvIkSOu5V999ZW1cOFC64cffjjv93rVqlVu7aKioqwJEyac7VQC9QI9GGhw\nysvLtXHjRkVFRenUqVM6ceKE7Ha7+vTpo/Xr16usrKzKOjt37pRlWbrxxhvd5g8ZMsT1+ptvvlFB\nQYEGDx7s1qZt27bq3r27PvroI7f5nTt3/lP19+zZ0/Xaz89Pl19+uY4fP+7WJjIy0vXabrcrODhY\nXbp00QUXXOCa36JFC504ceKs+zkzcLO6XfkfffSRunXrpjZt2rjNj4mJUWFhob788su/XN/NN98s\nf///Gzo2aNAgSdLHH38s6XTPyPjx41VYWKhdu3Zp3bp1yszMlHT6Etdvnev8R0REqHHjxho5cqTm\nzZun9957T61bt9bDDz+sVq1anfd7/dvjlaRLLrlEv/7661n3D9QHDPJEg7N161YVFBQoMzPT9eHz\nW5s3b9awYcPc5hUWFkqSQkJC3OZfdNFFrtfHjh2TJIWGhlbZZsuWLd2uzUtSkyZN/lT9jRs3dpu2\n2WxVnpfRtGlTj+t5ctlll0mSDh8+fNYBkD/++KNCQ0Pl5+en48ePq1OnTlXatGzZUpL0yy+//OX6\nWrVq5TZ95vyfOfd79+7V7NmztXfvXl144YW68sorXcfx+3N0rvPfunVrvfrqq0pNTdX69eu1cuVK\nNWnSRMOHD9cjjzxy3u/174/Nz89P5eXlHo8X8GUEDDQ469at0yWXXKKFCxdWWTZ16lStXr26SsA4\n88H2888/u30wnQkeklyDHX97/f2MgoICBQcHG6m/tvTp00fS6UD2RwHDsizFx8erdevWeu2119S8\neXMdPXq0Srsz58PE8Z/5YD/jzP4uuugiFRcX64477pDD4dCmTZsUFhYmPz8/bd26VW+//fZ578vp\ndOq5555TeXm5cnNztXHjRq1atUqtW7fWgAEDJNWf9xqoCVwiQYNy9OhRbd26VYMHD1avXr2q/Awd\nOlSffvqpDhw44LZejx49ZLfb9c4777jN/89//uN63aFDB4WGhurNN990a3Po0CHl5uaqR48e51Wr\nt58t0bFjR91www1KTU3VoUOHqixfsWKFjhw54gpjV199tfbs2VOlbWZmpkJCQtShQ4e/XFNOTo7b\n9Ntvvy2bzabrr79eX3/9tY4dO6bExER17NhRfn6n/3vbtm2bpNODLasrLS1NAwYMUFlZmfz9/dWz\nZ0/NnTtXTZo00eHDh42/10B9RA8GGpQNGzaovLxcMTExf7h8+PDheuWVV/T666+7zW/Tpo3i4uL0\n7LPPqqKiQp07d9Y777yjLVu2SDrd5e3n56cpU6Zo+vTpmjx5smJjY3Xs2DEtXrxYQUFBuv3228+r\n1mbNmmn//v2usQ3eMHv2bI0bN06jRo1SUlKSunfvrtLSUmVnZ2vjxo0aMmSIRo8eLUkaP368MjMz\nddttt2nSpEkKDg7Wxo0b9eGHH2ru3LlGAtOePXv00EMPadiwYTpw4IAWL16s0aNHq02bNmrRooUC\nAwOVmpqqRo0ayd/fX2+//bYyMjIk6bzGPPTu3VuLFi3Svffeq6SkJAUEBCgrK0snT57UTTfdZPy9\nBuojAgYalPXr1+vKK6/8w7ECkuRwONS1a1dlZmaqffv2atSokWvZE088oSZNmmj58uUqLi7Wtdde\nq3vvvVdLlixxXTYZMWKEmjZtqpdeekkTJ05U06ZNdf3112vq1KlVxg94cscdd2j+/PmaMGGCli9f\n/ucP+i9o3bq10tPTtWLFCr311ltatmyZ/P391aFDByUnJysmJkY2m03S6fEIq1ev1tNPP63k5GT9\n73//U3h4uJ5//nljDzBLTExUUVGRJk6cqODgYN1zzz26++67JUlBQUGuW0offPBBNW3aVJ07d9ar\nr76qO++8U7t27ap2HZ06ddJLL72kJUuW6OGHH1ZZWZk6deqkF154Qddcc40ks+81UB/ZrN+PfAJQ\nxbFjx7Rt2zb17dvX7fr6woULtW7dOu3cudOL1TUM4eHhuueeezR58mRvlwKgGujBAKqhcePGevLJ\nJ9W5c2eNGzdOTZo0UW5urlauXOn6Bg0A+D8EDKAaGjVqpLS0ND377LN65JFH9Ouvv6pt27aaNm2a\nxo4d6+3yAKDO4RIJAAAwjttUAQCAcQQMAABgHAEDAAAYR8AAAADGETAAAIBxBAwAAGAcAQMAABhH\nwAAAAMYRMAAAgHEEDAAAYBwBAwAAGEfAAAAAxtXqX1M9efKk8vLyFBoaKrvdXpu7BgAABlVUVKig\noEBOp1MXXnhhleW1GjDy8vL409YAANQjq1atUs+ePavMr9WAERoa6irmkksuqc1dAwAAg3744QeN\nHTvW9dn+e7UaMM5cFrnkkkt0+eWX1+auAQBADTjbkAcGeQIAAOMIGAAAwDgCBgAAMI6AAQAAjKtW\nwPjss8+UlJRUZX5OTo7i4uIUHx+vNWvWGC8OAAD4Jo93kbz88svKzMxU48aN3eafOnVK8+fPV0ZG\nhho3bqwxY8YoKipKLVu2rLFiAQCAb/DYg9G2bVstXry4yvyvvvpKbdu2VfPmzRUQEKAePXro448/\nrpEiAQCAb/EYMAYNGiR//6odHcXFxQoKCnJNN23aVMXFxWarqwFOp1M2m83Yj9Pp9PYheY3pc9nQ\nzydQV/H/Jv6MP/2grcDAQJWUlLimS0pK3AJHXZWXl1etdjabTZZl1XA1vo1zCTQM/K7jz/jTd5GE\nhYUpPz9fx44dU1lZmXbt2qXu3bubrA0AAPio8+7ByMrKUmlpqeLj4/XII49owoQJsixLcXFxatWq\nVU3UCAAAfEy1Asbll1/uug116NChrvlRUVGKioqqmcoAAIDP4kFbAADAOAIGAAAwjoABAACMI2AA\nAADjCBgAAMC4P/2gLdRfedFd1blpgLHtlQ+LUEWs2WekfF5SJuc7+4xuE/WH0+nUvn1m/3107dq1\n2g+c8gUhISEqKioyvl2bzWZ0e8HBwSosLDS6TdQOAgaquCp7f51/Gt9VNpvqdoXwpvMJAg316ZNF\nRUU+cdymAwtqD5dIAACAcfWiB4OuPqBh8IXfdX7PgdPqRcCgqw9oGHzhd53fc+A0LpEAAADjCBgA\nAMA4AgYAADCOgAEAAIwjYAAAAOPqxV0kQF1VU7dVmsatlQBMI2AANcgXbquUuLUSgHlcIgEAAMYR\nMAAAgHEEDAAAYBxjMACgAcodEK6K2O7eLsOj3AHh3i4Bf1K9CBj8ogDA+YnYctAnBiBH2Gyq+1Xi\nj9SLgMEvCgAAdUu9CBgwr67fthgcHOztEgAA50DAQBWme4NsNptP9DABAMypNwGjrn/jlvjW3RD5\nyvigvQO7eLsEAPVMvQgYNfHtmG/dMOH/5Rwwvs2a+LfpNLo1AOA5GAAAoAYQMAAAgHEEDAAAYBwB\nAwAAGEfAAAAAxnkMGJWVlZoxY4bi4+OVlJSk/Px8t+UbNmzQ0KFDdcstt2jt2rU1VigAAPAdHgNG\ndna2ysrKlJ6erqlTpyolJcW1rLCwUM8995xWrlypV199VVlZWfruu+9qtGAAAFD3eQwYu3fvVt++\nfSVJERERysvLcy377rvvFB4erhYtWsjPz09XXXWVPvvss5qrFgAA+ASPAaO4uFiBgYGuabvdrvLy\ncklSu3bt9OWXX+ro0aP69ddftWPHDpWWltZctQAAwCd4fJJnYGCgSkpKXNOVlZXy9z+9WvPmzTV9\n+nRNmjRJLVq0UNeuXev847CdTqf27dtXrbbVefx4165d3Xp1GhLT51LifPJv89x84dHruQPCvV1C\ntfEnFlCTPAaMyMhIbdmyRTfffLNyc3PlcDhcy8rLy7V//3699tprOnXqlMaPH6/JkyfXaMF/VX37\nD9ebOJdmcT49i9hysM4/wj/CZlPdrvA0/sQCaprHgBEdHa3t27crISFBlmUpOTlZWVlZKi0tVXx8\nvCRp+PDhatSokcaPH6+QkJAaLxoAANRtHgOGn5+f5syZ4zYvLCzM9XrixImaOHGi+coAAIDP4kFb\nAADAOAIGAAAwjoABAACM8zgGAwDqkrp+ayW3VQKnETAA+AxurQR8B5dIAACAcQQMAABgHAEDAHBO\nTqdTNpvN44+karVzOp1ePiLUBsZgAADOicfY48+gBwMAABhHwAAAAMYRMAAAgHEEDAAAYBwBAwAA\nGEfAAAAAxhEwAACAcQQMAABgHAEDAAAYR8AAAADGETAAAIBxBAwAAGAcAQMAABhHwAAAAMYRMAAA\ngHEEDAAAYBwBAwAAGEfAAFDvOJ1O2Wy2av1IqlY7p9Pp5aMCfIu/twsAANPy8vK8XQLQ4NGDAQAA\njCNgAAAA4wgYAADAOAIGAAAwjoABAACM8xgwKisrNWPGDMXHxyspKUn5+fluyzMzMzV8+HDFxcXp\ntddeq7FCAQCA7/B4m2p2drbKysqUnp6u3NxcpaSkaOnSpa7lCxcu1KZNm9SkSRMNHjxYgwcPVvPm\nzWu0aAAAULd5DBi7d+9W3759JUkRERFV7i8PDw/XL7/8In9/f1mW5XpwDQAAaLg8Bozi4mIFBga6\npu12u8rLy+Xvf3rVjh07Ki4uTo0bN1Z0dLSaNWtWc9UCAACf4HEMRmBgoEpKSlzTlZWVrnBx4MAB\nvffee3r33XeVk5OjwsJCbd68ueaqBQAAPsFjwIiMjNS2bdskSbm5uXI4HK5lQUFBuvDCC9WoUSPZ\n7XaFhIToxIkTNVctAADwCR4vkURHR2v79u1KSEiQZVlKTk5WVlaWSktLFR8fr/j4eN1yyy264IIL\n1LZtWw0fPrw26gYAAHWYx4Dh5+enOXPmuM0LCwtzvR4zZozGjBljvjIAAOCzeNAWAAAwjoABAACM\nI2AAAADjCBgAAMA4AgYAADCOgAEAAIwjYAAAAOMIGAAAwDgCBgAAMI6AAQAAjCNgAAAA4wgYAADA\nOAIGAAAwjoABAACMI2AAAADjCBgAAMA4AgYAADCOgAEAAIwjYAAAAOMIGAAAwDgCBgAAMI6AAQAA\njCNgAAAA4wgYAADAOAIGAAAwjoABAACMI2AAAADjCBgAAMA4AgYAADCOgAEAAIwjYAAAAOMIGAAA\nwDh/Tw0qKys1a9YsHTx4UAEBAZo3b57atWsnSSooKNCUKVNcbT///HNNnTpVY8aMqbmKAQBAnecx\nYGRnZ6usrEzp6enKzc1VSkqKli5dKkkKDQ3VypUrJUmffvqpnnnmGY0ePbpmKwYAAHWex4Cxe/du\n9e3bV5IUERGhvLy8Km0sy9LcuXO1aNEi2e1281UCAACf4nEMRnFxsQIDA13Tdrtd5eXlbm1ycnLU\nsWNHXXHFFeYrBAAAPsdjwAgMDFRJSYlrurKyUv7+7h0fmZmZXBoBAAAuHgNGZGSktm3bJknKzc2V\nw+Go0iYvL0+RkZHmqwMAAD7JY8CIjo5WQECAEhISNH/+fE2fPl1ZWVlKT0+XJBUWFiowMFA2m63G\niwUAwJc5nU7ZbDajP06n09uH9Yc8DvL08/PTnDlz3OaFhYW5XoeEhGjjxo3mKwMAoJ75oxslzsZm\ns8myrBqspmbxoC0AAGAcAQMAABhHwAAAAMYRMAAAgHEEDAAAYJzHu0gAAMC5hYSEqKioyPh2TT4C\nIjg4WIWFhca25wkBAwCAv6ioqKjO31Ja28+r4hIJAAAwjoABAACMI2AAAADjCBgAAMA4AgYAADCO\ngAEAAIwjYAAAAOMIGAAAwDgCBgAAMI6AAQAAjCNgAAAA4wgYAADAOAIGAAAwjoABAACMI2AAAADj\nCBgAAMA4AgYAADCOgAEAAIwjYAAAAOMIGAAAwDgCBgAAMI6AAQAAjCNgAAAA4wgYAADAOAIGAAAw\njoABAACM8/fUoLKyUrNmzdLBgwcVEBCgefPmqV27dq7le/bsUUpKiizLUmhoqJ566ik1atSoRosG\nAKAuyR0QrorY7t4u45xyB4TX6v48Bozs7GyVlZUpPT1dubm5SklJ0dKlSyVJlmXpiSee0HPPPad2\n7dpp7dq1+v7773XFFVfUeOEAANQVEVsOyrIsb5dxThE2m2qzQo8BY/fu3erbt68kKSIiQnl5ea5l\n33zzjVq0aKG0tDR98cUX6tevH+ECAAB4HoNRXFyswMBA17Tdbld5ebkkqaioSJ9++qkSExO1fPly\nffjhh9qxY0fNVQsAAHyCx4ARGBiokpIS13RlZaX8/U93fLRo0ULt2rVTWFiYLrjgAvXt29ethwMA\nADRMHgNGZGSktm3bJknKzc2Vw+FwLWvTpo1KSkqUn58vSdq1a5c6duxYQ6UCAABf4XEMRnR0tLZv\n366EhARZlqXk5GRlZWWptLRU8fHxevLJJzV16lRZlqXu3burf//+tVA2AACoyzwGDD8/P82ZM8dt\nXlhYmOv1tddeq4yMDPOVAQAAn8WDtgAAgHEEDAAAYBwBAwAAGEfAAAAAxhEwAACAcQQMAABgHAED\nAAAYR8AAAADGETAAAIBxBAwAAGCcx0eFAwAAz2w2m7dLOKfg4OBa3R8BAwCAv8iyLOPbtNlsNbLd\n2sIlEgAAYBwBAwAAGEfAAAAAxhEwAACAcQQMAABgHAEDAAAYR8AAAADGETAAAIBxBAwAAGAcAQMA\nABhHwAAAAMYRMAAAgHEEDAAAYBwBAwAAGEfAAAAAxhEwAACAcQQMAABgHAEDAAAYR8AAAADGETAA\nAIBx/p4aVFZWatasWTp48KACAgI0b948tWvXzrU8LS1Na9euVUhIiCRp9uzZuuKKK2quYgAAUOd5\nDBjZ2dkqKytTenq6cnNzlZKSoqVLl7qW5+XlacGCBXI6nTVaKAAA8B0eA8bu3bvVt29fSVJERITy\n8vLclu/bt0+pqakqKChQ//79dffdd9dMpQAAwGd4HINRXFyswMBA17Tdbld5eblrevDgwZo1a5ZW\nrFih3bt3a8uWLTVTKQAA8BkeA0ZgYKBKSkpc05WVlfL3P93xYVmWxo0bp5CQEAUEBKhfv37av39/\nzVULAAB8gseAERkZqW3btkmScnNz5XA4XMuKi4s1ZMgQlZSUyLIs7dy5k7EYAADA8xiM6Ohobd++\nXQkJCbIsS8nJycrKylJpaani4+M1efJk3XrrrQoICNC1116rfv361UbdAACgDvMYMPz8/DRnzhy3\neWFhYa7XsbGxio2NNV8ZAADwWTxoCwAAGEfAAAAAxhEwAACAcQQMAABgHAEDAAAYR8AAAKCWOJ1O\n2Wy2av1Iqla7uvr8KY+3qQIAADN+//e86jN6MAAAgHEEDAAAYBwBAwAAGEfAAAAAxhEwAACAcQQM\nAABgHAEDAAAYR8AAAADG1eqDtioqKiRJP/zwQ23uFgAAGHbms/zMZ/vv1WrAKCgokCSNHTu2NncL\nAABqSEFBgdq1a1dlvs2yLKu2ijh58qTy8vIUGhoqu91eW7sFAACGVVRUqKCgQE6nUxdeeGGV5bUa\nMAAAQMPAIE8AAGAcAQMAABhHwAAAAMYRMAAAgHENPmDs3LlTkydPdpuXlJSkkSNHKikpSWPHjtXQ\noUO1detWL1VY93k6hwkJCXrooYdUVFTk1mbYsGGaPXt2bZZap/3+PP773//WkCFDdOutt2rixIlu\nba+//npJ0vr16xUVFaXi4mLXssmTJ2vnzp21U3Qd9MUXX+iuu+5SUlKS4uLi9Nxzz+nMWPa33npL\nERER+vHHH13tFy9erEGDBikpKUlJSUkaOnSoli5dKkmaOnWqkpKSFBUV5Wozd+5crxxXXbJz5071\n6NFDR44ccc1btGiR1q9fL6fTqaSkJCUmJmrEiBHauHGjFyutm1JTU3XbbbcpMTFRSUlJysvLU1RU\nlH57z8WpU6cUFRWlX375ReHh4ZoxY4bbNubNm6eoqKjaLv281OpzMHzJggULFBYWJkn6+uuv9cAD\nD6hfv35ersq3/PYcZmZmasaMGVq8eLEkaffu3XI4HPrwww9VXFyswMBAb5Za52zatEmvvPKK0tLS\ntGjRIm3dulUbNmxQbGxslba//vqrkpOTlZyc7IVK65YTJ05oypQpWrx4sdq3b6+Kigo9+OCDev31\n1zVmzBitXbtWSUlJWrNmjSZNmuRa77bbbtOYMWMkSWVlZbr55ps1evRoPf3005JOh5CWLVu62kAK\nCAjQ9OnTtXz5ctlsNtf85s2ba+XKlZKkX375RYMGDVJMTIxbm4bsyy+/VE5OjlavXi2bzabPP/9c\n06ZNU9u2bfXRRx+pV69ekqScnBz16tVLQUFBatGihXbt2qXy8nL5+/uroqJCe/fu9fKReNbgezCq\n4/Dhw2rWrJm3y/BpMTEx2rdvn/73v/9JktauXatBgwYpOjpaGzZs8HJ1dcuGDRuUlpam5cuXq2XL\nlpLk+tD8o6fgxsbG6uuvv9aWLVtqu9Q6591331WvXr3Uvn17SZLdbteCBQsUFxenQ4cO6fjx47rz\nzju1ceNGnTp16g+3UVRUpPLycjVq1KgWK/c9vXv3VvPmzbVq1aqztikuLlazZs0IF78RFBSkw4cP\nKyMjQz/++KM6d+6sjIwMjR492u3/wnXr1ik+Pl6S5O/vr2uuuUbbt2+XJP33v//Vdddd55X6zwcB\n4yymTZumhIQE3XDDDVqzZo3mz5/v7ZJ8XrNmzXTixAkVFxdr9+7d6t+/v0aMGKHVq1d7u7Q6Y9eu\nXVqzZo2OHz/u9vjdVq1a6cEHH9Rjjz1WZR273a6UlBQlJydXuQzV0Pz0009q06aN27ymTZsqICBA\nGRkZiouLU7NmzRQREaF33nnH1SYtLU2JiYn629/+psmTJ2vevHn0qlXDrFmzlJaWpvz8fNe848eP\nuy4vx8TE6Oabb/ZihXVPq1attHTpUn3yySeKj4/XTTfdpC1btmjgwIH6+OOPdfLkSf300086evSo\nIiIiXOsNGTJEb731lqTTPZxDhw711iFUG5dIzuJM9/7rr7+uTZs26dJLL/V2ST7NsiwdPXpUF110\nkV5//XVVVlbq7rvvlnT6MbM7duzQtdde6+UqvS80NFTLly/X2rVr9fDDD+vll192LYuJiVF2drZe\ne+21Kus9IXvtAAAC3ElEQVS1b99et956q2bPnt2gvy22bt1a+/fvd5t36NAhHTlyRFlZWbrsssuU\nk5Oj48eP69VXX3V9+J25RJKXl6cpU6a4ekBwbsHBwXr00Uc1bdo0RUZGSnK/RFJcXKyEhARdd911\n6tmzpzdLrTPy8/MVGBjo+tK6d+9e3XnnnerVq5cGDhyo7OxsHT58WHFxcW7r9ejRQ7Nnz1ZRUZGO\nHTumyy67zBvlnxd6MDxISEjQpZdeqmeeecbbpfi0jIwM9e7dW35+fsrIyNCLL76oZcuWadmyZXr8\n8cfP2c3akLRr106NGjVSYmKiLrjgAtdgwzNmzZqlV155RSUlJVXWTUxMVFFRkT788MPaKrfOGTBg\ngN5//319++23kk4PlEtJSdHnn38up9OplStXatmyZcrIyNDPP/+sAwcOuK3vdDp15513asqUKaqs\nrPTGIficqKgodejQQW+88UaVZU2bNlVQUNBZL0c1RAcPHtScOXNUVlYmSerQoYOaNWsmu92uUaNG\nadOmTcrOzlZMTIzbejabTf369dOsWbM0cOBAb5R+3ujBkLR9+3aNGDHCNf3TTz+5LX/ssccUExOj\nYcOGqVOnTrVdnk/4o3M4bdo0NW7cWNLpbsGZM2dq3759sixLHTt2dLUdNGiQ5s+fryNHjtBT9BvJ\nycmKjY2V3W53fdMOCQnRI488ovvvv79Ke5vNpvnz5/tE12lNCQwMVEpKih5//HFZlqWSkhINGDBA\nO3bs0KhRo9zajhw5UqtWrdLFF1/sNn/UqFHavHmzVq9ezR9mrKbHHnvMFWzPXCKRTg+Yveqqq9S7\nd29vllen3Hjjjfrqq680cuRINWnSRJZl6R//+IeCgoIUFBSk0tJShYWFKSgoqMq6Q4cO1ciRIzVn\nzhwvVH7++FskAADAOC6RAAAA4wgYAADAOAIGAAAwjoABAACMI2AAAADjCBgAAMA4AgYAADCOgAEA\nAIz7/4dfBKkcrbPSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116954110>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compare Algorithms\n",
    "fig = plt.figure()\n",
    "fig.suptitle( 'Algorithm Comparison' )\n",
    "ax = fig.add_subplot(111)\n",
    "plt.boxplot(results)\n",
    "ax.set_xticklabels(names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observation\n",
    "> The results show a similar tight distribution for all classifiers except SVM which is encouraging, suggesting low variance. The poor results for SVM are surprising.\n",
    "\n",
    "> It is possible the varied distribution of the attributes may have an effect on the accuracy of algorithms such as SVM. In the next section we will repeat this spot-check with a standardized copy of the training dataset.\n",
    "\n",
    "\n",
    "### 2.1 Evaluate Algorithms: Standardize Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sheeroh/anaconda/lib/python2.7/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, _DataConversionWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ScaledLR: 0.972372 (0.034455)\n",
      "ScaledLDA: 0.957372 (0.029611)\n",
      "ScaledKNN: 0.947308 (0.044048)\n",
      "ScaledCART: 0.912115 (0.045257)\n",
      "ScaledNB: 0.934679 (0.041102)\n",
      "ScaledSVM: 0.964872 (0.032078)\n"
     ]
    }
   ],
   "source": [
    "# Standardize the dataset\n",
    "pipelines = []\n",
    "pipelines.append(( 'ScaledLR' , Pipeline([( 'Scaler' , StandardScaler()),( 'LR' ,\n",
    "    LogisticRegression())])))\n",
    "pipelines.append(( 'ScaledLDA' , Pipeline([( 'Scaler' , StandardScaler()),( 'LDA' ,\n",
    "    LinearDiscriminantAnalysis())])))\n",
    "pipelines.append(( 'ScaledKNN' , Pipeline([( 'Scaler' , StandardScaler()),( 'KNN' ,\n",
    "    KNeighborsClassifier())])))\n",
    "pipelines.append(( 'ScaledCART' , Pipeline([( 'Scaler' , StandardScaler()),( 'CART' ,\n",
    "    DecisionTreeClassifier())])))\n",
    "pipelines.append(( 'ScaledNB' , Pipeline([( 'Scaler' , StandardScaler()),( 'NB' ,\n",
    "    GaussianNB())])))\n",
    "pipelines.append(( 'ScaledSVM' , Pipeline([( 'Scaler' , StandardScaler()),( 'SVM' , SVC())])))\n",
    "\n",
    "results = []\n",
    "names = []\n",
    "for name, model in pipelines:\n",
    "  kfold = KFold(n=num_instances, n_folds=num_folds, random_state=seed)\n",
    "  cv_results = cross_val_score(model, X_train, y_train, cv=kfold,\n",
    "      scoring=scoring)\n",
    "  results.append(cv_results)\n",
    "  names.append(name)\n",
    "  msg = \"%s: %f (%f)\" % (name, cv_results.mean(), cv_results.std())\n",
    "  print(msg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAEkCAYAAAD9xzGUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X1YVHX+//HncCOkgEC4ursqauuwAhJiWWaaikQraSIq\niGKKlfn1tjRvV8XVkLJ083a11DbTFfMu0WoL1MibsthQ0WSzzBV1TRZTARWR8/vDy/k1oaIFHiZf\nj+viujjnc86Z9+czA/OaczcWwzAMREREREziZHYBIiIicmdTGBERERFTKYyIiIiIqRRGRERExFQK\nIyIiImIqhRER+Vl0IZ6IVBaFEXEohw4dYtSoUTz88MMEBwfz0EMP8X//93988cUXVfJ4HTt2ZPTo\n0b94O0eOHCEgIIB169bd1PIbNmwgICCAhISEa7bPnTuXgIAASktLf3FtN+Onj/fFF1/wzDPP2Nrz\n8vIICAjgnXfeuS31/NhHH33E008/TZs2bQgNDaVz587MmTOHs2fP3vZaqsrtfr5FbjcXswsQuVlf\nf/01vXr1IiQkhAkTJuDn50d+fj6rVq0iISGBefPmER4ebnaZlWLt2rVYrVZ2797NN998wz333GNq\nPT179qRt27a4uFz5l7Fq1Sq++eYbU2syDINx48axadMmunfvTmxsLLVq1WLfvn0sWbKEDz/8kLfe\negtfX19T66wMPx1/kV8bvbLFYSxbtgwvLy/eeOMNXF1dbfMjIiLo3r07s2fP/lWEkf/85z98/vnn\nLFiwgIkTJ/KPf/yDP//5z6bWVK9ePerVq2dqDT+1dOlSNmzYwLx584iIiLDNb926NQ8//DA9e/Zk\n5syZzJgxw8QqK0d1HH+RyqTDNOIw8vPzMQyj3LkKrq6ujB49ml69etnN3759O/Hx8bRo0YKHH36Y\nCRMmUFBQYGs/ePAgQ4cO5cEHHyQoKIi2bdsybdo0zp8/f90aysrKeOONN3j00UcJDg6mU6dOLFmy\npFxNH3zwAV26dCEkJITu3buTm5t70/1cu3YtNWvWpE2bNjz++OO8++67N6zpqvXr19O5c2eaN29O\n165d2bVrF4GBgXaHhr777juGDx/Oww8/zL333ktCQoLdIa6rh1vefPNNOnfuTEhICCtWrLA7TDBu\n3DjS0tI4duxYuUNP+fn5PP/887Rs2ZKWLVvy/PPP2415QkICEyZMYNGiRbRr1457772Xp556ilOn\nTrF+/XoiIiJo0aIF/fr14+jRo9fta2lpKUuWLKFdu3Z2QeSqwMBARo0aRWBgoG3exYsXWbBgAY89\n9hjNmzcnIiKCxYsXU1ZW9ovrS0hIYPTo0SxYsICHHnqIsLAwBg8eXK4P6enpttdkcHAwjz32GMuX\nL7+l8Qc4evQogwYN4sEHHyQkJITo6Gg2btxo91g3+1z/85//5LnnnqNly5aEhYXx3HPPkZ+ff92x\nF6kShoiDWLFihWG1Wo0nnnjCeOutt4zc3FyjrKzsmst+/PHHxh//+Efj2WefNbZs2WK8++67xsMP\nP2zExcUZhmEYJ0+eNMLCwozExERj69atxs6dO40ZM2YYVqvVmD9/vm07HTp0MEaNGmWbTkpKMgID\nA43Zs2cb27dvN1577TWjWbNmxksvvWRbJj093QgICDCef/55IzMz01i0aJERGhpqWK1WY+3atTfs\n4+XLl4127doZEyZMMAzDML766ivDarUa77zzjt1yc+bMMaxWq3Hp0iXDMAxj/fr1htVqNSZOnGhk\nZmYa8+fPL/eYX3/9tdGiRQujW7duxnvvvWd89NFHRr9+/YzAwEBj586dhmEYxtGjRw2r1WqEhoYa\nq1evNj788EPj6NGjdo935MgRIzEx0WjTpo3x5ZdfGv/73/9s6zVr1sz4y1/+YuzcudNYtmyZERgY\naIwYMcJWd9++fY0WLVoY8fHxxrZt24zU1FQjKCjIiIyMNLp06WJ8+OGHRlpamhEaGmokJiZed5y+\n/PJLw2q1GitWrLjheF5VVlZm9O/f3wgNDTUWL15sbN++3Zg1a5bRrFkzY9y4cb+4vr59+xotW7Y0\nIiIijM2bNxtpaWlG+/btjfbt2xtFRUWGYRjG1q1bDavVakyfPt3YuXOnsXXrVuOpp54yrFar8fnn\nn9/0+F++fNn405/+ZPTt29f22n3++ecNq9Vq7N69+5af6/vuu8+YNm2asWPHDuPtt982mjdvbvec\nidwOCiPiUObOnWuEhIQYVqvVsFqtxv33328MHz7c2LVrl91yPXr0MLp06WIXVrZs2WJ06tTJOHr0\nqPHJJ58Y8fHxxtmzZ+3We/zxx40nn3zSNv3jMHL48GEjICDAmDdvnt06ixYtMpo1a2YcP37cMAzD\n6N69uxEdHV1umZsJI9u2bTOsVqvxxRdf2OZFR0cb3bt3t1vup2Gkffv2xqBBg274mCNGjDBatWpl\n1+dLly4ZkZGRRrdu3QzD+P9vUGPHjr3h440aNcro0KGDrf3qesOGDbNbb+TIkUarVq1s03379jWC\ngoKMgoIC27yBAwcaVqvV+Pbbb23z/vKXvxihoaHXHaf33nvPsFqtxscff3zdZX7s6rhu2LDBbv78\n+fMNq9VqfPXVV7+ovr59+xqBgYHGkSNHbPP2799vWK1WY/ny5YZhGMYbb7xhjB492u7xT58+bReA\nb2b8T506Va4vly5dMmbMmGFkZWUZhnFrz/VPg8e4ceOM5s2b33A8RSqbDtOIQxk6dCjbt29n9uzZ\nxMbG4uvrywcffMCTTz7JSy+9BFzZHb9v3z46deqExWKxrduhQwc++ugj6tevz8MPP8yKFStwd3fn\n0KFDZGRksHDhQgoKCigpKbnmY3/66acYhkF4eDilpaW2n/DwcC5fvsynn37KhQsX2L9/Px07drRb\nNzIy8qb6t3btWn7/+9/TtGlTzp49y9mzZ3nsscfIyclh375911znyJEjHD9+nEcffdRuflRUlN30\n7t27adeuHZ6enrZ5Li4uREVFceDAAc6dO2eb/8c//vGm6v2pli1b2k03aNCAM2fO2M1r3LgxPj4+\ntmk/Pz+8vLxo3LixbZ63tzfFxcXXvXrk6omcPz7EciO7d+/GycmJP/3pT3bzu3btamv/pfWFhobS\nsGFD23RgYCANGjSwbXvgwIHMnDmToqIicnJyeO+991i0aBFAudfcjcb/7rvvxmq1MnnyZEaPHs2G\nDRs4e/Ys48aNIywszNafm32ur65zVb169bh48eJNj61IZdAJrOJwPD096dy5M507dwbg22+/ZeLE\niSxdupTo6Ghq166NYRjcfffd191GWVkZs2bNYsWKFRQXF1OvXj1CQkJwc3O77v0zTp8+DcATTzxx\nzfaTJ09y5swZDMOwezMDqFOnToX9KigoYMuWLVy6dIn777+/XPuqVato3rz5NdcDyl014ufnZzd9\n5syZcvN+vNyP36Bq1apVYb3XUrNmTbtpi8VSbjw9PDzKrXfXXXfd0uP87ne/A+DYsWPXXaagoICa\nNWvi7u7OmTNnqF27NjVq1LBb5urz8uPLgH9ufXXr1i037+6777aFsYKCAqZMmUJ6ejoWiwV/f3/u\nu+8+oPw9W240/haLhaVLl/K3v/2NDz/8kLS0NJydnWnbti1Tpkzhd7/73S091z/tm5PTlc+oZWVl\ntt9FqprCiDiEkydPEhMTw9ChQ4mLi7Nra9KkCRMmTKBHjx4cOnSIRx55BIvFYnfiJFw56XH79u2E\nhISwevVqli1bRlJSEpGRkXh5eQHQo0eP69ZwdZlly5bZfeK86je/+Q3e3t44OTmVOwHwapC5kY0b\nN3Lp0iXmzJmDt7e3XdvixYvZvHkz48aNK/fYV6+y+Olj/u9//7Obrl279jVPTDx16hRw5dP+T8es\numrWrBl+fn5kZmbSp0+fay6TnJxMRkYGmZmZ1K5dmzNnzlBSUmIXSL7//nuAcuHx5/jhhx/KzcvP\nz+fee+8FYPTo0Rw6dIilS5cSFhaGm5sb58+fZ/Xq1bf8WHXq1GHSpElMmjSJr7/+2rZnLykpicWL\nF/+qnmu5Myj2ikPw8/PDxcWFFStWXPPKksOHDwNgtVqpVasWzZo1IyMjw+4T56effsqgQYP4z3/+\nQ1ZWFk2aNKFnz562kHHy5En+/e9/X3fPyNVPsQUFBTRv3tz2c+HCBf76179y6tQp3NzcaNGiBR9+\n+KHdbu6tW7dW2Me1a9fSvHlzIiMjeeCBB+x++vbty/nz51m/fn259erVq0fDhg1JT0+3m//hhx/a\nTd9///1kZmbafSq+fPkymzdvJjAwsNxejRtxdna+6WWrgpOTE/3792fbtm1kZGSUa9+zZw///Oc/\n6dixI56enrRq1YqysjLef/99u+WuXoHy08NLP8eXX35p9wafk5NDXl4ebdq0ASArK4uIiAhat26N\nm5sbAJmZmcCt3c32X//6Fw899BB79+4FoGnTpjz77LM8+OCDHD9+HKjc51rkdtCeEXEIzs7OTJ06\nlSFDhtC9e3f69u3LH/7wB0pLS9m9ezd///vf6d27N3/4wx8AGD58OIMHD2bEiBHExMRw+vRpZs+e\nTZs2bbj33nsJCQnhk08+YfHixYSGhnLkyBEWLVpESUkJxcXF16whICCArl27MmXKFI4fP05ISAhH\njx5l9uzZeHt7Y7VaARg5ciT9+/dnyJAhxMXFcfjwYf72t7/dsH979+7l3//+NxMmTLhme9u2balT\npw6pqan069fPrs1isTB8+HBGjx7N5MmTefTRRzl48CDz588H/v9u96FDh5KZmUm/fv145plncHNz\nY/ny5Rw9erTC+n7Ky8uL/Px8Pv74Y5o1a3ZL61aW/v378/nnnzN8+HB69OhB+/btcXFxISsri7fe\neouGDRsyadIkANq1a8cDDzxAUlISp06dIjAwkN27d/P666/z+OOP/+xzZH7s/PnzPP300wwePJii\noiJmz56N1WqlS5cuAISEhLB582ZCQkKoV68eX375JYsWLcJisVz3NXctgYGBuLu788ILLzB06FB+\n85vfsGfPHrZv386QIUOAyn2uRW4HhRFxGI888ghr1qzh9ddf54033iA/Px8XFxcCAgKYNGkS3bt3\nty3boUMHFi1axLx58xg6dCg+Pj48+uijjBw5EovFwqBBgzh9+jRvvfUW8+fP57e//S1PPPEEFouF\nv/3tb5w+ffqau+5nzJjB4sWLeeedd5gzZw4+Pj6Eh4czYsQI2+7/Vq1asXjxYmbPns2wYcNo0KAB\nycnJDB48+Lp9W7t2Lc7OzrbzYH7KxcWFLl26sHTpUruTLa/q0qULxcXFLFmyhHXr1tG0aVMmTpzI\nxIkTbZ+CmzZtysqVK5k1a5Yt9ISEhPD3v//9mueo3EiPHj345JNPGDJkCMOHD79u3VXJ1dWVBQsW\nsHr1at59910++OADLl68SIMGDUhMTOTJJ5+0HdKyWCwsWrSIOXPmsHz5cv73v//x+9//nhEjRjBw\n4MBKqSc0NJR27doxceJE24nOY8aMsb0uUlJSmDZtGtOmTQOgUaNGTJ06lY0bN5KVlXXTj+Pu7s7S\npUuZNWsWKSkpnDlzhvr16/Pcc8/Z+lKZz7XI7WAxbmX/oIhUS5s2bSIwMJAmTZrY5n388cc888wz\nvPvuu5XyyV+uLyEhgdLSUv7xj3+YXYqIQ9KeEZFfgY0bNzJ79mxGjhzJb3/7W44cOcLcuXNp1aqV\ngoiIVHsKIyK/Ai+99BKvvvoqM2fOpKCgAD8/PyIjIxk+fLjZpYmIVEiHaURERMRUurRXRERETKUw\nIiIiIqZSGBERERFTKYyIiIiIqRRGRERExFQKIyIiImIqhRERERExlcKIiIiImEphREREREylMCIi\nIiKmUhgRERERUymMiIiIiKmq7bf2XrhwgZycHOrUqYOzs7PZ5YiIiMgvcPnyZU6dOkVwcDDu7u52\nbdU2jOTk5NCnTx+zyxAREZFKtGLFCu677z67edU2jNSpUwe4UnS9evVMrkZERER+if/+97/06dPH\n9v7+Y9U2jFw9NFOvXj3q169vcjUiIiJSGa516oVOYBURERFTKYyIiIiIqRRGRERExFQKIyIiImKq\nmwoje/bsISEhodz8LVu2EBMTQ2xsLKtXrwagrKyMyZMnExsbS0JCAkeOHAHgyJEj9O7dm/j4eKZM\nmUJZWVkldkNEREQcVYVh5PXXX+fPf/4zFy9etJt/6dIlZsyYwdKlS1m+fDmpqank5+eTnp5OSUkJ\nqampjBo1ipSUFABmzJjByJEjWblyJYZhkJGRUTU9EhEREYdSYRhp2LAhc+fOLTf/m2++oWHDhtSu\nXZsaNWrQsmVLPv/8c7Kysmjbti0AoaGh5OTkALB//35atWoFQLt27di5c2dl9kNEREQcVIVhJDIy\nEheX8rcjKSwsxNPT0zZdq1YtCgsLKSwsxMPDwzbf2dmZ0tJSDMPAYrHYlj137lxl1H/LfH19sVgs\n1f7H19fXlPGpKsHBwZU+RsHBwWZ3yzQaT6mu9NqsPHfSWP7sm555eHhQVFRkmy4qKsLT07Pc/LKy\nMlxcXHBycrJb1svL6+c+9C9y+vRpDMMw5bFvxdXg9mtxdQ/ZzbBYLA7xHJnpZsdTYym3m16bledO\n+r/5s6+mueeeezhy5Ag//PADJSUlfPHFF7Ro0YKwsDAyMzMByM7Oxmq1AhAYGMhnn30GQGZmZrn7\n0ouIiMid6Zb3jKSlpVFcXExsbCzjxo1j4MCBGIZBTEwMdevWJSIigh07dhAXF4dhGCQnJwMwduxY\nJk2axKxZs2jSpAmRkZGV3hkRERFxPBajmu7XycvLIzw8nIyMjEr9bhpH2ZXlKHVWhTu575VNYynV\nlV6blcsRxvNG7+u66ZmIiIiYSmFERERETKUwIiIiIqb62Zf2ioiIyK3z9fXl9OnTlb7dyr4lhI+P\nDwUFBZW6zetRGBEREbmNdL+r8nSYRkREREylMCIiIiKmUhgRERERUymMiIiIiKkURkRERMRUd9zV\nNNkdArjcrYXZZVQou0OA2SWIiIjcFndcGAndmusQl1SFWixU/ypFRER+OR2mEREREVMpjIiIiIip\nFEZERETEVAojIiIiYiqFERERETGVwoiIiIiYSmFERERETKUwIiIiIqZSGBERERFTKYyIiIiIqSq8\nHXxZWRlJSUnk5uZSo0YNpk+fjr+/v619w4YNLFmyBE9PT6Kjo+nZsyfr1q1j/fr1AFy8eJGvvvqK\nHTt2kJeXx6BBg2jUqBEAvXv3pnPnzlXTMxEREXEIFYaR9PR0SkpKSE1NJTs7m5SUFBYuXAhAQUEB\nc+bMYd26dXh5edG/f39at25N9+7d6d69OwBTp04lJiYGLy8v9u/fz4ABA0hMTKzaXomIiIjDqPAw\nTVZWFm3btgUgNDSUnJwcW1teXh4BAQF4e3vj5ORE8+bN2bNnj6193759HDp0iNjYWABycnLYtm0b\nffr0YcKECRQWFlZ2f0RERMTBVLhnpLCwEA8PD9u0s7MzpaWluLi44O/vz6FDh8jPz6dWrVrs2rXL\ndggGYNGiRQwZMsQ2HRISQs+ePQkODmbhwoXMnz+fsWPHVm6P5Lby9fXl9OnTlb5di8VSqdvz8fGh\noKCgUrdZ2TSW5ggODmb//v2Vus2goCC7D24iP5bdIYDL3VqYXUaFsjsE3LbHqjCMeHh4UFRUZJsu\nKyvDxeXKarVr12b8+PEMGzYMb29vgoKC8PHxAeDs2bMcPnyYBx980LZuREQEXl5ett+nTZtWqZ2R\n229r6G8I9vKveEGT5Zw9b3YJFTp9+jSGYZhdRoUqO9yY7VZCg8VicYjnSKq30K25DvE6CrVYuF1V\nVhhGwsLC2Lp1K507dyY7Oxur1WprKy0t5cCBA6xcuZJLly4xYMAAnnvuOQA+//xzWrdubbetgQMH\nMmnSJEJCQti1axdBQUGV3B253fRHJSIiv1SFYSQiIoIdO3YQFxeHYRgkJyeTlpZGcXGx7VyQ6Oho\n3NzcGDBgAL6+vgAcPnyY+vXr220rKSmJadOm4erqip+fn/aMiIiICBajmn6szcvLIzw8nIyMjHKh\n5pdwlN2sqrNyOUKdjlAjOE6dVeFO7ntlu5PH0lH6Xtl13uh9XTc9ExEREVMpjIiIiIipFEZERETE\nVBWewCoiInc23QNHqprCiIiI3JDugSNVTYdpRERExFQKIyIiImIqhRERERExlcKIiIiImEphRERE\nREylMCIiIiKmUhgRERERUymMiIiIiKkURkRERMRUCiMiIiJiKoURERERMZXCiIiIiJhKYURERERM\npTAiIiIipnIxuwAzOMLXTPv4+JhdgoiIyG1xx4URwzAqfZsWi6VKtisiInIn0GEaERERMZXCiIiI\niJiqwsM0ZWVlJCUlkZubS40aNZg+fTr+/v629g0bNrBkyRI8PT2Jjo6mZ8+eAERHR+Ph4QFA/fr1\nmTFjBkeOHGHcuHFYLBaaNm3KlClTcHJSHhIREbmTVRhG0tPTKSkpITU1lezsbFJSUli4cCEABQUF\nzJkzh3Xr1uHl5UX//v1p3bo1derUwTAMli9fbretGTNmMHLkSB544AEmT55MRkYGERERVdMzERER\ncQgV7pbIysqibdu2AISGhpKTk2Nry8vLIyAgAG9vb5ycnGjevDl79uzh4MGDnD9/nsTERPr160d2\ndjYA+/fvp1WrVgC0a9eOnTt3VkWfRERExIFUuGeksLDQdrgFwNnZmdLSUlxcXPD39+fQoUPk5+dT\nq1Ytdu3aRaNGjXB3d2fgwIH07NmT7777jqeffpoPPvgAwzBsl9XWqlWLc+fOVV3PRBxMdocALndr\nYXYZFcruEGB2CSLyK1NhGPHw8KCoqMg2XVZWhovLldVq167N+PHjGTZsGN7e3gQFBeHj40Pjxo3x\n9/fHYrHQuHFjvL29OXXqlN35IUVFRXh5eVVBl0QcU+jWXIe4RDzUYqH6VylSvel+V/YqPEwTFhZG\nZmYmANnZ2VitVltbaWkpBw4cYOXKlbz22mt8++23hIWFsWbNGlJSUgA4efIkhYWF1KlTh8DAQD77\n7DMAMjMzue+++6qiTyIiItWWYRiV/lMV2y0oKLhtY1LhnpGIiAh27NhBXFwchmGQnJxMWloaxcXF\nxMbGAleunHFzc2PAgAH4+vrSo0cPxo8fT+/evbFYLCQnJ+Pi4sLYsWOZNGkSs2bNokmTJkRGRlZ5\nB0VERKR6sxjVdL9wXl4e4eHhZGRkUL9+fbPLuaE7+Q6sjtJ3R6jTEWoEx6mzKtypfXeUfjtKnVXB\nEfp+o/d13eRDRERETKUwIiIiIqa6474oT0TuDL6+vpw+fbrSt1vZV0H4+Pjc1hMFRaojhRER+VU6\nffp0tT+GDo5xiadIVdNhGhERETGVwoiIiIiYSmFERERETKUwIiIiIqZSGBERERFTKYyIiIiIqXRp\nr4iI3FB2hwAud2thdhkVyu4QYHYJ8jMpjIiIyA2Fbs11iHu2hFosVP8q5Vp0mEZERERMpTAiIiIi\nplIYEREREVMpjIiIiIipFEZERETEVLqaRn4xR/jWUR8fH7NLEBGR61AYkV+kKi73s1gsDnEZoYiI\nVA6FEZFqRHuZROROpDAiUk1oL5OI3Kl0AquIiIiYqsI9I2VlZSQlJZGbm0uNGjWYPn06/v7+tvYN\nGzawZMkSPD09iY6OpmfPnly6dIkJEyZw7NgxSkpKGDx4MOHh4Rw4cIBBgwbRqFEjAHr37k3nzp2r\nrHMiIiJS/VUYRtLT0ykpKSE1NZXs7GxSUlJYuHAhAAUFBcyZM4d169bh5eVF//79ad26NZ999hne\n3t7MnDmTH374gW7duhEeHs7+/fsZMGAAiYmJVd4xERERcQwVhpGsrCzatm0LQGhoKDk5Oba2vLw8\nAgIC8Pb2BqB58+bs2bOHxx57jMjISODKcXBnZ2cAcnJyOHz4MBkZGfj7+zNhwgQ8PDwqvVMiIiLi\nOCo8Z6SwsNAuMDg7O1NaWgqAv78/hw4dIj8/n/Pnz7Nr1y6Ki4upVasWHh4eFBYWMnz4cEaOHAlA\nSEgIY8aMYcWKFTRo0ID58+dXUbdERETEUVS4Z8TDw4OioiLbdFlZGS4uV1arXbs248ePZ9iwYXh7\nexMUFGS77O/EiRMMGTKE+Ph4unTpAkBERAReXl6236dNm1bpHRIRERHHUuGekbCwMDIzMwHIzs7G\narXa2kpLSzlw4AArV67ktdde49tvvyUsLIz8/HwSExN54YUX6NGjh235gQMHsnfvXgB27dpFUFBQ\nZfdHREREHEyFe0YiIiLYsWMHcXFxGIZBcnIyaWlpFBcXExsbC0B0dDRubm4MGDAAX19fpk+fztmz\nZ1mwYAELFiwA4PXXXycpKYlp06bh6uqKn5+f9oyIiIgIFqOa3hEpLy+P8PBwMjIyqF+/vtnl3JBu\nLFW5NJ6V504eS0fpuyPU6Qg1guPUWRUcoe83el/XTc9ERETEVAojIiIiYiqFERERETGVwoiIiIiY\nSmFERERETKUwIiIiIqZSGBERERFTKYyIiIiIqRRGRERExFQKIyIiImKqCr+bRkTEEWV3COBytxZm\nl1Gh7A4BZpcgYjqFERH5VQrdmlvtv6sDINRiofpXKVK1dJhGRERETKUwIiIiIqZSGBERERFTKYyI\niIiIqRRGRERExFQKIyIiImIqXdorIiIVslgsZpdQIR8fH7NLkJ9JYURERG6oKu7XYrFYHOI+MHJ7\n6DCNiIiImEphRERERExV4WGasrIykpKSyM3NpUaNGkyfPh1/f39b+4YNG1iyZAmenp5ER0fTs2fP\n665z5MgRxo0bh8VioWnTpkyZMgUnJ+UhERGRO1mFSSA9PZ2SkhJSU1MZNWoUKSkptraCggLmzJnD\n8uXLefvtt0lLSyMvL++668yYMYORI0eycuVKDMMgIyOj6nomIiIiDqHCMJKVlUXbtm0BCA0NJScn\nx9aWl5dHQEAA3t7eODk50bx5c/bs2XPddfbv30+rVq0AaNeuHTt37qz0DomIiIhjqTCMFBYW4uHh\nYZt2dnaN9AluAAAVDklEQVSmtLQUAH9/fw4dOkR+fj7nz59n165dFBcXX3cdwzBsl4fVqlWLc+fO\nVXZ/KlVwcDAWi6XCH+CmlrNYLAQHB5vcK3Pc7FjeynjeqWMJem2K3AnupP+bFZ4z4uHhQVFRkW26\nrKwMF5crq9WuXZvx48czbNgwvL29CQoKwsfH57rr/Pj8kKKiIry8vCqzL5Xux3uB5JfRWFYujafI\nr9+d9Hde4Z6RsLAwMjMzAcjOzsZqtdraSktLOXDgACtXruS1117j22+/JSws7LrrBAYG8tlnnwGQ\nmZnJfffdV+kdEhEREcdS4Z6RiIgIduzYQVxcHIZhkJycTFpaGsXFxcTGxgIQHR2Nm5sbAwYMwNfX\n95rrAIwdO5ZJkyYxa9YsmjRpQmRkZNX2TkRERKo9i1FNb4GXl5dHeHg4GRkZ1K9f3+xyRMTBOMod\nPh2lzsp2p/b7Tnaj93Xd5ENERERMpTAiIiIiplIYEREREVMpjIiIiIipFEZERETEVAojIiIiYiqF\nERERETGVwoiIiIiYSmFERERETKUwIiIiIqZSGBERERFTKYyIiIiIqRRGRERExFQKIyIiImIqhRER\nERExlcKIiIiImEphREREREylMCIiIiKmcjG7ABGRqmKxWMwuoUI+Pj5mlyBiOoUREflVMgyj0rdp\nsViqZLsidzodphERERFTKYyIiIiIqRRGRERExFQVnjNSVlZGUlISubm51KhRg+nTp+Pv729r37hx\nI8uWLcPJyYmYmBji4+NZt24d69evB+DixYt89dVX7Nixg7y8PAYNGkSjRo0A6N27N507d66anomI\niIhDqDCMpKenU1JSQmpqKtnZ2aSkpLBw4UJb+8svv8ymTZuoWbMmUVFRREVF0b17d7p37w7A1KlT\niYmJwcvLi/379zNgwAASExOrrkciIiLiUCo8TJOVlUXbtm0BCA0NJScnx649ICCAc+fOUVJSgmEY\ndpfS7du3j0OHDhEbGwtATk4O27Zto0+fPkyYMIHCwsLK7IuIiIg4oArDSGFhIR4eHrZpZ2dnSktL\nbdNNmzYlJiaGqKgo2rdvj5eXl61t0aJFDBkyxDYdEhLCmDFjWLFiBQ0aNGD+/PmV1Q8RERFxUBWG\nEQ8PD4qKimzTZWVluLhcObpz8OBBtm3bRkZGBlu2bKGgoID3338fgLNnz3L48GEefPBB27oREREE\nBwfbfj9w4ECldkZEREQcT4VhJCwsjMzMTACys7OxWq22Nk9PT9zd3XFzc8PZ2RlfX1/Onj0LwOef\nf07r1q3ttjVw4ED27t0LwK5duwgKCqq0joiIiIhjqvAE1oiICHbs2EFcXByGYZCcnExaWhrFxcXE\nxsYSGxtLfHw8rq6uNGzYkOjoaAAOHz5M/fr17baVlJTEtGnTcHV1xc/Pj2nTplVNr0RExBTBwcHs\n37//ppa92dv1BwUFlTtfUX5dLEY1vbdxXl4e4eHhZGRklAs1IiJm0O3gRX6+G72v66ZnIiIiYiqF\nERERETGVwoiIiIiYSmFERERETKUwIiIiIqZSGBERERFTKYyIiIiIqRRGRERExFQKIyIiImIqhRER\nERExlcKIiIiImEphREREREylMCIiIiKmUhgRERERUymMiIiIiKkURkRERMRUCiMiIiJiKoURERER\nMZXCiIiIiJhKYURERERMpTAiIiIiplIYEREREVO5VLRAWVkZSUlJ5ObmUqNGDaZPn46/v7+tfePG\njSxbtgwnJydiYmKIj48HIDo6Gg8PDwDq16/PjBkzOHLkCOPGjcNisdC0aVOmTJmCk5PykIiIyJ2s\nwjCSnp5OSUkJqampZGdnk5KSwsKFC23tL7/8Mps2baJmzZpERUURFRWFu7s7hmGwfPlyu23NmDGD\nkSNH8sADDzB58mQyMjKIiIio/F6JiIiIw6hwt0RWVhZt27YFIDQ0lJycHLv2gIAAzp07R0lJCYZh\nYLFYOHjwIOfPnycxMZF+/fqRnZ0NwP79+2nVqhUA7dq1Y+fOnZXdHxEREXEwFe4ZKSwstB1uAXB2\ndqa0tBQXlyurNm3alJiYGO666y4iIiLw8vLC3d2dgQMH0rNnT7777juefvppPvjgA1tYAahVqxbn\nzp2rom6JiIiIo6hwz4iHhwdFRUW26bKyMlsQOXjwINu2bSMjI4MtW7ZQUFDA+++/T+PGjenatSsW\ni4XGjRvj7e3NqVOn7M4PKSoqwsvLqwq6JCIiIo6kwjASFhZGZmYmANnZ2VitVlubp6cn7u7uuLm5\n4ezsjK+vL2fPnmXNmjWkpKQAcPLkSQoLC6lTpw6BgYF89tlnAGRmZnLfffdVRZ9ERETEgVR4mCYi\nIoIdO3YQFxeHYRgkJyeTlpZGcXExsbGxxMbGEh8fj6urKw0bNiQ6OhqA8ePH07t3bywWC8nJybi4\nuDB27FgmTZrErFmzaNKkCZGRkVXeQREREaneLIZhGGYXcS15eXmEh4eTkZFB/fr1zS5HRASLxUI1\n/ZcpUu3d6H1dN/kQERERUymMiIiIiKkURkRERMRUCiMiIiJiKoURERERMZXCiIiIiJhKYURERERM\npTAiIiIiplIYEREREVMpjIiIiIipFEZERETEVAojIiIiYiqFERERETGVwoiIiIiYSmFERERETKUw\nIiIiIqZSGBERERFTKYyIiIiIqRRGRERExFQKIyIiImIqhRERERExlcKIiIiImMqlogXKyspISkoi\nNzeXGjVqMH36dPz9/W3tGzduZNmyZTg5ORETE0N8fDyXLl1iwoQJHDt2jJKSEgYPHkx4eDgHDhxg\n0KBBNGrUCIDevXvTuXPnKuuciIiIVH8VhpH09HRKSkpITU0lOzublJQUFi5caGt/+eWX2bRpEzVr\n1iQqKoqoqCjS09Px9vZm5syZ/PDDD3Tr1o3w8HD279/PgAEDSExMrNJOiYiIiOOoMIxkZWXRtm1b\nAEJDQ8nJybFrDwgI4Ny5c7i4uGAYBhaLhccee4zIyEgADMPA2dkZgJycHA4fPkxGRgb+/v5MmDAB\nDw+Pyu6TiIiIOJAKzxkpLCy0CwzOzs6Ulpbapps2bUpMTAxRUVG0b98eLy8vatWqhYeHB4WFhQwf\nPpyRI0cCEBISwpgxY1ixYgUNGjRg/vz5VdAlEZGbFxwcjMViuakf4KaWCw4ONrlXIo6lwjDi4eFB\nUVGRbbqsrAwXlys7VA4ePMi2bdvIyMhgy5YtFBQU8P777wNw4sQJ+vXrxxNPPEGXLl0AiIiIsP2R\nRkREcODAgUrvkIjIrcjJycEwjEr9+ekeZBG5sQrDSFhYGJmZmQBkZ2djtVptbZ6enri7u+Pm5oaz\nszO+vr6cPXuW/Px8EhMTeeGFF+jRo4dt+YEDB7J3714Adu3aRVBQUGX3R0RERBxMheeMREREsGPH\nDuLi4jAMg+TkZNLS0iguLiY2NpbY2Fji4+NxdXWlYcOGREdH8/LLL3P27FkWLFjAggULAHj99ddJ\nSkpi2rRpuLq64ufnx7Rp06q8gyIiIlK9WQzDMMwu4lry8vIIDw8nIyOD+vXrm12OiIiI/AI3el/X\nTc9ERETEVAojIiIiYiqFERERETGVwoiIiIiYSmFERERETFXhpb1muXz5MgD//e9/Ta5EREREfqmr\n7+dX399/rNqGkVOnTgHQp08fkysRERGRynLq1Cn8/f3t5lXb+4xcuHCBnJwc6tSpY/uiPREREXFM\nly9f5tSpUwQHB+Pu7m7XVm3DiIiIiNwZdAKriIiImEphREREREylMCIiIiKmUhgRERERU92RYWTx\n4sX079+fvn37kpCQQE5Ozk2vm5eXR69evW5q2YsXL9KxY0cAxo0bR2ZmZrlthYWFkZCQQEJCAr16\n9aJ///6cOXPm5jtjguo4fn379qVXr168/fbbdsvs3buX4OBg9u7de9M13k5mj2VpaSkjR44kKSkJ\nwzAICAggPT3dtl5mZibjxo0DICEhgeTk5Gtu0yxmjB9Aamoqffr0ISEhgbi4OD777DO75Z999lkG\nDRpkN69jx462deLj4+nWrRv79u0jNzfX9j+gefPmtmW2bdt20325Hcx6rQ4dOtSuvU2bNgCsW7eO\n9u3b28azb9++HDt27KZrup3MGLuCggKGDRtGYmIicXFxTJw4kQsXLjBnzhxee+01u/U++ugjRo8e\nzbp16wgICCA7O9vWdunSJR544AHmzp170zX/HNX2PiNV5dChQ2zZsoV//OMfWCwWvvrqK8aOHcvG\njRtNqecPf/gDy5cvt02/+uqrrFmzhoEDB5pST0Wq8/hdunSJIUOG8Lvf/c72B7l69WoGDBjAypUr\nCQkJMaXG6zF7LC9dusRzzz1Ho0aNGD16NAB33XUXKSkphIWF4evrW26dzZs306lTJ1q1anVbarwR\ns8Zv8+bN7NixgzfffBNXV1eOHj1K3759Wb9+Pb6+vhw/fpzi4mJKS0s5evQoDRo0sK27dOlS3Nzc\nAPjkk0+YN28eixYtsr2GO3bsaLdMdWHmazUrK4sNGzbQrVu3cm2PP/647bWbmprKkiVLmDx5cpXX\ndCvMGrs33niDhx56iN69ewPw4osvsmrVKmJiYujfvz/Dhw/HYrEAsHbtWgYOHMjRo0dp0qQJmzdv\nJjQ0FLjyOvX09KzSWuEODCOenp4cP36cNWvW0K5dO5o1a8aaNWvYs2cPycnJlJWVUbduXV555RX2\n7t3LvHnzMAyDoqIiXn31VVxdXW3b2r17N7Nnz8bZ2ZkGDRrwl7/8hZKSEkaPHs3Zs2dp2LDhLdVm\nGAYnTpy45fVup+o8fq6urvTr148NGzbQsWNHioqK+PTTT9m8eTNdunShoKDgmm+wZjFzLEtKShg2\nbBjBwcF2nzxr1arFgAEDSEpKYs6cOeVqnjhxIpMmTWLdunW4uJj778Os8Vu1ahXjx4+3rd+gQQM2\nbNiAj48PcOUfe3h4OO7u7qxcuZKxY8des/7jx4/j5eVVhSNUecx8rT7//PPMnTuXBx98kHr16l23\nxjNnzlSrv++rzBo7Pz8//vnPf+Lv709YWBhjx47FYrHg7OyMv78/X3zxBffffz+nTp3i2LFj3H//\n/Rw9epR27dqxfft2ysrKcHJyYvPmzURFRVX5ON1xh2nq1q3LwoUL+de//kVsbCyPPfYYW7duZfLk\nySQnJ/POO+/wyCOP8M033/D1118zc+ZMli9fzqOPPsoHH3xg245hGEyaNIl58+bx9ttvU7duXdav\nX8+qVauwWq2sWLGCuLi4Cus5dOgQCQkJdOnShcjISPz9/YmOjq7KIfhFqtv4/ZSfnx+nT58G4L33\n3iMiIgI3Nzf+9Kc/sWbNmkobh8pg5li++OKLFBcXc/LkyXJ1xcfHU1hYSFpaWrm2gIAAunXrRkpK\nSuUPyC0ya/y+//57u70dgC2IlJWVsWnTJp544gmioqJ47733uHDhgm25xMREevToQbt27di7d+91\ng0p1Y+ZrtW7duowYMYKJEyeWq2vTpk0kJCTQvXt3Fi9eTKdOnap8LG6VWWPXv39/Hn/8cZYsWULb\ntm0ZOnQo33//PQC9evXi3XffBWDDhg3ExMTY1nN1dSU0NJTdu3dTWFhIYWHhDUNgZbnj9owcOXIE\nDw8PZsyYAcC+fft4+umnKSws5J577gGgZ8+eAJw4cYIXX3yRmjVrcvLkScLCwmzbKSgo4Pvvv2fk\nyJHAlTvGPvTQQxQUFPDII48AcO+991b46fHqYYYLFy7w7LPPcvfdd5v+ifNGqtv4/dSxY8dsfzjv\nvPMOzs7ODBw4kAsXLvDf//6Xp556Cien6pHBzRzLvn378uSTT9KnTx82btxI165dbW0Wi4Xk5GT6\n9OnD4MGDy9X9zDPP0Lt373Ln8NxuZo3f73//e06cOGG36/qTTz4hICCAr776iqKiIkaNGgVcCSdp\naWm2Oq4egpk1axZ5eXncfffdVTlElcbsv/uuXbuSnp7OypUr7eb/+DDNrl27GDZsGB999FEVjMDP\nZ9bYffrpp3Tr1o0ePXpQUlLC66+/TnJyMnPnzqVDhw7MmjWLixcvsnnzZt588027mh9//HE2b97M\niRMniIiI4NKlS1U6RnAHhpHc3FxSU1NZuHAhNWrUoHHjxnh5eVGvXj2+++47GjVqxOLFi2ncuDGT\nJ0/mo48+wsPDg7Fjx/Ljm9X6+PhQr149FixYgKenJxkZGdSsWZPc3Fyys7Pp1KkTBw4coLS09Kbq\ncnd355VXXqFbt26EhYXxxz/+saqG4BepruMHVw49vPXWWwwaNIjc3FwuX77M6tWrbe0DBgxg69at\nhIeHV+qY/FxmjmXTpk1xcXHhlVdeoXfv3gQFBdn+MQLUq1ePYcOG8dJLL9n+0V3l7OxMSkoKTz31\nVNUP0g2YNX4xMTEsWLCAV155BRcXFw4fPsyf//xn1q1bx5o1a5g+fTrt27cHrpzvMH36dNubzVUj\nR46kX79+rFy50iG+f6s6/N0nJSXRq1cvioqKrlnjb3/729vypnmrzBq7t956i++//55u3bpRo0YN\nmjZtyrfffgtc2fvRqVMnFixYwD333IO3t7ddzQ888ADJycl8//33vPrqq9fcS1rZ7rgw8uijj/LN\nN9/Qo0cPatasiWEYjBkzht/85jdMmDABJycn6tSpQ//+/enatSt9+vThrrvuws/Pz7aLC8DJyYmJ\nEyfyzDPPYBgGtWrV4uWXXyYsLIwxY8bQu3dvmjRpYne878UXX+Svf/0rAI0bN+a5556zq83Pz48x\nY8YwefJkVq1aVW0+wf9YdRu/q4e5LBYLpaWldOnShYceeojp06fzxBNP2NXes2dPVqxYUW3CiJlj\neVWDBg144YUXGDFiBO+8845dW7du3a77KbNJkyY8+eST/P3vf6/cQbkFZo1fVFQUp06dIj4+HldX\nVy5fvszMmTMxDIM9e/Ywe/Zs27ZbtmzJxYsX+de//mVXu5OTE9OnT6dv37506tSJunXr3p5B+5mq\nw2vV19eXcePGMWTIENu8TZs2sWfPHpydnSkqKmLq1Km3ZTxuhVljN3XqVKZOncqbb76Ju7s7Pj4+\nJCUl2bbXs2dPoqKiWLp0abmanZycaNOmDSdOnMDDw6PKxwj03TQiIiJisur30VtERETuKAojIiIi\nYiqFERERETGVwoiIiIiYSmFERERETKUwIiIiIqZSGBERERFTKYyIiIiIqf4fK2e0O5Gdm1gAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11274ca90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compare Algorithms\n",
    "fig = plt.figure()\n",
    "fig.suptitle( 'Scaled Algorithm Comparison' )\n",
    "ax = fig.add_subplot(111)\n",
    "plt.boxplot(results)\n",
    "ax.set_xticklabels(names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observations\n",
    "> The results show that standardization of the data has lifted the skill of SVM to be the most accurate algorithm tested so far.\n",
    "\n",
    "The results suggest digging deeper into the SVM and LDA and LR algorithms. It is very likely that configuration beyond the default may yield even more accurate models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Algorithm Tuning\n",
    "In this section we investigate tuning the parameters for three algorithms that show promise from the spot-checking in the previous section: LR, LDA and SVM.\n",
    "\n",
    "### Tuning hyper-parameters - SVC estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Model Training Accuracy: 0.925 +/- 0.041\n",
      "('--> Tuned Parameters Best Score: ', 0.94472361809045224)\n",
      "('--> Best Parameters: \\n', {'clf__gamma': 0.001, 'clf__C': 100.0, 'clf__kernel': 'rbf'})\n"
     ]
    }
   ],
   "source": [
    "#Make Support Vector Classifier Pipeline\n",
    "pipe_svc = Pipeline([('scl', StandardScaler()),\n",
    "                     ('pca', PCA(n_components=2)),\n",
    "                     ('clf', SVC(probability=True, verbose=False))])\n",
    "\n",
    "#Fit Pipeline to training Data\n",
    "pipe_svc.fit(X_train, y_train)\n",
    "\n",
    "#print('--> Fitted Pipeline to training Data')\n",
    "\n",
    "scores = cross_val_score(estimator=pipe_svc, X=X_train, y=y_train, cv=10, n_jobs=1, verbose=0)\n",
    "print('--> Model Training Accuracy: %.3f +/- %.3f' %(np.mean(scores), np.std(scores)))\n",
    "\n",
    "#Tune Hyperparameters\n",
    "param_range = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]\n",
    "param_grid = [{'clf__C': param_range,'clf__kernel': ['linear']},\n",
    "              {'clf__C': param_range,'clf__gamma': param_range,\n",
    "               'clf__kernel': ['rbf']}]\n",
    "gs_svc = GridSearchCV(estimator=pipe_svc,\n",
    "                  param_grid=param_grid,\n",
    "                  scoring='accuracy',\n",
    "                  cv=10,\n",
    "                  n_jobs=1)\n",
    "gs_svc = gs_svc.fit(X_train, y_train)\n",
    "print('--> Tuned Parameters Best Score: ',gs.best_score_)\n",
    "print('--> Best Parameters: \\n',gs.best_params_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning the hyper-parameters - k-NN hyperparameters\n",
    " For your standard k-NN implementation, there are two primary hyperparameters that you’ll want to tune:\n",
    "\n",
    "* The number of neighbors k.\n",
    "* The distance metric/similarity function.\n",
    "\n",
    "Both of these values can dramatically affect the accuracy of your k-NN classifier. Grid object is ready to do 10-fold cross validation on a KNN model using classification accuracy as the evaluation metric\n",
    "In addition, there is a parameter grid to repeat the 10-fold cross validation process 30 times\n",
    "Each time, the n_neighbors parameter should be given a different value from the list\n",
    "We can't give GridSearchCV just a list\n",
    "We've to specify n_neighbors should take on 1 through 30\n",
    "You can set n_jobs = -1 to run computations in parallel (if supported by your computer and OS) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Model Training Accuracy: 0.927 +/- 0.045\n",
      "('--> Tuned Parameters Best Score: ', 0.94472361809045224)\n",
      "('--> Best Parameters: \\n', {'clf__gamma': 0.001, 'clf__C': 100.0, 'clf__kernel': 'rbf'})\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
    "\n",
    "pipe_knn = Pipeline([('scl', StandardScaler()),\n",
    "                     ('pca', PCA(n_components=2)),\n",
    "                     ('clf', KNeighborsClassifier())])\n",
    "            \n",
    "#Fit Pipeline to training Data\n",
    "pipe_knn.fit(X_train, y_train) \n",
    "\n",
    "scores = cross_val_score(estimator=pipe_knn, \n",
    "                         X=X_train, \n",
    "                         y=y_train, \n",
    "                         cv=10,\n",
    "                         n_jobs=1)\n",
    "print('--> Model Training Accuracy: %.3f +/- %.3f' %(np.mean(scores), np.std(scores)))\n",
    "\n",
    "#Tune Hyperparameters\n",
    "param_range = range(1, 31)\n",
    "param_grid = [{'clf__n_neighbors': param_range}]\n",
    "# instantiate the grid\n",
    "grid = GridSearchCV(estimator=pipe_knn, \n",
    "                    param_grid=param_grid, \n",
    "                    cv=10, \n",
    "                    scoring='accuracy')\n",
    "gs_knn = grid.fit(X_train, y_train)\n",
    "print('--> Tuned Parameters Best Score: ',gs.best_score_)\n",
    "print('--> Best Parameters: \\n',gs.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finalize Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Final Model Training Accuracy: 0.944 +/- 0.032\n",
      "--> Final Accuracy on Test set: 0.94737\n"
     ]
    }
   ],
   "source": [
    "#Use best parameters\n",
    "clf_svc = gs.best_estimator_\n",
    "\n",
    "#Get Final Scores\n",
    "clf_svc.fit(X_train, y_train)\n",
    "scores = cross_val_score(estimator=clf_svc,\n",
    "                         X=X_train,\n",
    "                         y=y_train,\n",
    "                         cv=10,\n",
    "                         n_jobs=1)\n",
    "print('--> Final Model Training Accuracy: %.3f +/- %.3f' %(np.mean(scores), np.std(scores)))\n",
    "\n",
    "print('--> Final Accuracy on Test set: %.5f' % clf_svc.score(X_test,y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.947368421053\n",
      "[[113   3]\n",
      " [  6  49]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          B       0.95      0.97      0.96       116\n",
      "          M       0.94      0.89      0.92        55\n",
      "\n",
      "avg / total       0.95      0.95      0.95       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf_svc.fit(X_train, y_train)\n",
    "y_pred = clf_svc.predict(X_test)\n",
    "\n",
    "print(accuracy_score(y_test, y_pred))\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "Worked through a classification predictive modeling machine learning problem from end-to-end using Python. Specifically, the steps covered were:\n",
    "1. Problem Definition (Breast Cancer data).\n",
    "2. Loading the Dataset.\n",
    "3. Analyze Data (same scale but di↵erent distributions of data).\n",
    "    * Evaluate Algorithms (KNN looked good).\n",
    "    * Evaluate Algorithms with Standardization (KNN and SVM looked good).\n",
    "4. Algorithm Tuning (K=19 for KNN was good, SVM with an RBF kernel and C=100 was best).. \n",
    "5. Finalize Model (use all training data and confirm using validation dataset)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  },
  "nbpresent": {
   "slides": {
    "24b4c65f-bb0a-40e5-82d1-e7a32fd7cb81": {
     "id": "24b4c65f-bb0a-40e5-82d1-e7a32fd7cb81",
     "prev": "c916bff9-01b0-4f88-8dfe-48366cbcebbd",
     "regions": {
      "96cec8ec-f802-4388-92b6-3575ae6d35fc": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "57b6e52c-2e81-423d-9902-e57a54fadab3",
        "part": "whole"
       },
       "id": "96cec8ec-f802-4388-92b6-3575ae6d35fc"
      }
     }
    },
    "3fe8f6e5-8355-49bd-8b72-54cc03842fdd": {
     "id": "3fe8f6e5-8355-49bd-8b72-54cc03842fdd",
     "prev": "24b4c65f-bb0a-40e5-82d1-e7a32fd7cb81",
     "regions": {
      "15ef1e17-3829-4deb-a95f-49af770554e4": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "7c2600b8-f656-4a31-8fff-826146198290",
        "part": "whole"
       },
       "id": "15ef1e17-3829-4deb-a95f-49af770554e4"
      },
      "26043033-eeee-4f5f-9f64-94e8af33b4ad": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "7b8e7745-abd5-4313-b1d9-a571faf34d6f",
        "part": "whole"
       },
       "id": "26043033-eeee-4f5f-9f64-94e8af33b4ad"
      },
      "32421559-c70d-4046-9ffb-e3fe7750d949": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "a1cd24b6-d66a-4452-a9b6-1feecc23ef3b",
        "part": "whole"
       },
       "id": "32421559-c70d-4046-9ffb-e3fe7750d949"
      },
      "6b5e1491-f4a8-4cb5-8a9a-653689ecf24f": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "2db6cf7b-a8d0-4f65-b9ba-f833f21849fd",
        "part": "whole"
       },
       "id": "6b5e1491-f4a8-4cb5-8a9a-653689ecf24f"
      },
      "82991730-4d08-4637-8391-26e15b890186": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "226eb66f-3bbc-4911-ae3e-3a7953db372a",
        "part": "whole"
       },
       "id": "82991730-4d08-4637-8391-26e15b890186"
      },
      "9c7c33ce-2582-4e10-a1b8-c334961967b6": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "aaff86c0-1ec6-4fda-bdc5-6f1fad2d1bed",
        "part": "whole"
       },
       "id": "9c7c33ce-2582-4e10-a1b8-c334961967b6"
      },
      "b852c892-5a54-4981-94b4-335b73256228": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "9e64f8b8-8d22-4b11-b893-21e18aca1896",
        "part": "whole"
       },
       "id": "b852c892-5a54-4981-94b4-335b73256228"
      },
      "bc2c634a-7f01-458a-875d-f2d9490c5d7f": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "d5d5f892-f456-4167-beca-f90a955c41b8",
        "part": "whole"
       },
       "id": "bc2c634a-7f01-458a-875d-f2d9490c5d7f"
      },
      "bee14e73-b23f-4351-949d-127e5e7725a1": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "f57a9cbe-11fb-48ec-b081-046da330c180",
        "part": "whole"
       },
       "id": "bee14e73-b23f-4351-949d-127e5e7725a1"
      },
      "c15582a0-2c07-4410-b1b3-8f6941367456": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "8b6fc11b-bdf0-4429-858b-594d9d08ffe8",
        "part": "whole"
       },
       "id": "c15582a0-2c07-4410-b1b3-8f6941367456"
      },
      "cbd8001c-ce07-4722-aed4-db80545c6aad": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "b6901fc4-1011-4b0b-a585-b408c98dccc5",
        "part": "whole"
       },
       "id": "cbd8001c-ce07-4722-aed4-db80545c6aad"
      },
      "d174d0a5-e24c-47a2-bcb3-12b795cd4d34": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "13a04041-12c5-40d9-aa3e-3c31b7c6700f",
        "part": "whole"
       },
       "id": "d174d0a5-e24c-47a2-bcb3-12b795cd4d34"
      },
      "f979a251-a052-4653-8dc4-76989d6d5564": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "4d43cfde-2f01-48e6-ae90-86aa87ed0658",
        "part": "whole"
       },
       "id": "f979a251-a052-4653-8dc4-76989d6d5564"
      },
      "fc37e2ef-dace-4e14-bd2c-d2f1720fb59b": {
       "attrs": {
        "height": 0.4,
        "width": 0.8,
        "x": 0.1,
        "y": 0.5
       },
       "content": {
        "cell": "f5fca3f9-acc2-4cba-9ceb-3eb63d0c6336",
        "part": "whole"
       },
       "id": "fc37e2ef-dace-4e14-bd2c-d2f1720fb59b"
      }
     }
    },
    "c916bff9-01b0-4f88-8dfe-48366cbcebbd": {
     "id": "c916bff9-01b0-4f88-8dfe-48366cbcebbd",
     "prev": null,
     "regions": {
      "b30819be-7928-4220-b15f-b3ca846f4850": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "b1daa992-4a28-496f-b325-6e8be1a7844c",
        "part": "whole"
       },
       "id": "b30819be-7928-4220-b15f-b3ca846f4850"
      }
     }
    }
   },
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}