solutions/ch_10/exercise_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiclass Voting Classifier to Predict Wine Quality Score\n",
    "\n",
    "## Wine Data\n",
    "Data from http://archive.ics.uci.edu/ml/datasets/Wine+Quality\n",
    "\n",
    "### Citations\n",
    "<pre>\n",
    "Dua, D. and Karra Taniskidou, E. (2017). \n",
    "UCI Machine Learning Repository [http://archive.ics.uci.edu/ml/index.php]. \n",
    "Irvine, CA: University of California, School of Information and Computer Science.\n",
    "</pre>\n",
    "\n",
    "<pre>\n",
    "P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. \n",
    "Modeling wine preferences by data mining from physicochemical properties.\n",
    "In Decision Support Systems, Elsevier, 47(4):547-553. ISSN: 0167-9236.\n",
    "</pre>\n",
    "\n",
    "Available at:\n",
    "- [@Elsevier](http://dx.doi.org/10.1016/j.dss.2009.05.016)\n",
    "- [Pre-press (pdf)](http://www3.dsi.uminho.pt/pcortez/winequality09.pdf)\n",
    "- [bib](http://www3.dsi.uminho.pt/pcortez/dss09.bib)\n",
    "\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "red_wine = pd.read_csv('../../ch_10/data/winequality-red.csv')\n",
    "white_wine = pd.read_csv('../../ch_10/data/winequality-white.csv', sep=';')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed acidity</th>\n",
       "      <th>volatile acidity</th>\n",
       "      <th>citric acid</th>\n",
       "      <th>residual sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free sulfur dioxide</th>\n",
       "      <th>total sulfur dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>quality</th>\n",
       "      <th>kind</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>848</th>\n",
       "      <td>6.4</td>\n",
       "      <td>0.64</td>\n",
       "      <td>0.21</td>\n",
       "      <td>1.8</td>\n",
       "      <td>0.081</td>\n",
       "      <td>14.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.99689</td>\n",
       "      <td>3.59</td>\n",
       "      <td>0.66</td>\n",
       "      <td>9.8</td>\n",
       "      <td>5</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2529</th>\n",
       "      <td>6.6</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.13</td>\n",
       "      <td>12.8</td>\n",
       "      <td>0.044</td>\n",
       "      <td>26.0</td>\n",
       "      <td>158.0</td>\n",
       "      <td>0.99772</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.47</td>\n",
       "      <td>9.0</td>\n",
       "      <td>5</td>\n",
       "      <td>white</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>5.6</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.09</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0.049</td>\n",
       "      <td>17.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>0.99370</td>\n",
       "      <td>3.63</td>\n",
       "      <td>0.63</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>244</th>\n",
       "      <td>15.0</td>\n",
       "      <td>0.21</td>\n",
       "      <td>0.44</td>\n",
       "      <td>2.2</td>\n",
       "      <td>0.075</td>\n",
       "      <td>10.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.00005</td>\n",
       "      <td>3.07</td>\n",
       "      <td>0.84</td>\n",
       "      <td>9.2</td>\n",
       "      <td>7</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1551</th>\n",
       "      <td>6.6</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.99</td>\n",
       "      <td>1.2</td>\n",
       "      <td>0.122</td>\n",
       "      <td>45.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>0.99360</td>\n",
       "      <td>3.09</td>\n",
       "      <td>0.31</td>\n",
       "      <td>8.7</td>\n",
       "      <td>6</td>\n",
       "      <td>white</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
       "848             6.4              0.64         0.21             1.8      0.081   \n",
       "2529            6.6              0.42         0.13            12.8      0.044   \n",
       "131             5.6              0.50         0.09             2.3      0.049   \n",
       "244            15.0              0.21         0.44             2.2      0.075   \n",
       "1551            6.6              0.19         0.99             1.2      0.122   \n",
       "\n",
       "      free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
       "848                  14.0                  31.0  0.99689  3.59       0.66   \n",
       "2529                 26.0                 158.0  0.99772  3.24       0.47   \n",
       "131                  17.0                  99.0  0.99370  3.63       0.63   \n",
       "244                  10.0                  24.0  1.00005  3.07       0.84   \n",
       "1551                 45.0                 129.0  0.99360  3.09       0.31   \n",
       "\n",
       "      alcohol  quality   kind  \n",
       "848       9.8        5    red  \n",
       "2529      9.0        5  white  \n",
       "131      13.0        5    red  \n",
       "244       9.2        7    red  \n",
       "1551      8.7        6  white  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine = pd.concat([white_wine.assign(kind='white'), red_wine.assign(kind='red')])\n",
    "wine.sample(5, random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:title={'center':' Wine Quality Scores'}, xlabel='count of wines', ylabel='quality score'>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAscAAADgCAYAAADvwMuJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAq+0lEQVR4nO3deXhV1dn38e+dAQQHEMXKZJGCgECIEMGROiJFxKEIWKaKSAceq6UUFV6waPoUqShSFbC0gjiAw1NBpVEqINQyRQhhUGRKS5BaUIPDQQjJ/f5xNmkIGU4iJycJv891nSv7rL3W2vc+yxNvVtbe29wdERERERGBuFgHICIiIiJSVSg5FhEREREJKDkWEREREQkoORYRERERCSg5FhEREREJKDkWEREREQkoORYRAcxsgJm9Hes4KsLMZplZarB9uZltiXVMIiLVlZJjEamRzGyGmU0r9D7RzL4uoewid3/e3btHKZbaZvY7M/uXmR0ws61mNsrM7Hgfy92Xu3vrQsfOMrNrKtqfmd1hZh+a2Zdm9omZLTSzU49PtCIiVY+SYxGpqZYB3Qq9TwH+BVxepAzg/SjH8jJwNdATOBUYBPwEmBzl434rZvZ94H+B29z9VKAtMO84HyPhePYnIvJtKTkWkZpqGdDWzM4M3l8OzAVOLlK2wt1zzezHZvb3I43NzM3sp8Esb46ZPVl4ptfMhprZB2b2uZm9ZWbfLS4IM7sa6A780N03uvthd18JDATuNrMWQb2jZnjN7Ddm9lyh9y+b2b/NbL+ZLTOzdiUc7wozyw625wDnAK+b2VdmNtrM3jSzu4q0yTSzm4vp7sLg81kH4O6fuftsd/8yaFfHzCab2T+DuP5uZnWCfb3NbFPw2S01s7aFjpdlZveaWSbwtZklmNlFZvaPoP56M7uiUP0fm9mOYPZ6p5kNKO7cRUSOByXHIlIjufsu4J/8d6a4G7Ac+EeRsmWldNOLcIKYBPQFrgMwsxuBMcAtQMOg3xdL6ONaYFUQT+H4VgHZhGeUI/FXoBVwFrAWeL6sBu4+iPBs+Q3ufoq7TwJmE07MCc6lI9AEeLOYLlYB15nZBDO71MxqF9n/CNAZuARoAIwG8s3sPMKfxz2EP5+FhBP0WoXa3gZcD9QHvhMcPzXoZxTwqpk1NLOTganAD4LZ60uAjLLOXUSkopQci0hN9i7QzczigC7ASsKJ7JGyS4M6JZno7jnu/i9gCZAclP8U+J27f+DuhwkvPUguYfb4TGBPCf3vIZw8lsnd/+zuX7r7QeA3QEczqxdJ2yIWAOeZWavg/SBgnrsfKuaYywn/A6AT4eT1UzN71Mzig89vKHC3u+929zx3/0cQXz/gTXdf5O65hJPoOoQT2yOmuvsudz9AOFlf6O4L3T3f3RcB6YSXoQDkA+3NrI6773H3TRU4bxGRiCg5FpGa7Mi64w7ADncPAX8vVFaH8OxoSf5daDsEnBJsfxd4PFgCkAN8BhjhGdii9gGNSui/UbC/VEEyOtHMtpvZF0BWsOvMUpoVy92/IbxueGCQ4N4GzCml/l/d/QbCM7o3Aj8GhgXHPgnYXkyzxoRn7Y/0kQ/s4ujPp/BM+neBW498nsFnehnQyN2/Jpxs/xTYEywLaVOukxYRKQclxyJSky0DOhL+8/3yoGwT0CwoWxMki+W1C/iJu9cv9Krj7v8opu7fgK5m1qxwoZl1Jbwe+MjM9ddA3UJVzi60/SPCiek1QD2g+ZFuIojViymbDQwgvKQj5O4ryuwkPKP7DrAYaE84qf8G+F4x1T8mnPCGgwyv1W4G7C4hrl3AnCKf58nuPjE49lvufi3hf0x8CPyxrHhFRCpKybGI1Fjuvg34BLibIDl2dyc8W3w3pa83Ls104P4jF8WZWT0zu7WEGP4GvEN4DW27YBb4IuA54Fl3P3JP4gygf3B7uRSgT6FuTgUOAp8STqD/txyxfgK0KBLTCsJLFSZTyqyxmd1oZv3N7HQL6wJ8H1gZzAb/GXjUzBoH53VxsC75JeB6M7vazBKBXwXxF/ePBwh/FjeY2XVBPycFFxY2NbPvBHGcHPTxVRC7iEhUKDkWkZpuGeF1ve8VKltO+MK2CiXH7v4X4GFgbrDMYSPwg1Ka/JDwmuU0wrOtK4Lt4YXqjCM8C/s5MAF4odC+ZwkvU9gNbCa8djpSvwP+X7BcYVSRPjsQTkxL8jlwJ7AV+CKo+3t3P3Ix4ChgA7CG8NKSh4G4IOEfCPyB8AzzDYQvCjxmXTMUXDx55CLHvYRnkn9N+P9RccBIwrPRnxFOzn9WjvMXESkXC0+iiIhIZTGz2YTX5V5fUsJYCTEMBoa7+2WxOL6ISFWlmWMRkco3jPBa5E6xOLiZ1QV+Djwdi+OLiFRlmjkWETmBmNl1wP8RTs5/GNyKTkREAkqORUREREQCWlYhIiIiIhJQciwiIiIiEkiIdQCF9ejRw9PS0mIdhoiIiIjUbCU+RKlKzRzv21fmU1RFRERERKKmSiXHIiIiIiKxpORYRERERCSg5FhEREREJKDkWEREREQkUKUeAlK7UStvNGTKMeVZE6+v/GBEREREpKaqHnerKEtaWhqtW7emZcuWTJw48Zj9s2bNomHDhiQnJ5OcnMzMmTMB2LJlC507dyYpKYkVK1YAcPjwYa655hpCoVClnoOIiIiIVF1Ru8+xmZ0ELANqB8d5xd0fqGh/eXl5jBgxgkWLFtG0aVMuvPBCevfuzfnnn39UvX79+vHEE08cVTZjxgwef/xxmjdvzt13382rr77KtGnTGDhwIHXr1q1oSCIiIiJSw0Rz5vggcJW7dwSSgR5mdlFFO1u9ejUtW7akRYsW1KpVi/79+zN//vyI2iYmJhIKhQiFQiQmJpKTk8Prr7/O4MGDKxqOiIiIiNRAUZs59vBi5q+Ct4nBq8ILnHfv3k2zZs0K3jdt2pRVq1YdU+/VV19l2bJlnHfeeTz22GM0a9aMESNGMHjwYA4ePMiMGTN46KGHGDNmDHFx1WpViYiIiIhEWVSzQzOLN7MM4D/AInc/Jps1s+Fmlm5m6Xmh/d/qeDfccANZWVlkZmZy7bXXMmTIEADOOeccli5dyooVK6hbty7Z2dm0bduWQYMG0a9fPz766KNvdVwRERERqRmimhy7e567JwNNgS5m1r6YOk+7e4q7p8TXrVdiX02aNGHXrl0F77Ozs2nSpMlRdc444wxq164NwLBhw3j//feP6Wfs2LGkpqYydepUhg0bxqRJk5gwYULFTlBEREREapRKWVfg7jnAEqBHRfu48MIL2bp1Kzt37uTQoUPMnTuX3r17H1Vnz549BdsLFiygbdu2R+1/9913ady4Ma1atSIUChEXF0dcXJzuWCEiIiIiQHTvVtEQyHX3HDOrA1wLPFzR/hISEnjiiSe47rrryMvLY+jQobRr147x48eTkpJC7969mTp1KgsWLCAhIYEGDRowa9asgvbuTmpqKvPmzQNg+PDhDBgwgMOHDzNt2rRvd7IiIiIiUiNE7SEgZpYEzAbiCc9Qv+TuD5bWRg8BEREREZFKUOJDQKJ5t4pM4IJo9S8iIiIicrxVqcdHp6SkeHp6eqzDEBEREZGarWY8PlpEREREJJqUHIuIiIiIBJQci4iIiIgElByLiIiIiASUHIuIiIiIBJQci4iIiIgElByLiIiIiASUHIuIiIiIBJQci4iIiIgElByLiIiIiASUHIuIiIiIBMzdYx1DgdqNWnmjIVNiHUaFZU28PtYhiIiIiEjZrKQdmjk+zr755hu6dOlCx44dadeuHQ888MAxdaZPn06HDh1ITk7msssuY/PmzQC89957JCUlkZKSwtatWwHIycmhe/fu5OfnV+p5iIiIiJyIop4cm1m8ma0zszeifayqoHbt2ixevJj169eTkZFBWloaK1euPKrOj370IzZs2EBGRgajR49m5MiRAEyePJmFCxcyZcoUpk+fDkBqaipjxowhLk7/jhERERGJtsrIuO4GPqiE41QJZsYpp5wCQG5uLrm5uZgdPXN/2mmnFWx//fXXBfsTExMJhUKEQiESExPZvn07u3bt4oorrqi0+EVEREROZAnR7NzMmgLXA78FRkbzWFVJXl4enTt3Ztu2bYwYMYKuXbseU+fJJ5/k0Ucf5dChQyxevBiA+++/n8GDB1OnTh3mzJnDqFGjSE1NrezwRURERE5Y0Z45ngKMBk6oBbPx8fFkZGSQnZ3N6tWr2bhx4zF1RowYwfbt23n44YcLEuDk5GRWrlzJkiVL2LFjB40aNcLd6devHwMHDuSTTz6p7FMREREROaFELTk2s17Af9z9/TLqDTezdDNLzwvtj1Y4MVG/fn2uvPJK0tLSSqzTv39/XnvttaPK3J3U1FTGjRvHhAkTmDRpEnfeeSdTp06NcsQiIiIiJ7ZozhxfCvQ2syxgLnCVmT1XtJK7P+3uKe6eEl+3XhTDqRx79+4lJycHgAMHDrBo0SLatGlzVJ0jd6IAePPNN2nVqtVR+5999ll69uxJgwYNCIVCxMXFERcXRygUinr8IiIiIieyqK05dvf7gfsBzOwKYJS7D4zW8aqKPXv2MGTIEPLy8sjPz6dv37706tWL8ePHk5KSQu/evXniiSf429/+RmJiIqeffjqzZ88uaB8KhZg1axZvv/02ACNHjqRnz57UqlWLF154IVanJSIiInJCqJSHgBRKjnuVVk8PARERERGRSlDiQ0CiereKI9x9KbC0Mo4lIiIiIlJRVerx0SkpKZ6enh7rMERERESkZtPjo0VEREREyqLkWEREREQkoORYRERERCSg5FhEREREJKDkWEREREQkoORYRERERCSg5FhEREREJKDkWEREREQkoORYRERERCSg5FhEREREJFBmcmxmdc1snJn9MXjfysx6RT80EREREZHKlRBBnWeA94GLg/e7gZeBN453MBt276f5fW8e725FRESkFFkTr491CCJVRiTLKr7n7pOAXAB3DwEW1ahERESkUn3zzTd06dKFjh070q5dOx544AEABgwYQOvWrWnfvj1Dhw4lNze32Pb33nsv7du3p3379sybN6+gfMCAASQlJTFmzJiCstTUVF577bWono9IRUWSHB8yszqAA5jZ94CDkXRuZllmtsHMMsws/VvEKSIiIlFUu3ZtFi9ezPr168nIyCAtLY2VK1cyYMAAPvzwQzZs2MCBAweYOXPmMW3ffPNN1q5dS0ZGBqtWreKRRx7hiy++IDMzkzp16pCZmcmaNWvYv38/e/bsYdWqVdx0002Vf5IiEYhkWcUDQBrQzMyeBy4FflyOY1zp7vsqEJuIiIhUEjPjlFNOASA3N5fc3FzMjJ49exbU6dKlC9nZ2ce03bx5M926dSMhIYGEhASSkpJIS0ujQ4cOHDhwgPz8fHJzc4mPj2f8+PFMmDCh0s5LpLxKnTk2szjgdOAWwgnxi0CKuy+NemQiIiJSqfLy8khOTuass87i2muvpWvXrgX7cnNzmTNnDj169DimXceOHUlLSyMUCrFv3z6WLFnCrl27aNu2LQ0bNqRTp07ccMMNbNu2jfz8fDp16lSZpyVSLqXOHLt7vpmNdveXgIpcKefA22bmwAx3f7oiQYqIiEj0xcfHk5GRQU5ODjfffDMbN26kffv2APz85z+nW7duXH755ce06969O2vWrOGSSy6hYcOGXHzxxcTHxwMwZcqUgno33HADM2bM4Le//S3r16/n2muv5c4776yUcxOJVCRrjv9mZqPMrJmZNTjyirD/y9y9E/ADYISZdStawcyGm1m6maXnhfaXJ3YRERGJgvr163PllVeSlpYGwIQJE9i7dy+PPvpoiW3Gjh1LRkYGixYtwt0577zzjto/f/58OnfuzFdffcX27dt56aWXeOWVVwiFQlE9F5HyiiQ57geMAJYRvqXb+0BEF9e5++7g53+AvwBdiqnztLunuHtKfN16kcYtIiIix9HevXvJyckB4MCBAyxatIg2bdowc+ZM3nrrLV588UXi4opPG/Ly8vj0008ByMzMJDMzk+7duxfsz83NZcqUKYwePZoDBw5gZgXtDh06FN0TEymnMi/Ic/dzK9KxmZ0MxLn7l8F2d+DBivQlIiIi0bVnzx6GDBlCXl4e+fn59O3bl169epGQkMB3v/tdLr44/LiDW265hfHjx5Oens706dOZOXMmubm5BcstTjvtNJ577jkSEv6bYjz55JMMGTKEunXrkpSURCgUokOHDvTs2ZP69evH4nRFSmTuXnoFs0TgZ8CRJRFLCa8fLv5Gh/9t14LwbDGEk/AX3P23pbWp3aiVNxoypeyoRURE5LjRQ0DkBFTiMzsiuZXbNCAReCp4PygoG1ZaI3ffAXSMMEARERERkZiLJDm+0N0LJ7mLzWx9tAISEREREYmVSJLjPDP7nrtvh4LlEnnRCKZDk3qk6087IiIiIhIjkSTHvwaWmNkOwuszvgvcHtWoRERERERiIJK7VbxjZq2A1kHRFnc/GN2wREREREQqX5n3OTazEUAdd89090ygrpn9PPqhiYiIiIhUrkgeAnKnu+cceePunwN61qOIiIiI1DiRJMfxduRRNoCZxQO1oheSiIiIiEhsRHJBXhowz8xmBO9/EpSJiIiIiNQokSTH9wLDCT8lD2ARMDNqEYmIiIiIxEgkd6vIB6YD082sAdDU3aNyn2MRERERkViK5G4VS83stCAxfh/4o5k9Fv3QREREREQqVyQX5NVz9y+AW4Bn3b0rcHV0wxIRERERqXyRrDlOMLNGQF9gbDSD2bB7P83vezOahxARERGRGMuaeH2sQyhRJDPHDwJvAdvcfY2ZtQC2RjcsEREREZHKV2Zy7O4vu3uSu/88eL/D3X8Y/dBEREREpKbLy8vjggsuoFevXgDccccddOzYkaSkJPr06cNXX311TJvnn3+e5OTkgldcXBwZGRkcPHiQHj160L59e5566qmC+sOHD2ft2rURxRPJzHGFmVl9M3vFzD40sw/M7OJoHk9EREREqpfHH3+ctm3bFrx/7LHHWL9+PZmZmZxzzjk88cQTx7QZMGAAGRkZZGRkMGfOHM4991ySk5N56623uOyyy8jMzGTOnDkArF+/nry8PDp16hRRPFFNjoHHgTR3bwN0BD6I8vFEREREpJrIzs7mzTffZNiwYQVlp512GgDuzoEDByj0oOZivfjii/Tv3x+AxMREQqEQubm5uDsA48aN46GHHoo4pkhu5RYfcW9Ht6sHdAP+BODuh9w9pyJ9iYiIiEjNc8899zBp0iTi4o5OSW+//XbOPvtsPvzwQ+66665S+5g3bx633XYbANdeey1ZWVlcdNFF/OIXv2DBggV06tSJxo0bRxxTJDPHW83s92Z2fsS9hp0L7AWeMbN1ZjbTzE4uWsnMhptZupml54X2l/MQIiIiIlIdvfHGG5x11ll07tz5mH3PPPMMH3/8MW3btmXevHkl9rFq1Srq1q1L+/btAUhISOCFF15g3bp13HrrrUyZMoVf/epXjBw5kj59+rBgwYIy44okOe4IfATMNLOVQTJ7WgTtEoBOwDR3vwD4GrivaCV3f9rdU9w9Jb5uvQi6FREREZHq7r333mPBggU0b96c/v37s3jxYgYOHFiwPz4+nv79+/Pqq6+W2MfcuXMLZo2Leuqppxg8eDArV66kXr16zJs3j8mTJ5cZVyR3q/jS3f/o7pcA9wIPAHvMbLaZtSylaTaQ7e6rgvevEE6WRUREROQE97vf/Y7s7GyysrKYO3cuV111FXPmzGHbtm1AeM3xggULaNOmTbHt8/PzeemllwrWGxf2+eef88YbbzB48GBCoRBxcXGYGQcOHCgzrojWHJtZbzP7CzAFmAy0AF4HFpbUzt3/Dewys9ZB0dXA5jIjEhEREZETkrszZMgQOnToQIcOHdizZw/jx48HYMGCBQXbAMuWLaNZs2a0aNHimH4efPBBxo4dS1xcHNdddx3Lly+nQ4cODBo0qMwY7MiVfCVWMNsBLAH+5O7/KLJvqrv/opS2ycBMoBawA7jd3T8vqX7tRq280ZApZQYtIiIiItVXFXhCXom3wIjk8dGD3f3vR/Vmdqm7v1daYgzg7hlASkQhioiIiIjEWCQzx2vdvVNZZcdDSkqKp6enH+9uRUREREQKK//McfA0u0uAhmY2stCu04AK3ftYRERERKQqK21ZRS3glKDOqYXKvwD6RDMoEREREZFYKDE5dvd3gXfNbJa7/7MSYxIRERERiYnSllVMcfd7gCfM7JiFye7eO5qBiYiIiIhUttKWVcwJfj5SGYGIiIiIiMRaacsq3g9+vlt54YiIiIiIxE5pyyo2ACXe583dk6ISkYiIiIhIjJS2rKJXpUUhIiIiIlIFlLasQneoEBEREZETSlxZFczsIjNbY2ZfmdkhM8szsy8qIzgRERERkcpUZnIMPAHcBmwF6gDDgCejGZSIiIiISCyYe4nX3IUrmKW7e4qZZR65CM/M1rn7Bcc7mNqNWnmjIVOOd7ciUomyJl4f6xBERETKYiXtiGTmOGRmtYAMM5tkZr+MsJ2InKCGDh3KWWedRfv27Y8q/8Mf/kCbNm1o164do0ePLrZtTk4Offr0oU2bNrRt25YVK1YAcO+995KUlMTgwYML6j733HNMmTIlauchIiInnkiS3EFAPPA/wNdAM+CHZTUys9ZmllHo9YWZ3fOtohWRauHHP/4xaWlpR5UtWbKE+fPns379ejZt2sSoUaOKbXv33XfTo0cPPvzwQ9avX0/btm3Zv38/a9euJTMzk1q1arFhwwYOHDjAM888w4gRIyrjlERE5ARR2q3cgKPuWnEAmBBpx+6+BUgGMLN4YDfwl/KHKCLVTbdu3cjKyjqqbNq0adx3333Url0bgLPOOuuYdvv372fZsmXMmjULgFq1alGrVi2+/PJLcnNzcXdCoRCJiYk88sgj3HXXXSQmJkb7dERE5AQSyd0qdprZjqKvch7namC7bg8ncuL66KOPWL58OV27duX73/8+a9asOabOzp07adiwIbfffjsXXHABw4YN4+uvv+bUU0+lZ8+eXHDBBTRq1Ih69eqxatUqbrrppso/ERERqdEiWVaRAlwYvC4HpgLPlfM4/YEXy9lGRGqQw4cP89lnn7Fy5Up+//vf07dvX4peEHz48GHWrl3Lz372M9atW8fJJ5/MxIkTARg9ejQZGRlMnjyZcePG8eCDDzJz5kz69u1LampqLE5JRERqoDKTY3f/tNBrt7tPASK+HD24mK838HIJ+4ebWbqZpeeF9kfarYhUM02bNuWWW27BzOjSpQtxcXHs27fvmDpNmzala9euAPTp04e1a9ceVWfdunW4O61bt+bll1/mpZdeYvv27WzdurXSzkVERGquSJZVdCr0SjGznxLBWuVCfgCsdfdPitvp7k+7e4q7p8TXrVeObkWkOrnppptYsmQJEF5icejQIc4888yj6px99tk0a9aMLVu2APDOO+9w/vnnH1Vn3LhxPPTQQ+Tm5pKXlwdAXFwcoVCoEs5CRERqukiS3MmFtg8DWUDfchzjNrSkQuSEctttt7F06VL27dtH06ZNmTBhAkOHDmXo0KG0b9+eWrVqMXv2bMyMjz/+mGHDhrFw4UIgfLu3AQMGcOjQIVq0aMEzzzxT0O9rr71GSkoKjRs3BiA5OZkOHTqQlJREx44dY3KuIiJSs5T5EJBv1bnZycC/gBbuXuaaCT0ERKT600NARESkGijxISBlzhyb2cjS9rv7o6Xs+xo4o6xjiIiIiIhUBZEsqzhyt4oFwfsbgNXAcb/6pUOTeqRr1klEREREYiSS5Lgp0MndvwQws98Ab7r7wGgGJiIiIiJS2SK5z/F3gEOF3h8KykREREREapRIZo6fBVab2ZFHP98EzIpWQCIiIiIisVJmcuzuvzWzvxJ+Oh7A7e6+LrphiYiIiIhUvoge5uHua4G1ZVYUEREREanGIllzLCIiIiJyQlByLCIiIiISUHIsIiIiIhJQciwiIiIiElByLCIiIiISUHIsIiIiIhIwd491DAVqN2rljYZMiXUYEcmaeH2sQxARERGRirGSdmjm+Fv45ptv6NKlCx07dqRdu3Y88MADx9Q5ePAg/fr1o2XLlnTt2pWsrCwA3nvvPZKSkkhJSWHr1q0A5OTk0L17d/Lz8yvzNEREREQkENXk2Mx+aWabzGyjmb1oZidF83iVrXbt2ixevJj169eTkZFBWloaK1euPKrOn/70J04//XS2bdvGL3/5S+69914AJk+ezMKFC5kyZQrTp08HIDU1lTFjxhAXp3+ziIiIiMRC1LIwM2sC/AJIcff2QDzQP1rHiwUz45RTTgEgNzeX3NxczI6epZ8/fz5DhgwBoE+fPrzzzju4O4mJiYRCIUKhEImJiWzfvp1du3ZxxRVXVPZpiIiIiEggosdHf8v+65hZLlAX+DjKx6t0eXl5dO7cmW3btjFixAi6du161P7du3fTrFkzABISEqhXrx6ffvop999/P4MHD6ZOnTrMmTOHUaNGkZqaGotTEBEREZFA1GaO3X038AjwL2APsN/d347W8WIlPj6ejIwMsrOzWb16NRs3boyoXXJyMitXrmTJkiXs2LGDRo0a4e7069ePgQMH8sknn0Q5chEREREpKprLKk4HbgTOBRoDJ5vZwGLqDTezdDNLzwvtj1Y4UVe/fn2uvPJK0tLSjipv0qQJu3btAuDw4cPs37+fM844o2C/u5Oamsq4ceOYMGECkyZN4s4772Tq1KmVGr+IiIiIRPeCvGuAne6+191zgf8DLilayd2fdvcUd0+Jr1sviuEcf3v37iUnJweAAwcOsGjRItq0aXNUnd69ezN79mwAXnnlFa666qqj1iU/++yz9OzZkwYNGhAKhYiLiyMuLo5QKFRp5yEiIiIiYdFcc/wv4CIzqwscAK4G0qN4vEq3Z88ehgwZQl5eHvn5+fTt25devXoxfvx4UlJS6N27N3fccQeDBg2iZcuWNGjQgLlz5xa0D4VCzJo1i7ffDq82GTlyJD179qRWrVq88MILsTotERERkRNWVB8CYmYTgH7AYWAdMMzdD5ZUXw8BEREREZFKUOJDQKJ6twp3fwA49skYIiIiIiJVkJ42ISIiIiISiOqyivJKSUnx9PQatSxZRERERKqeEpdVaOZYRERERCSg5FhEREREJKDkWEREREQkoORYRERERCSg5FhEREREJKDkWEREREQkoORYRERERCSg5FhEREREJKDkWEREREQkoORYRERERCRQpZLjDbv3xzoEERERETmBVankWEREREQklqpkcpyWlkbr1q1p2bIlEydOPGb/smXL6NSpEwkJCbzyyisF5Vu2bKFz584kJSWxYsUKAA4fPsw111xDKBSqtPhFREREpHqKanJsZneb2UYz22Rm90TSJi8vjxEjRvDXv/6VzZs38+KLL7J58+aj6pxzzjnMmjWLH/3oR0eVz5gxg8cff5yFCxfyyCOPADBt2jQGDhxI3bp1j89JiYiIiEiNlRCtjs2sPXAn0AU4BKSZ2Rvuvq20dqtXr6Zly5a0aNECgP79+zN//nzOP//8gjrNmzcHIC7u6Nw+MTGRUChEKBQiMTGRnJwcXn/9ddLS0o7jmYmIiIhITRW15BhoC6xy9xCAmb0L3AJMKq3R7t27adasWcH7pk2bsmrVqogOOGLECAYPHszBgweZMWMGDz30EGPGjDkmiRYRERERKU40s8aNwOVmdoaZ1QV6As2KVjKz4WaWbmbpeaFvd7eKc845h6VLl7JixQrq1q1LdnY2bdu2ZdCgQfTr14+PPvroW/UvIiIiIjVb1GaO3f0DM3sYeBv4GsgA8oqp9zTwNEDtRq28SZMm7Nq1q2B/dnY2TZo0Kffxx44dS2pqKlOnTmXYsGE0b96cMWPG8Pzzz1fshERERESkxovqegN3/5O7d3b3bsDnQJlTtxdeeCFbt25l586dHDp0iLlz59K7d+9yHffdd9+lcePGtGrVilAoRFxcHHFxcbpjhYiIiIiUytw9ep2bneXu/zGzcwjPIF/k7jkl1a/dqJUf3LOVhQsXcs8995CXl8fQoUMZO3Ys48ePJyUlhd69e7NmzRpuvvlmPv/8c0466STOPvtsNm3aBIC70717d+bNm0eDBg344IMPGDBgAIcPH2batGlceumlUTtfEREREakWrMQdUU6OlwNnALnASHd/p7T6R5JjEREREZEoKjE5jubdKnD3y6PZv4iIiIjI8VSl7nHWoUm9WIcgIiIiIiewKpUci4iIiIjEkpJjEREREZGAkmMRERERkYCSYxERERGRgJJjEREREZFAVO9zXF5m9iWwJdZxSIWdCeyLdRBSYRq/6k9jWL1p/Ko/jWH1sc/dexS3I6r3Oa6ALe6eEusgpGLMLF3jV31p/Ko/jWH1pvGr/jSGNYOWVYiIiIiIBJQci4iIiIgEqlpy/HSsA5BvReNXvWn8qj+NYfWm8av+NIY1QJW6IE9EREREJJaq2syxiIiIiEjMVInk2Mx6mNkWM9tmZvfFOh4pnpllmdkGM8sws/SgrIGZLTKzrcHP04NyM7OpwZhmmlmn2EZ/YjKzP5vZf8xsY6Gyco+ZmQ0J6m81syGxOJcTUQnj9xsz2x18DzPMrGehffcH47fFzK4rVK7fsTFgZs3MbImZbTazTWZ2d1Cu72A1UcoY6ntYk7l7TF9APLAdaAHUAtYD58c6Lr2KHass4MwiZZOA+4Lt+4CHg+2ewF8BAy4CVsU6/hPxBXQDOgEbKzpmQANgR/Dz9GD79Fif24nwKmH8fgOMKqbu+cHvz9rAucHv1Xj9jo3p+DUCOgXbpwIfBeOk72A1eZUyhvoe1uBXVZg57gJsc/cd7n4ImAvcGOOYJHI3ArOD7dnATYXKn/WwlUB9M2sUg/hOaO6+DPisSHF5x+w6YJG7f+bunwOLgGJvnC7HVwnjV5IbgbnuftDddwLbCP9+1e/YGHH3Pe6+Ntj+EvgAaIK+g9VGKWNYEn0Pa4CqkBw3AXYVep9N6f/hSew48LaZvW9mw4Oy77j7nmD738B3gm2Na9VV3jHTWFY9/xP82f3PR/4kj8avSjOz5sAFwCr0HayWiowh6HtYY1WF5Fiqj8vcvRPwA2CEmXUrvNPdnXACLdWExqxamgZ8D0gG9gCTYxqNlMnMTgFeBe5x9y8K79N3sHooZgz1PazBqkJyvBtoVuh906BMqhh33x38/A/wF8J/JvrkyHKJ4Od/guoa16qrvGOmsaxC3P0Td89z93zgj4S/h6Dxq5LMLJFwUvW8u/9fUKzvYDVS3Bjqe1izVYXkeA3QyszONbNaQH9gQYxjkiLM7GQzO/XINtAd2Eh4rI5cOT0EmB9sLwAGB1dfXwTsL/RnRImt8o7ZW0B3Mzs9+NNh96BMYqDI2v2bCX8PITx+/c2stpmdC7QCVqPfsTFjZgb8CfjA3R8ttEvfwWqipDHU97BmS4h1AO5+2Mz+h/AXPR74s7tvinFYcqzvAH8J/54gAXjB3dPMbA3wkpndAfwT6BvUX0j4yuttQAi4vfJDFjN7EbgCONPMsoEHgImUY8zc/TMze4jwL3eAB9090ovE5FsoYfyuMLNkwn+KzwJ+AuDum8zsJWAzcBgY4e55QT/6HRsblwKDgA1mlhGUjUHfweqkpDG8Td/DmktPyBMRERERCVSFZRUiIiIiIlWCkmMRERERkYCSYxERERGRgJJjEREREZGAkmMRERERkYCSYxGRKsbM7jGzuuVsc7mZbTKzDDOrU862C82sfrmCFBGpoXQrNxGRKsbMsoAUd99XjjbTgb+7+3NRC0xE5ASgmWMRkXIys8Fmlmlm681sTlDW3MwWB+XvmNk5QfksM+tTqO1Xwc8rzGypmb1iZh+a2fPBk9F+ATQGlpjZkmKOfbWZrTOzDWb25+BJXMMIP0jiITN7vkj9Xwd9YmaPmdniYPuqI3XNLMvMzgzO4QMz+2MwC/32kVloM/uemaWZ2ftmttzM2gTlt5rZxuCzWHa8P2sRkcqm5FhEpBzMrB3w/4Cr3L0jcHew6w/AbHdPAp4HpkbQ3QXAPcD5QAvgUnefCnwMXOnuVxY59knALKCfu3cg/LTKn7n7TMKPov21uw8ocozlwOXBdgpwipklBmXFJbOtgCfdvR2QA/wwKH8auMvdOwOjgKeC8vHAdcFn0TuCcxYRqdKUHIuIlM9VwMtHljwUeozvxcALwfYc4LII+lrt7tnung9kAM3LqN8a2OnuHwXvZwPdymjzPtDZzE4DDgIrCCfJlxNOnIva6e4Zhdo2N7NTgEuAl4NH6M4AGgV13gNmmdmdhB+LKyJSrSXEOgARkRruMMFEhJnFAbUK7TtYaDuPKPxOdvdcM9sJ/Bj4B5AJXAm0BD4opknRmOoQjj/H3ZOL6f+nZtYVuB5438w6u/unx/UkREQqkWaORUTKZzFwq5mdAWBmDYLyfwD9g+0B/HdWNgvoHGz3BhIjOMaXwKnFlG8hPJPbMng/CHg3gv6WE14KsSzY/imwziO8ItvdvwB2mtmtAMHa6I7B9vfcfZW7jwf2As0i6VNEpKpSciwiUg7uvgn4LfCuma0HHg123QXcbmaZhJPWI2uR/wh8P6h7MfB1BId5GkgrekGeu38D3E54ecMGIB+YHkF/ywkvg1jh7p8A31D8korSDADuCM5jE3BjUP774OLAjYT/gbC+nP2KiFQpupWbiIiIiEhAM8ciIiIiIgElxyIiIiIiASXHIiIiIiIBJcciIiIiIgElxyIiIiIiASXHIiIiIiIBJcciIiIiIgElxyIiIiIigf8P0x+xO7CKOqQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_quality_scores(df, kind):\n",
    "    ax = df.quality.value_counts().sort_index().plot.barh(\n",
    "        title=f'{kind.title()} Wine Quality Scores', figsize=(12, 3)\n",
    "    )\n",
    "    ax.axes.invert_yaxis()\n",
    "    for bar in ax.patches:\n",
    "        ax.text(\n",
    "            bar.get_width(), \n",
    "            bar.get_y() + bar.get_height()/2, \n",
    "            f'{bar.get_width()/df.shape[0]:.1%}',\n",
    "            verticalalignment='center'\n",
    "        )\n",
    "    plt.xlabel('count of wines')\n",
    "    plt.ylabel('quality score')\n",
    "\n",
    "    for spine in ['top', 'right']:\n",
    "        ax.spines[spine].set_visible(False)\n",
    "\n",
    "    return ax\n",
    "\n",
    "plot_quality_scores(wine, '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 6497 entries, 0 to 1598\n",
      "Data columns (total 13 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   fixed acidity         6497 non-null   float64\n",
      " 1   volatile acidity      6497 non-null   float64\n",
      " 2   citric acid           6497 non-null   float64\n",
      " 3   residual sugar        6497 non-null   float64\n",
      " 4   chlorides             6497 non-null   float64\n",
      " 5   free sulfur dioxide   6497 non-null   float64\n",
      " 6   total sulfur dioxide  6497 non-null   float64\n",
      " 7   density               6497 non-null   float64\n",
      " 8   pH                    6497 non-null   float64\n",
      " 9   sulphates             6497 non-null   float64\n",
      " 10  alcohol               6497 non-null   float64\n",
      " 11  quality               6497 non-null   int64  \n",
      " 12  kind                  6497 non-null   object \n",
      "dtypes: float64(11), int64(1), object(1)\n",
      "memory usage: 710.6+ KB\n"
     ]
    }
   ],
   "source": [
    "wine.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed acidity</th>\n",
       "      <th>volatile acidity</th>\n",
       "      <th>citric acid</th>\n",
       "      <th>residual sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free sulfur dioxide</th>\n",
       "      <th>total sulfur dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>quality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "      <td>6497.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>7.215307</td>\n",
       "      <td>0.339666</td>\n",
       "      <td>0.318633</td>\n",
       "      <td>5.443235</td>\n",
       "      <td>0.056034</td>\n",
       "      <td>30.525319</td>\n",
       "      <td>115.744574</td>\n",
       "      <td>0.994697</td>\n",
       "      <td>3.218501</td>\n",
       "      <td>0.531268</td>\n",
       "      <td>10.491801</td>\n",
       "      <td>5.818378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.296434</td>\n",
       "      <td>0.164636</td>\n",
       "      <td>0.145318</td>\n",
       "      <td>4.757804</td>\n",
       "      <td>0.035034</td>\n",
       "      <td>17.749400</td>\n",
       "      <td>56.521855</td>\n",
       "      <td>0.002999</td>\n",
       "      <td>0.160787</td>\n",
       "      <td>0.148806</td>\n",
       "      <td>1.192712</td>\n",
       "      <td>0.873255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>3.800000</td>\n",
       "      <td>0.080000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.009000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>0.987110</td>\n",
       "      <td>2.720000</td>\n",
       "      <td>0.220000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>6.400000</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>1.800000</td>\n",
       "      <td>0.038000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>77.000000</td>\n",
       "      <td>0.992340</td>\n",
       "      <td>3.110000</td>\n",
       "      <td>0.430000</td>\n",
       "      <td>9.500000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.290000</td>\n",
       "      <td>0.310000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.047000</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>118.000000</td>\n",
       "      <td>0.994890</td>\n",
       "      <td>3.210000</td>\n",
       "      <td>0.510000</td>\n",
       "      <td>10.300000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>7.700000</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.390000</td>\n",
       "      <td>8.100000</td>\n",
       "      <td>0.065000</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>156.000000</td>\n",
       "      <td>0.996990</td>\n",
       "      <td>3.320000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>11.300000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>15.900000</td>\n",
       "      <td>1.580000</td>\n",
       "      <td>1.660000</td>\n",
       "      <td>65.800000</td>\n",
       "      <td>0.611000</td>\n",
       "      <td>289.000000</td>\n",
       "      <td>440.000000</td>\n",
       "      <td>1.038980</td>\n",
       "      <td>4.010000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>14.900000</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       fixed acidity  volatile acidity  citric acid  residual sugar  \\\n",
       "count    6497.000000       6497.000000  6497.000000     6497.000000   \n",
       "mean        7.215307          0.339666     0.318633        5.443235   \n",
       "std         1.296434          0.164636     0.145318        4.757804   \n",
       "min         3.800000          0.080000     0.000000        0.600000   \n",
       "25%         6.400000          0.230000     0.250000        1.800000   \n",
       "50%         7.000000          0.290000     0.310000        3.000000   \n",
       "75%         7.700000          0.400000     0.390000        8.100000   \n",
       "max        15.900000          1.580000     1.660000       65.800000   \n",
       "\n",
       "         chlorides  free sulfur dioxide  total sulfur dioxide      density  \\\n",
       "count  6497.000000          6497.000000           6497.000000  6497.000000   \n",
       "mean      0.056034            30.525319            115.744574     0.994697   \n",
       "std       0.035034            17.749400             56.521855     0.002999   \n",
       "min       0.009000             1.000000              6.000000     0.987110   \n",
       "25%       0.038000            17.000000             77.000000     0.992340   \n",
       "50%       0.047000            29.000000            118.000000     0.994890   \n",
       "75%       0.065000            41.000000            156.000000     0.996990   \n",
       "max       0.611000           289.000000            440.000000     1.038980   \n",
       "\n",
       "                pH    sulphates      alcohol      quality  \n",
       "count  6497.000000  6497.000000  6497.000000  6497.000000  \n",
       "mean      3.218501     0.531268    10.491801     5.818378  \n",
       "std       0.160787     0.148806     1.192712     0.873255  \n",
       "min       2.720000     0.220000     8.000000     3.000000  \n",
       "25%       3.110000     0.430000     9.500000     5.000000  \n",
       "50%       3.210000     0.510000    10.300000     6.000000  \n",
       "75%       3.320000     0.600000    11.300000     6.000000  \n",
       "max       4.010000     2.000000    14.900000     9.000000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "white    4898\n",
       "red      1599\n",
       "Name: kind, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine.kind.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed acidity</th>\n",
       "      <th>volatile acidity</th>\n",
       "      <th>citric acid</th>\n",
       "      <th>residual sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free sulfur dioxide</th>\n",
       "      <th>total sulfur dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>kind</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>987</th>\n",
       "      <td>7.1</td>\n",
       "      <td>0.36</td>\n",
       "      <td>0.30</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.080</td>\n",
       "      <td>35.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>0.99693</td>\n",
       "      <td>3.44</td>\n",
       "      <td>0.50</td>\n",
       "      <td>9.4</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1219</th>\n",
       "      <td>8.2</td>\n",
       "      <td>0.37</td>\n",
       "      <td>0.36</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.034</td>\n",
       "      <td>17.0</td>\n",
       "      <td>93.0</td>\n",
       "      <td>0.99060</td>\n",
       "      <td>3.04</td>\n",
       "      <td>0.32</td>\n",
       "      <td>11.7</td>\n",
       "      <td>white</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2380</th>\n",
       "      <td>6.4</td>\n",
       "      <td>0.27</td>\n",
       "      <td>0.19</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.085</td>\n",
       "      <td>21.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>0.99516</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.64</td>\n",
       "      <td>9.5</td>\n",
       "      <td>white</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2770</th>\n",
       "      <td>6.4</td>\n",
       "      <td>0.44</td>\n",
       "      <td>0.44</td>\n",
       "      <td>14.4</td>\n",
       "      <td>0.048</td>\n",
       "      <td>29.0</td>\n",
       "      <td>228.0</td>\n",
       "      <td>0.99955</td>\n",
       "      <td>3.26</td>\n",
       "      <td>0.54</td>\n",
       "      <td>8.8</td>\n",
       "      <td>white</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>8.2</td>\n",
       "      <td>0.40</td>\n",
       "      <td>0.48</td>\n",
       "      <td>13.7</td>\n",
       "      <td>0.042</td>\n",
       "      <td>59.0</td>\n",
       "      <td>169.0</td>\n",
       "      <td>0.99860</td>\n",
       "      <td>3.10</td>\n",
       "      <td>0.52</td>\n",
       "      <td>9.4</td>\n",
       "      <td>white</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
       "987             7.1              0.36         0.30             1.6      0.080   \n",
       "1219            8.2              0.37         0.36             1.0      0.034   \n",
       "2380            6.4              0.27         0.19             1.9      0.085   \n",
       "2770            6.4              0.44         0.44            14.4      0.048   \n",
       "857             8.2              0.40         0.48            13.7      0.042   \n",
       "\n",
       "      free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
       "987                  35.0                  70.0  0.99693  3.44       0.50   \n",
       "1219                 17.0                  93.0  0.99060  3.04       0.32   \n",
       "2380                 21.0                 196.0  0.99516  3.49       0.64   \n",
       "2770                 29.0                 228.0  0.99955  3.26       0.54   \n",
       "857                  59.0                 169.0  0.99860  3.10       0.52   \n",
       "\n",
       "      alcohol   kind  \n",
       "987       9.4    red  \n",
       "1219     11.7  white  \n",
       "2380      9.5  white  \n",
       "2770      8.8  white  \n",
       "857       9.4  white  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "wine_y = wine.pop('quality')\n",
    "wine_X = wine\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    wine_X, wine_y, test_size=0.25, random_state=0, stratify=wine_y\n",
    ")\n",
    "\n",
    "X_train.sample(5, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build models\n",
    "### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('transformer', ColumnTransformer([\n",
    "        ('scale', StandardScaler(), slice(0, -1)),\n",
    "        ('encode', OneHotEncoder(sparse=False), [-1])\n",
    "    ])),\n",
    "    ('rf', RandomForestClassifier(n_estimators=100, random_state=0))\n",
    "])\n",
    "\n",
    "search_space = {\n",
    "    'rf__max_depth': np.arange(5, 20, 5)\n",
    "}\n",
    "\n",
    "rf = GridSearchCV(pipeline, search_space, scoring='f1_macro', cv=5).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosted Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('transformer', ColumnTransformer([\n",
    "        ('scale', StandardScaler(), slice(0, -1)),\n",
    "        ('encode', OneHotEncoder(sparse=False), [-1])\n",
    "    ])),\n",
    "    ('gb', GradientBoostingClassifier(random_state=0))\n",
    "])\n",
    "\n",
    "search_space = {\n",
    "    'gb__max_depth': np.arange(3, 12, 3)\n",
    "}\n",
    "\n",
    "gb = GridSearchCV(pipeline, search_space, scoring='f1_macro', cv=5).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k-NN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('transformer', ColumnTransformer([\n",
    "        ('scale', StandardScaler(), slice(0, -1)),\n",
    "        ('encode', OneHotEncoder(sparse=False), [-1])\n",
    "    ])),\n",
    "    ('knn', KNeighborsClassifier())\n",
    "])\n",
    "\n",
    "search_space = {\n",
    "    'knn__n_neighbors': np.arange(1, 5)\n",
    "}\n",
    "\n",
    "knn = GridSearchCV(pipeline, search_space, scoring='f1_macro', cv=5).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('transformer', ColumnTransformer([\n",
    "        ('scale', StandardScaler(), slice(0, -1)),\n",
    "        ('encode', OneHotEncoder(sparse=False), [-1])\n",
    "    ])),\n",
    "    ('lr', LogisticRegression(random_state=0, multi_class='multinomial', max_iter=1000)))\n",
    "])\n",
    "\n",
    "search_space = {\n",
    "    'lr__C': [0.1, 0.5, 1, 5]\n",
    "}\n",
    "\n",
    "lr = GridSearchCV(pipeline, search_space, scoring='f1_macro', cv=5).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "nb = Pipeline([\n",
    "    ('transformer', ColumnTransformer([\n",
    "        ('scale', StandardScaler(), slice(0, -1)),\n",
    "        ('encode', OneHotEncoder(sparse=False), [-1])\n",
    "    ])),\n",
    "    ('nb', GaussianNB())\n",
    "]).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine agreement between the models with Cohen's Kappa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's Kappa between random forest and gradient boosting is: 0.78\n",
      "Cohen's Kappa between random forest and knn is: 0.55\n",
      "Cohen's Kappa between random forest and logistic regression is: 0.51\n",
      "Cohen's Kappa between random forest and naive bayes is: 0.13\n",
      "Cohen's Kappa between gradient boosting and knn is: 0.55\n",
      "Cohen's Kappa between gradient boosting and logistic regression is: 0.45\n",
      "Cohen's Kappa between gradient boosting and naive bayes is: 0.13\n",
      "Cohen's Kappa between knn and logistic regression is: 0.24\n",
      "Cohen's Kappa between knn and naive bayes is: 0.09\n",
      "Cohen's Kappa between logistic regression and naive bayes is: 0.19\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "from sklearn.metrics import cohen_kappa_score\n",
    "\n",
    "models = zip(\n",
    "    ['random forest', 'gradient boosting', 'knn', 'logistic regression', 'naive bayes'], \n",
    "    [rf, gb, knn, lr, nb]\n",
    ")\n",
    "\n",
    "def get_preds(model, test_X_data):\n",
    "    return model.predict(test_X_data)\n",
    "\n",
    "for ((model_1_name, model_1), (model_2_name, model_2)) in itertools.combinations(models, 2):\n",
    "    score = cohen_kappa_score(*map(get_preds, [model_1, model_2], itertools.repeat(X_test)))\n",
    "    print(f\"Cohen's Kappa between {model_1_name} and {model_2_name} is: {score:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Voting Classifier with Majority Rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "majority_rules = VotingClassifier(\n",
    "    [('rf', rf.best_estimator_), ('gb', gb.best_estimator_), \n",
    "     ('knn', knn.best_estimator_), ('lr', lr.best_estimator_),\n",
    "     ('nb', nb)],\n",
    "    voting='hard',\n",
    "    weights=[1, 1, 1, 1, .5]\n",
    ").fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6658461538461539"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "majority_rules.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = majority_rules.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the classification report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           3       0.00      0.00      0.00         8\n",
      "           4       0.70      0.13      0.22        54\n",
      "           5       0.70      0.70      0.70       535\n",
      "           6       0.62      0.78      0.69       709\n",
      "           7       0.74      0.46      0.57       270\n",
      "           8       0.90      0.38      0.53        48\n",
      "           9       0.00      0.00      0.00         1\n",
      "\n",
      "    accuracy                           0.67      1625\n",
      "   macro avg       0.52      0.35      0.39      1625\n",
      "weighted avg       0.68      0.67      0.65      1625\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefaniemolin/book_env/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test, preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:title={'center':'Confusion Matrix'}, xlabel='Actual', ylabel='Model Prediction'>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEWCAYAAADy2YssAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA22ElEQVR4nO3dd3xUVdrA8d+ThAgkdElACQKCIkVRUbGAAoI0pakooqK42WUVRUSluAhYVlwb6lpQVMAFwcKKgoivgiAqUqWIuqgIAZLQS1DTnvePexOHQCYzyZTM8Hz93I8zd+7c50wmeTjnnnvOEVXFGGOiUUy4C2CMMcFiCc4YE7UswRljopYlOGNM1LIEZ4yJWpbgjDFRyxJclBGRSiLygYjsF5G3y3CeG0RkQSDLFg4i8pGI3BzucpjwsAQXJiLSX0RWiMghEdnh/iFeEoBTXw0kA7VU9ZrSnkRV/6OqnQNQniOIyGUioiIyu8j+s9z9i3w8z1gRebOk41S1q6pOKWVxTYSzBBcGIjIMeAZ4FCcZ1QdeAHoG4PSnAD+qam4AzhUsO4ELRaSWx76bgR8DFUAc9vt9vFNV20K4AdWAQ8A1Xo45AScBbne3Z4AT3NcuA9KAe4BMYAdwi/vaOCAbyHFjDALGAm96nLsBoECc+3wg8DNwEPgFuMFj/xce77sIWA7sd/9/kcdri4CHgKXueRYAJxbz2QrK/xJwu7svFtgGjAEWeRw7EdgKHABWAm3d/V2KfM5vPcrxiFuO34DG7r7b3NdfBN71OP8E4FNAwv17YVtwNvsXLvQuBCoCs70cMxpoA7QCzgLOBx7weL0OTqI8GSeJ/VtEaqjqgzi1wpmqmqiqk70VREQSgGeBrqpaBSeJrTnGcTWBue6xtYCngLlFamD9gVuAJCAeGO4tNjAVuMl9fAWwHieZe1qO8zOoCUwH3haRiqo6v8jnPMvjPTcCqUAV4Nci57sHaCkiA0WkLc7P7mZ1s52JPpbgQq8WsEu9NyFvAMaraqaq7sSpmd3o8XqO+3qOqs7DqcWcXsry5AMtRKSSqu5Q1Q3HOKY78D9Vnaaquao6A/geuNLjmNdV9UdV/Q2YhZOYiqWqXwI1ReR0nEQ39RjHvKmqu92YT+LUbEv6nG+o6gb3PTlFzncY5+f4FPAmMERV00o4n4lgluBCbzdwoojEeTnmJI6sffzq7is8R5EEeRhI9LcgqpoF9AP+BuwQkbki0tSH8hSU6WSP5+mlKM804A6gPceo0YrIcBHZ6PYI78OptZ5Ywjm3entRVZfhNMkFJxGbKGYJLvS+Av4Aenk5ZjtOZ0GB+hzdfPNVFlDZ43kdzxdV9WNV7QTUxamVveJDeQrKtK2UZSowDfg7MM+tXRVym5D3AdcCNVS1Os71PykoejHn9NrcFJHbcWqC293zmyhmCS7EVHU/zsX0f4tILxGpLCIVRKSriDzuHjYDeEBEaovIie7xJd4SUYw1QDsRqS8i1YCRBS+ISLKI9HSvxf2B09TNP8Y55gGnube2xIlIP6AZ8GEpywSAqv4CXIpzzbGoKkAuTo9rnIiMAap6vJ4BNPCnp1RETgMeBgbgNFXvE5FWpSu9iQSW4MLAvZ40DKfjYCdOs+oO4L/uIQ8DK4C1wDpglbuvNLE+AWa651rJkUkpxi3HdmAPTrIZfIxz7AZ64Fyk341T8+mhqrtKU6Yi5/5CVY9VO/0YmI9z68ivwO8c2fwsuIl5t4isKimOe0ngTWCCqn6rqv8DRgHTROSEsnwGU36JdSAZY6KV1eCMMVHLEpwxJmpZgjPGRC1LcMaYqOXtZtOQ+z3X+z1MxpiyqRhXeB9hqVU6+w6f/05/W/18meOVhdXgjDFRq1zV4IwxESCCZqGyBGeM8U9MbLhL4DNLcMYY/0hYL6v5xRKcMcY/1kQ1xkQtq8EZY6KW1eCMMVHLanDGmKhlvajGmKhlTVRjTNSKoCZq5KRiL5YuWcxV3a+gR5dOTH5lUsjijnlgJJe1vZA+PXuELKancH1ui338xT6CxPi+hVnQSiAiFUXkGxH5VkQ2iMi4YMTJy8vj0UfG88JLrzJ7zlzmz/uQnzZtCkaoo/Ts1YcXX341JLGKCufnttjHV+yjWIIDnEVMOriL8rYCuohIm0AHWb9uLSkpp1AvJYUK8fF06dadRQs/DXSYYzq39XlUrVYtJLGKCufnttjHV+yjxMb6voVZ0BKcOg65Tyu4W8CnQ8rMyKBO3T9XwktKTiYjIyPQYcqdcH5ui318xT6KiO9bmAW1DikisSKyBsgEPnEX3S16TKqIrBCRFWG9rmCM8U0ENVGD2ouqqnlAKxGpDswWkRaqur7IMZOASVC6CS+TkpNJ3/HnouqZGRkkJyeXqdyRIJyf22IfX7GPUg5qZr4KSYpV1X3AQqBLoM/dvEVLtmzZTFraVnKys5k/by6Xtu8Q6DDlTjg/t8U+vmIfxWpwICK1gRxV3ScilYBOwIRAx4mLi2Pk6DEMTr2N/Pw8evXuS+PGTQId5pjuHz6MFcu/Yd++vXTq0I7Btw+hT99rQhI7nJ/bYh9fsY8SwBqciGwGDgJ5QK6qthaRmjiLlTcANgPXqupeERFgItANOAwMVFWvi34HbeFnETkTmALE4tQUZ6nqeG/vsTUZjAmugKzJ0PVp39dk+Ohur/HcBNdaVXd57Hsc2KOqj4nICKCGqt4vIt2AITgJ7gJgoqpe4O38QavBqepa4Oxgnd8YEybBb3r2BC5zH08BFgH3u/unqlMr+1pEqotIXVXdUdyJwt9INsZEFj9uE/G8S8LdUoucTYEFIrLS47Vkj6SVDhT0ppwMbPV4b5q7r1g2FtUY4x8/anCed0kU4xJV3SYiScAnIvJ9kferiJT60pXV4Iwx/glgL6qqbnP/nwnMBs4HMkSkLoD7/0z38G1Aisfb67n7imUJzhjjn5hY3zcvRCRBRKoUPAY6A+uBOcDN7mE3A++7j+cAN4mjDbDf2/U3sCaqMcZfgbtNJBlnAAA4uWi6qs4XkeXALBEZBPwKXOsePw+nB3UTzm0it5QUwBKcMcY/AepFVdWfgbOOsX830PEY+xW43Z8YluCMMf6JoKFaluCMMX4RS3DGmGhlCS7C5OeHb4RYTEzk/LIYAyAR9DtrCc4Y4xerwRljopYlOGNM1LIEZ4yJXpGT3yzBGWP8YzU4Y0zUiomJnCHsluCMMX6xGpwxJnpFTn6zBGeM8Y/V4IwxUcsSnDEmakXSUK3I6Q7xYumSxVzV/Qp6dOnE5Fe8Tf8eWJt/+Zl+V/cq3C5pcy7/mTYlJLHHPDCSy9peSJ+ePUISz1P6jh0MGngjva/sRu+ruofsMxcI1/d9PMf2JM5iMj5t4Rb0BCcisSKyWkQ+DMb58/LyePSR8bzw0qvMnjOX+fM+5KdNm4IR6igNGjZi5jv/ZeY7/2X6zHepWLES7TteHpLYPXv14cWXXw1JrKJi42IZft8IZn8wjzdnzOStGdND9jMP5/d9vMYuyhLcke4CNgbr5OvXrSUl5RTqpaRQIT6eLt26s2jhp8EKV6xvln1FvZQUTjrJ6ypmAXNu6/OoWq1aSGIVVbt2Emc0aw5AQkIijRo1IjMzIySxw/l9H6+xi7IE5xKRekB3IGhVjcyMDOrUrVP4PCk5mYyM0Pyxefr4o3l06do95HHDbdu2NL7fuJGWZx4183RQhPP7Pl5jF2UJ7k/PAPcB+cUd4LkwbDivK5RFTk42ny/6jE6du4S7KCF1OCuLe4beyb0jRpGYmBju4phQET+2MAtaL6qI9AAyVXWliFxW3HGeC8P+novfM08mJSeTviO98HlmRgbJycle3hF4XyxZQtMzmlHrxBNDGjeccnJyGDb0Trp1v5LLO3UOWdxwft/Ha+yiImmoVjBLejFwlYhsBt4COojIm4EO0rxFS7Zs2Uxa2lZysrOZP28ul7bvEOgwXs3/aO5x1TxVVcaOGU2jRo24aWCJK7cFVDi/7+M1dlGR1EQNWg1OVUcCIwHcGtxwVR0Q6DhxcXGMHD2Gwam3kZ+fR6/efWncuEmgwxTrt8OHWfbVUh4YMy5kMQHuHz6MFcu/Yd++vXTq0I7Btw+hT99rQhJ79aqVfDjnfZqcdhrX9ukJwJChw2jb7tKgxw7n9328xj5K+POWz8RZajDIQf5McF5v2ipNEzUQbE0Gc7yoGFf29FR/yByf/2C2PHdVWH/BQzKSQVUXAYtCEcsYE1zloenpKxuqZYzxiyU4Y0zUiqSxqJbgjDF+sRqcMSZqWYIzxkStCMpv0TFdkjEmdAJ9o2/RGYdEpKGILBORTSIyU0Ti3f0nuM83ua83KOncluCMMX6JiRGfNx8VnXFoAvC0qjYG9gKD3P2DgL3u/qfd47yX1edPZYwxOE1UX7eSz3XkjEPiVPs6AO+4h0wBermPe7rPcV/vKCVUEy3BGWP84k8NznO2IHdLLXK6ZzhyxqFawD5VzXWfpwEFkyyeDGwFcF/f7x5frHLVyRCCUWPH9EdusbM5BV3np5eELfaL/c8JW+zm9aqGLXYkXSQvj/z5+XnOFnT0eXybcagsylWCM8aUfwG8TaRgxqFuQEWgKjARqC4icW4trR6wzT1+G5ACpIlIHFAN2O0tgDVRjTF+CdQ1OFUdqar1VLUBcB3wmareACwErnYPuxl43308x32O+/pnWsJsIVaDM8b4JQQTXt4PvCUiDwOrgcnu/snANBHZBOzBSYpeWYIzxvglGNcwPWccUtWfgfOPcczvgF+THlqCM8b4xYZqGWOiVgTlN0twxhj/RFINrsSrhSLSR0T+JyL7ReSAiBwUkQOhKJwxpvwJ5EiGYPOlBvc4cKWqBm11emNM5IikdUR8SXAZltyMMQUiqYnqS4JbISIzgf8CfxTsVNX3glUoY0z5FUH5zacEVxU4DHguX65AuUhwf/zxB7fefAM52dnk5uVxeacr+PsddwYtXkb6Dsb9YyR7du9CROjV91r69b8RgFkz3uTdWTOIiYnhoraXMmTo8DLFio+NYdJNragQG0NcjPDp9zuZtHgzk25sRcIJzldXo3IFNmw/yL3vrGdAmxS6tnBWO48VocGJlen89FIO/J7rLUyxJj01ntXLvqBq9RpMeHkmANNfmciqZUuIi6tA8kn1SB02hoTEKuxM3869qddSt159ABo3bcmgO0eW6fN7k5eXR/9+fUlKSua5F14OWpyili5ZzITHHiE/L5/efa9h0F+Kjh2PztieoqoGp6qlXrrcXdX+IJAH5Kpq69Keqzjx8fG88toUKldOICcnh1tu6s8lbdtx5lmtAh0KgNjYOO4cdh9Nz2hGVlYWA/tfzfkXXMiePbtZvOgzps2cTXx8PHv2eB0i55PsvHwGv/ktv+XkERsjvHrT2Xy5aQ+p09YUHjOhb3M+/3EXAG9+vZU3v94KQNsmtbj+/HqlTm4AbTv1oNOV1/LSEw8W7mtxzgX0u/V2YmPjmDH5OebMfIPrBw0BILnuyfzzhemljueP6W9OpWGjU8k6dCgk8cBJqo8+Mp6XX3md5ORk+ve7msvad+DUxo2jOnZREZTffOpFrScis0Uk093ededw8lV7VW0VjOTmlo/KlRMAyM3NJTc3N6j/wpxYuzZNz2gGQEJCAg0aNiJzZybvvf0WN91yG/Hx8QDUrOl1Fhef/ZaTB0BcjBAXK6jH2tgJ8bG0PqU6n/+w66j3dW6WxIINmWWKfUbLc0iscuSsH2ee24bYWOffxcZNW7BnV0aZYpRGRno6SxYvok/fq0s+OIDWr1tLSsop1EtJoUJ8PF26dWfRwk+jPnZRQZjwMnhl9eGY13EGuZ7kbh+4+8qNvLw8ru3bkw7tLqLNhRfR8syzQhJ3+/Zt/PjDRlq0OJMtv27m29UrufXGfgwedBPfbVgXkBgxAv+5rTUL7r6YZT/vZcP2g4WvXXr6iSzfvI+s7Lwj3nNCXAwXnlqTz77fGZAyFOfzBXM4q/VFhc93pm9n1O038NC9qXy/fnXQ4v5rwqMMHXYvIqGdKyIzI4M6desUPk9KTiYjIzQJPpyxiwr0lOXB5MtvSG1VfV1Vc93tDaC2j+dXYIGIrDzGRHcAR0yIN/nVY04bVaLY2Fhmvfs+H3/6OevXrWXT/34s1Xn8cfhwFiOH38XQ4SNJSEwkLy+P/fv3M3nqW9xx93BG3zeMEiY68Em+wg2vrqD7s1/R/KQqnFo7ofC1K5on8fGGo3/J2zWpxdq0/WVqnpbkvzNeIzY2jos7dAWges0TmTjtAx79938YkHo3/37sAQ5nBb75uHjRQmrUrEmz5i0Cfm7jm2hLcLtFZIC7MESsiAyghDmYPFyiqucAXYHbRaRd0QNUdZKqtlbV1oNuK9tF06pVq3Le+Rew9IvgTiKZm5PDyOFDuaJrD9p37ARAUnId2nfshIjQvMWZxMTEsG/v3oDFPPRHLit/3ceFjWoCUK1SBZrVrcrSTXuOOrZT8yQ+LmPz1JvPF3zA6mVf8Pf7Hir8Ja4QH0+VqtUBaNjkDJLr1iN925aAx16zehWfL/qMrp07MOLeYSz/5mtG3V+2zhxfJSUnk74jvfB5ZkYGycnJUR+7qEi60deXBHcrcC2QDuzAmYfJp44HVd3m/j8TmM0xZggoqz179nDggDOw4vfff+frr76kYcNGgQ5TSFV5ZNw/aNCwEf1vHFi4v91lHVi5/BsAtvy6mZycHKrXqFGmWNUrVyDR7S09IS6G8xvWYPPuwwB0PKM2X2zaTXbekbMRJ5wQyzn1qxd2PATatyu+5MN3pnHP2Cc5oWLFwv0H9u0lP89pKmfuSCN9+1aS6p5c3GlK7c6772HBp4v5aMFnPPavpzjv/DY8OuGJgMc5luYtWrJly2bS0raSk53N/HlzubR9h6iPXVQk1eB86UX9FbjK3xOLSAIQo6oH3cedgfH+F9G7XTsz+cfoEeTn5ZGvSucrutDusvaBDlPo2zWr+GjuHE5tcho39usNwOA7hnJlrz48PPYB+l99FXEVKjBm/KNl/oJPTIxn7JVNiREhRoT/25jJF5ucynPnZklM+fLoGlL702uz7Oe9/J5T9mnYn//naDauXcnBA/u4Y0B3rh6QypyZb5CTk80/R90O/Hk7yPfrV/PO1JeIjYsjRmK4dcgIEqtUK3MZypO4uDhGjh7D4NTbyM/Po1fvvjRu3CTqYxdVDvKWz6S460Qicp+qPi4izwFHHaSqXm82E5FGOLU2cBLpdFV9xNt7fss5Ok4o/J6TV/JBQWJrMoReJP2BBlrFOMr86Ts+95XPf6efDrkwrD9tbzW4guFZK0pzYnfSutB0ZxpjQiYmgv6FKDbBqeoH7sPDqvq252si4tesmsaY6BFB+c2nToZjjbcJ3hgcY0y5FhWdDCLSFegGnCwiz3q8VBUI3g1WxphyrRwMUPCZt2tw23Guv10FrPTYfxC4O5iFMsaUX+VhCJavvF2D+xb4VkRmA1mqmgcgIrHACSEqnzGmnJGyd8SGjC/X4BYAlTyeVwL+LzjFMcaUdzHi+xZuvswHV1FVCwcVquohEakcxDIZY8qx8tB54CtfanBZIlJ4R6iInAv8FrwiGWPKs0gai+pLDW4o8LaIbAcEqAP0C2ahjDHlV1Tc6FtAVZeLSFPgdHfXD6qaE4zCaHhGahEbxosFE/q2DFvstn1GhS32Z28/HLbY4RwmFh8X2jnsgiEqelFFpIOqfiYifYq8dJqI2KIzxhynIqgC57UGdynwGXDlMV4rN4vOGGNCKyqaqKr6oPv/Ui86Y4yJPpGT3rw3UYd5e6OqPhX44hhjyrtA3SYiIhWBxTgDB+KAd1T1QRFpCLwF1MIZRXWjqmaLyAnAVOBcnFnF+6nqZm8xvF3xrOJurYHBwMnu9jcgfBOJGWPCKoA3+v4BdFDVs4BWQBcRaQNMAJ5W1cbAXmCQe/wgYK+7/2n3OK+8NVHHAYjIYuAcVT3oPh8LzC2x6MaYqBSoXlR1ZtstGERQwd0U6AD0d/dPAcYCLwI93ccA7wDPi4iol9WdfOmzTgayPZ5nu/uMMcchf6ZL8lw1z91Si5wrVkTWAJnAJ8BPwD5VLZixKA2n5Yj7/60A7uv7cZqxxfLlRt+pwDfuoHuAXjhZ1RhzHPKnAqeqk4Bi1wN1J/FoJSLVcZY4aFrG4h3Blxt9HxGRj4C27q5bVDV4q/oaY8q1YIxFVdV9IrIQuBCoLiJxbi2tHrDNPWwbkAKkiUgcUI0SljD19bbqysABVZ3onrxhaT6EMSbyiR+b1/OI1HZrbohIJaATzlowC3GWJwW4GXjffTzHfY77+mferr+BDzU4EXkQpyf1dOB1nAuBbwIXl/ReY0z0CeDQxrrAFHeOyRhglqp+KCLfAW+JyMPAamCye/xkYJqIbAL2ANeVFMCXa3C9gbOBVQCqul1Eqvj9UYKoW+cOJCQkEBMTS2xsLNNnvRu0WOPHjOaLxYuoUbMmM99z1uX54fuNPPbwWP7IziYuNpb7R42hecszAxJv6rOPsG7FUqpUq8GY5/4DwIczXuWLBXOoUs1ZWLrngL/SovVFAKRt3sT0Fybw++HDSIww4onJVIgv/fyk388dx8GsP8jLzyc3L59Lbnic0X/txq19LmLnXqcD7MHn5/DxF99Rv25N1rz3AD/+mgnAN+s2c+cjb5Uq7uRnHmLNN0upWr0Gj7wwA4B3p73E6q+XICJUrV6D2+4eQ41atck6eIDJEx8mc8c2KsTHM+iuB6jX4NRSf2ZPof6+vVm6ZDETHnuE/Lx8eve9hkF/SS35TUEQqCaqqq7FyS1F9//MMRaJV9XfAb8WvPIlwWWrqoqIQuGCzj5xq5+vAi1wun9vVdWv/Cmgrya9NpUaZVxJ3hc9evbi2uv78+DoEYX7nnv6CW772+1cfEk7li75nGefeYKXJ08NSLwLO3bjsu5X88YzR66Z3fGq6+jUu/8R+/LycnnjqXEMvHsM9Ro24dCB/cTG+vIVe9cldSK792Udse+5NxfyzLRPjzr257RdtLnusTLHvOTyHnTscQ2vPDWucF+3vgPoe+PfAPhkzkzenzGZgXeM4INZb1C/0Wnc+cDjbN+6mWkv/ov7H/13mcsAof++i5OXl8ejj4zn5VdeJzk5mf79ruay9h04tXHjoMY9lggaqeXTNbhZIvIyzoW/v+DM5vuKj+efCMxX1aY4a6RuLOH4cu+cc8+jatXqR+wTEbIOObWZQ4cOUbt2UsDiNWl+NgmJvs1+sXH1N5zc4FTqNXRWPE+sWo2Y2NiAlSWUTm9xNglVjvzclSonFj7+4/ffCmsS27f8whlnngvASSkN2JWxg/17vV579lmov+/irF+3lpSUU6iXkkKF+Hi6dOvOooVH/wMTCjEiPm/h5vWfd3F+g2bidN0ewLkON0ZVPynpxCJSDWgHDARQ1WyOvJ8uYESEv6cOQgT6XtOPvteEdrq6YfeNZMjgvzDxqX+h+flMnjo96DEXzXuHZQs/on7jpvS9dQgJiVXJ2L4VRHj2waEcOrCP1m0vp3OfAWWKo6p88MIdqCqT313Ka+8tBeBv17Wjf4/zWfXdFkY89R77DjpzoDY4uRZfzbifg1m/M+7fH7J09U9l/qye3pnyIl9+No9KCYnc/88XAKjfqAkrv1zE6S3O5ucfNrA7M529uzKpVsPrLVKlFo7vOzMjgzp16xQ+T0pOZt3atUGPeyzlIG/5zGsNzu2hmKeqn6jqvao63Jfk5moI7AReF5HVIvLqsZq3njcCvvZqsbfLePX61OnMePs9nn/xFWbOmM7KFctLdZ7SenfWWwy7dwRzFyzk7ntH8NDYB4Iar13XPjz00tuMemYK1WrU4t3XngMgPy+Pn75by633jGX4Yy+x5uvP+f7bFWWK1fGWp7mo/wR63fECf+3XlovPOZVX3l5CsyvHcsF1j5G+6wCPDXNm1ErfdYDTuo7hwusncP+T7/HGowOpklCxzJ/X09U3D+apKR9w4WVX8OkHznrk3a+5icNZB/nHHQP45INZnHLqaUhM8Gquof6+y5tIWhfVlybqKhE5rxTnjsMZs/qiqp4NZAEjih6kqpNUtbWqtr71ttJdNE1KdgZW1KxViw4dL2fDutD+y/bhB/+lfcdOAFzeuQvfrV8X1HhVq9ckJjaWmJgYLunck83/+w6A6rVq07h5KxKrVif+hIq0OPcitvz0Q5libd+5H4Cdew8x57O1nNe8AZl7DpKfr6gqr723lNYtTgEgOyeXPfuda3WrN27l57RdNDklOM23Cy/rwoovFwJO0/W2u8fw0PNvknrPWA7s30dS3ZOCEhdC/32D8zueviO98HlmRgbJyeEZUBQr4vMWbr4kuAuAr0XkJxFZKyLrRMSXDJIGpKnqMvf5OwRhkP5vhw+TlXWo8PFXXy7l1CanBTqMV7VrJ7HKrTUu/+ZrUuqfEtR4+/fsKny85uvPOal+IwCanXMB23/9iew/ficvL5cf16+mbv0GpY5TuWI8iZVPKHx8+YVN2fDTduqc+Oe1sZ4dzuK7n3YAcGKNxMJxig1OrkXj+rX5JW3X0ScupfRtWwofr/p6MXXrOT/nrEMHyc1xJpn+/OP3Ob1FqyOu1wVaqL9vgOYtWrJly2bS0raSk53N/HlzubR9h6DHPZZoW1XritKcWFXTRWSriJyuqj8AHYHvSnMub3bv3s2wu+4AnJ6mrt16cPElbUt4V+mNvv8eVq74hn379tG902WkDr6D0WPG8+Tjj5KXl0d8/AmMGjO+5BP5aPITY/hx/WoOHdjHyFt70uP62/hx/SrSfvkfglAzqS43/P0+ABISq9Kx53U8ds8gEGhx7kW0bF362xWTalVh5lN/ASAuNpaZH63gky83Mvmhmzjz9HqoKr/u2MOQh53bOC45pzH/GNydnNw88vOVIY+8xd4Dh0sV+8UJD/D9ulUcOrCPu2/qQa8bUlm7Yinp27YgEkOtpDoMvP1+AHZs3cwrT41DRDi5fiNuvWt0qT9zUaH+vosTFxfHyNFjGJx6G/n5efTq3ZfGjZsEPe6xlIfE5Ssp7kZgEUkCRgGNgXXAP1X1gF8nF2mFc5tIPPAzzjCvvcUdfzjH+13JwZKbF561IABWbC72xxF03a9/MGyxbU2G8KgYV/b5Ku/54Aef/2CevPL0sKZDbz/tqTjXzZ4DEoFn/T25qq5xr6+dqaq9vCU3Y0xkiJYmal1VLajrfywiq0JRIGNM+VYO+g58VtJ9cDX4c8xsrOdzVd0T5LIZY8qhuAjKcN4SXDWc+dA9P01BLU6BRsEqlDGm/Iqg/OZ1yvIGISyHMSZClIchWL4q+0hsY8xxJYLymyU4Y4x/ykPvqK8swRlj/BLACS+DztvCzzW9vdF6UY05PkVQfvNag1uJ01t6rI9jvajGHKek7IMhQsZbL2rIF5YJV+9MfFz4vrCLGgdnzjJfbF86MWyxf8rIKvmgIImkXsDyKJJqcCUOjBPHABH5h/u8vogcNV+6Meb4EElDtXwZ+fsCzlqFBQsAHAQCM+G9MSbiRNKEl770ol6gqueIyGoAVd0rIvFBLpcxppyKDe+EKH7xJcHluOsWFqyqVRvID2qpjDHlViRdw/QlwT0LzAaSROQRnBWlj69J6I0xhcrDtTVflZjgVPU/IrISZ0ZeAXqpasQv/2eMKZ0IqsD5fKNvJjDD8zW70deY41NMNNwHx5E3+tYH9rqPqwNbcJYFNMYcZ6KiBldwo6+IvALMVtV57vOuQK+QlM4YU+7ERdBFOF86fNsUJDcAVf0IuCh4RTLGlGcivm/h5ksv6nYReQB4031+A7A9eEUyxpRn0XabyPXAgzi3igAsdveVG0uXLGbCY4+Qn5dP777XMOgvqSGJm75jB6NH3see3btBhKuvuZYbbrw5JLHHPDCSxZ8vombNWrz3/odBj5eRvoNx/xjJnt27EBF69b2Wfv1vZPT9w9iy+RcADh48SJUqVZg2c3YJZyvZpKfGs3rZF1StXoMJL88EYPorE1m1bAlxcRVIPqkeqcPGkJBYhZ3p27k39Vrq1qsPQOOmLRl058gylwFg3JhRLHF/zrNmfwDAM08+zuLPF1KhQgXqpdRn7PhHqVI1uEsRhvr79iZQ+U1EUnBW70vGud4/SVUnuh2cM4EGwGbgWneAgQATgW7AYWCgqnpdDKvYdVGPUZgqgKrqIR+PP90tZIFGwBhVfaa49/yei98LlObl5XFV9yt4+ZXXSU5Opn+/q3nsX09xauPG/p7Kbzt3ZrJr507OaNacrKxDXHdNX5559t8hib1yxXIqV67M6JH3l/oX/rfsPJ+P3bVzJ7t27aTpGc3IyspiYP+refyp52h46p+fdeKTE0hMrMKgv/69xPOVNNh+47pVVKxYmZeeeLAwwa1d+TXNW7UmNjaOGZOfA+D6QUPYmb6dJx68u/C4kjQ9qYpPxwGsWrGcSpUr8+DoEYUJ7qsvv+C889sQFxfHs08/AcCddw/36XxxsaXLDoH4viEw66K+sXyLz3+nA8+rX2w8EamLs3rfKje/rMS5vj8Q2KOqj4nICKCGqt4vIt2AITgJ7gJgoqpe4C2+L4PtW7rDtNYDG0RkpYi0KOl9qvqDqrZS1VbAuTgZt+z/tBexft1aUlJOoV5KChXi4+nSrTuLFn4a6DDHVLt2Emc0aw5AQkIijRo1IjMzIySxz219HlWrVQtJLIATa9em6RnNAEhISKBBw0Zk7swsfF1V+fSTj+nUpVtA4p3R8hwSqxxZKzrz3DbExjqNjsZNW7BnV/B/1ue0Po9qRX7OF150CXFxTjlanHkWGRnpQS9HqL9vb2JEfN68UdUdBTUwVT0IbAROBnoCU9zDpvBnp2ZPYKo6vgaqu0my+LL68HleBoap6imqegpwDzDJh/d56gj8pKq/+vm+EmVmZFCnbp3C50nJyWRkhCbJeNq2LY3vN26k5ZlnhTx2qG3fvo0ff9hIixZnFu5bs2olNWvWov4pDUJShs8XzOGs1n/2de1M386o22/goXtT+X796pCUAWDO7He5+JJ2IYtXHviT4EQkVURWeGzHvH4kIg2As4FlQLKq7nBfSsdpwoKT/LZ6vC3N3VcsX67BJajqwoInqrpIRBJ8eJ+n6/C4UdiT+4FTAZ5/4eWQXT8LpMNZWdwz9E7uHTGKxMTEcBcnqA4fzmLk8LsYOnwkCR6fdcH8uQGrvZXkvzNeIzY2jos7dAWges0TmTjtA6pUrc4v/9vIU+OGM+HlmVROCO53MXnSS8TGxdG1+5VBjVPe+NPGVdVJlFAhEpFE4F1gqKoe8JyFRFVVRPy+dFXAlwT3szsX3DT3+QDgZ18DuDOPXAUc86qv5w+gNNfgkpKTSd/xZxMhMyOD5ORkL+8IrJycHIYNvZNu3a/k8k6dQxY3HHJzchg5fChXdO1B+46d/tyfm8uiz/6PKdPfDnoZPl/wAauXfcGox14onI6nQnw8FeKdCW4aNjmD5Lr1SN+2hUanNQtaOea8/x5LFi/kxVfeKBfTAoVSID+uiFTASW7/UdX33N0ZIlJXVXe4TdCCayHbgBSPt9dz9xXLlybqrUBt4D13q+3u81VXYJWqBqXd2LxFS7Zs2Uxa2lZysrOZP28ul7bvEIxQR1FVxo4ZTaNGjbhp4C0hiRkuqsoj4/5Bg4aN6H/jwCNeW77sKxo0aEhScp1jvzlAvl3xJR++M417xj7JCRUrFu4/sG8v+XlOh0nmjjTSt28lqa7XlkuZfPnFEqa+Ppmnn32RSpUqBS1OeRWo+eDcXtHJwEZVfcrjpTlAwe0INwPve+y/yZ2Etw2w36Mpe+wYvvailpaIvAV8rKqvl3RsaWpwAEsWf87jjz1Kfn4evXr35S9/HVya0/ht1coV3HLTDTQ57TRixPm3YsjQYbRtd2nQY98/fBgrln/Dvn17qVmrFoNvH0Kfvtf4dQ5/elHXrF7J3269kVObnFZ48XjwHUO5qO2ljB8zihYtz6TPNdf5fL6SelGf/+doNq5dycED+6haoxZXD0hlzsw3yMnJJrGqc7G94HaQb774jHemOs3FGImh742pnNOm+Oti/vSijrpvGCtWLGffvr3UqlmLv/59CK9PnkROdjbVqlcHoOWZZzHqH+N8Ol9pe1ED8X1DYHpRZ67e5vPfab+zT/bWi3oJsARYx59TsI3CuQ43C2eI6K84t4nscRPi80AXnE7LW1R1hbf4xSY4EZnj7Y2qepW3191zJOCMW22kqvtLOr60Cc6Ujj8JLtDCuSaDPwku0Eqb4AIlEAnu7TXbff47vabVSWH9wN6uwV2I02MxAyej+l1QVc0CwreqijEm4CLpmqO3BFcH6IQzaqE/MBeYoaobQlEwY0z5FEEzlhdfVlXNU9X5qnoz0AbYBCwSkTtCVjpjTLkTNYvOiMgJQHecWlwD/py+3BhznAp/2vKdtxl9pwItgHnAOFVdH7JSGWPKrdhyUDPzlbca3AAgC7gLuNOjuik4NxgHd/oEY0y5FEH5zeuMvpF0LdEYEyISQY1UX4ZqGWNMoaiowRljzLFEy6paxhhzFKvBmYhQKT42bLFbpISvjyo7N7/kg4ImgrJDMaJtTQZjjCkUQasGWoIzxvjHelGNMVErglqoluCMMf6xGpwxJmrZNThjTNSyXlRjTNSKnPRmCc4Y4yerwRljolbkpDdLcMYYf0VQhrMEZ4zxSyQ1UaNizrelSxZzVfcr6NGlE5NfmWSxozh2+o4dDBp4I72v7Ebvq7rzn2lTghpv/JjRdL7sYvr1ubJw3w/fb+SWAf3of21vbrr+ajasWxvUMhQI5/ftSfzYwi2oCU5E7haRDSKyXkRmiEjFkt/ln7y8PB59ZDwvvPQqs+fMZf68D/lp06ZAh7HY5SR2bFwsw+8bwewP5vHmjJm8NWN6UGP36NmLZ188Mpk89/QT3Pa325k+azZ//fsQnn3miaDFLxDOn/lRIijDBS3BicjJwJ1Aa1VtAcQCvi997qP169aSknIK9VJSqBAfT5du3Vm08NNAh7HY5SR27dpJnNGsOQAJCYk0atSIzMyMoMU759zzqFq1+hH7RISsQ4cAOHToELVrJwUtfoFw/syLEj/+C7dgN1HjgEoiEgdUBrYHOkBmRgZ16tYpfJ6UnExGRvB+4S12eGN72rYtje83bqTlmWeFNO6w+0by7NNP0L1zeyY++Ti333l30GOWl585OGNRfd3CLWgJTlW3AU8AW4AdwH5VXVD0OBFJFZEVIrIinNcVTGQ5nJXFPUPv5N4Ro0hMTAxp7HdnvcWwe0cwd8FC7r53BA+NfSCk8cMtglqoQW2i1gB6Ag2Bk4AEERlQ9DhVnaSqrVW19aC/pPodJyk5mfQd6YXPMzMySE5OLn3BLXa5jg2Qk5PDsKF30q37lVzeqXPI4hb48IP/0r5jJwAu79yF79avC3rMcP/MPUXSws/BbKJeDvyiqjtVNQd4D7go0EGat2jJli2bSUvbSk52NvPnzeXS9h0CHcZil5PYqsrYMaNp1KgRNw28JSQxi6pdO4lVK5YDsPybr0mpf0rQY4bzZ15UJDVRg3kf3BagjYhUBn4DOgIrAh0kLi6OkaPHMDj1NvLz8+jVuy+NGzcJdBiLXU5ir161kg/nvE+T007j2j49ARgydBht210alHij77+HlSu+Yd++fXTvdBmpg+9g9JjxPPn4o+Tl5REffwKjxowPSmxP4fyZF1UO8pbPRFWDd3KRcUA/IBdYDdymqn8Ud/zvuQSvMMa4wrkmQ3xceG89rRhX9vz07daDPv+dnpVSxWs8EXkN6AFkundbICI1gZlAA2AzcK2q7hWnzTsR6AYcBgaq6ipv5w/qT1tVH1TVpqraQlVv9JbcjDGRIcC3ibwBdCmybwTwqao2AT51nwN0BZq4WyrwYkknj4qRDMaY0AnkNThVXQzsKbK7J1AwRGUK0Mtj/1R1fA1UF5G63s5vCc4Y45cQdDIkq+oO93E6UNBdfDKw1eO4NHdfsSzBGWP84k8T1fM+V3fz614wdToJSn1t3mYTMcb4xZ+amapOAvy9gz9DROqq6g63CZrp7t8GpHgcV8/dVyyrwRlj/BKCkQxzgJvdxzcD73vsv0kcbXBGR+041gkKWA3OGOOfAN4IJyIzgMuAE0UkDXgQeAyYJSKDgF+Ba93D5+HcIrIJ5zaREu/0Dup9cP6y++BMKNh9cGXzQ/phn/9OT69TOaz3BVsNzhjjl0gayWAJzhjjnwjKcJbgzHEn3M3ESFceJrL0lSU4Y4xfysMsIb6yBGeM8UsE5TdLcMYY/5SHiSx9ZQnOGOOXCMpvluCMMf6JoPxmCc4Y46cIynCW4IwxfrHbRIwxUcuuwRljolaMJThjTPSKnAxnCc4Y45dIaqJGxaC8pUsWc1X3K+jRpROTX/F38lCLbbEttj9CMOFl4Khq0DbgLmA9sAEYWtLxv+Wo+rsd+j1XO3TsqP/7eYseyPpDe/S4Utdv/J/f57HYFvt4iB2Iv+vt+/5QX7dg5hdftqDV4ESkBfAX4HzgLKCHiDQOdJz169aSknIK9VJSqBAfT5du3Vm08NNAh7HYFvu4jV2UiPi8hVswm6hnAMtU9bCq5gKfA30CHSQzI4M6desUPk9KTiYjIyPQYSy2xT5uYxcVSU3UYCa49UBbEaklIpVx5lJPKXqQ57Ji4byuYIzxTQjWRQ2YoPWiqupGEZkALACygDVA3jGOK1xWrDRrMiQlJ5O+I73weWZGBsnJyV7eETgW22IfD7GLiqSRDEHtRVXVyap6rqq2A/YCPwY6RvMWLdmyZTNpaVvJyc5m/ry5XNq+Q6DDWGyLfdzGPkoEtVGDeh+ciCSpaqaI1Me5/tYm0DHi4uIYOXoMg1NvIz8/j169+9K4cZNAh7HYFvu4jV1UOchbPgvqsoEisgSoBeQAw1TVa7ePLRtoTHAFYtnAPVl5Pv+d1kyIjd5lA1W1bTDPb4wJvfLQeeCrqBjJYIwxx2JjUY0xfomkGpwlOGOMXyLpNhFLcMYYv1gNzhgTtSzBGWOiljVRjTFRK5JqcHabiDHGL4EcqSUiXUTkBxHZJCIjAl1WS3DGGP8EKMOJSCzwb6Ar0Ay4XkSaBbKo1kQ1xvglJnBt1POBTar6M4CIvAX0BL4LVIByleDKMk5ORFLdqZdCzmJb7OMhdgF//k5FJBVI9dg1yaP8JwNbPV5LAy4oewn/FE1N1NSSD7HYFttih5KqTlLV1h5bSJNzNCU4Y0xk2caRs3zXc/cFjCU4Y0y4LAeaiEhDEYkHrgPmBDJAuboGV0bhvC5hsS328RA7oFQ1V0TuAD4GYoHXVHVDIGMEdcJLY4wJJ2uiGmOiliU4Y0zUivgEJyIVReQbEflWRDaIyLgQx48VkdUi8mEo47qxN4vIOhFZIyIrQhy7uoi8IyLfi8hGEbkwRHFPdz9vwXZARIaGIrYb/27392y9iMwQkYohjH2XG3dDKD9zJIv4a3AiIkCCqh4SkQrAF8Bdqvp1iOIPA1oDVVW1RyhiesTeDLRW1V2hjOvGngIsUdVX3R6wyqq6L8RliMW5reACVf01BPFOxvn9aqaqv4nILGCeqr4RgtgtgLdw7v7PBuYDf1PVTcGOHckivganjkPu0wruFpKsLSL1gO7Aq6GIV16ISDWgHTAZQFWzQ53cXB2Bn0KR3DzEAZVEJA6oDGwPUdwzgGWqelhVc4HPcZbiNF5EfIKDwmbiGiAT+ERVl4Uo9DPAfUB+iOIVpcACEVnpDokJlYbATuB1t3n+qogkhDB+geuAGaEKpqrbgCeALcAOYL+qLghR+PVAWxGpJSKVgW4ceZOsOYaoSHCqmqeqrXDuhD7frc4HlYj0ADJVdWWwY3lxiaqegzMbw+0i0i5EceOAc4AXVfVsIAsI+FQ33rjN4quAt0MYswbOYPCGwElAgogMCEVsVd0ITAAW4DRP1wB5oYgdyaIiwRVwm0kLgS4hCHcxcJV7HewtoIOIvBmCuIXcGgWqmgnMxrk+EwppQJpHTfkdnIQXSl2BVaqaEcKYlwO/qOpOVc0B3gMuClVwVZ2squeqajtgL/BjqGJHqohPcCJSW0Squ48rAZ2A74MdV1VHqmo9VW2A01T6TFVD8q85gIgkiEiVgsdAZ5xmTNCpajqwVUROd3d1JIBT3PjoekLYPHVtAdqISGW3c6sjsDFUwUUkyf1/fZzrb9NDFTtSRcNQrbrAFLdHLQaYpaohv2UjDJKB2c7fGXHAdFWdH8L4Q4D/uE3Fn4FbQhXYTeidgL+GKiaAqi4TkXeAVUAusJrQDp16V0RqATnA7WHq2IkoEX+biDHGFCfim6jGGFMcS3DGmKhlCc4YE7UswRljopYlOGNM1LIEZwAQkV4ioiLStITjhrpDhUobZ6CIPF/a9xvjD0twpsD1ODNlXF/CcUNxBpkbU+5ZgjOISCJwCTAIZ1RGwQQGT7jzj60VkSEicifOGMyFIrLQPe6Qx3muFpE33MdXisgydzD+/4lIcqg/lzHRMJLBlF1PYL6q/igiu0XkXJxxrQ2AVu7iIDVVdY87/117H+ag+wJoo6oqIrfhzLpyTzA/hDFFWYIz4DRLJ7qP33KfNwRecuceQ1X3+HnOesBMEakLxAO/BKisxvjMEtxxTkRqAh2AliKiOMu3Kc6alb7wHOvnOX33c8BTqjpHRC4Dxpa5sMb4ya7BmauBaap6iqo2UNUUnNrWt8Bf3ZlrCxIhwEGgisf7M0TkDBGJAXp77K/Gn6uU3xzUT2BMMSzBmetx5pLz9C7OLC1bgLUi8i3Q331tEjC/oJMBZ6LLD4EvcWa5LTAWeFtEVgIhXzPCGLDZRIwxUcxqcMaYqGUJzhgTtSzBGWOiliU4Y0zUsgRnjIlaluCMMVHLEpwxJmr9P4emWxqUEgqWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ml_utils.classification import confusion_matrix_visual\n",
    "\n",
    "confusion_matrix_visual(y_test, preds, np.sort(wine_y.unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<div>\n",
    "    <a href=\"./exercise_4.ipynb\">\n",
    "        <button>&#8592; Previous Solution</button>\n",
    "    </a>\n",
    "    <a href=\"../../ch_11/1-EDA_unlabeled_data.ipynb\">\n",
    "        <button style=\"float: right;\">Chapter 11 &#8594;</button>\n",
    "    </a>\n",
    "</div>\n",
    "<hr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}