a5-logit-and-rf-for-credit-scoring.ipynb

{"cells":[{"metadata":{"_uuid":"b61a17511d9e5e3ccb732c0210ba185c169f69ac"},"cell_type":"markdown","source":"<center>\n<img src=\"https://habrastorage.org/files/fd4/502/43d/fd450243dd604b81b9713213a247aa20.jpg\">\n## Open Machine Learning Course [mlcourse.ai](https://mlcourse.ai)\n<center>\n\nAuthor: Vitaly Radchenko (@vradchenko). This material is subject to the terms and conditions of the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. Free use is permitted for any non-commercial purpose."},{"metadata":{"_uuid":"d1a67717ad3317664bf3d108f1f0b0480dd79c63"},"cell_type":"markdown","source":"# <center>Assignment # 5 (demo)</center>\n## <center>Logistic Regression and Random Forest in the credit scoring problem</center>  "},{"metadata":{"_uuid":"7dbf48ad8bbb6a239269064331db677929a5fa3c"},"cell_type":"markdown","source":"In this assignment, you will build models and answer questions using data on credit scoring.\n\nPlease write your code in the cells with the \"Your code here\" placeholder. Then, answer the questions in the [form](https://docs.google.com/forms/d/1gKt0DA4So8ohKAHZNCk58ezvg7K_tik26d9QND7WC6M/edit).\n\nLet's start with a warm-up exercise."},{"metadata":{"_uuid":"5ffc2f53b8f50650fd793336a20a9f0807b8529a"},"cell_type":"markdown","source":"**Question 1.** There are 5 jurors in a courtroom. Each of them can correctly identify the guilt of the defendant with 70% probability, independent of one another. What is the probability that the jurors will jointly reach the correct verdict if the final decision is by majority vote?\n\n1. 70.00%\n2. 83.20%\n3. 83.70%\n4. 87.50%"},{"metadata":{"_uuid":"7ff86c79a045fcfc6bf8211e539c3ab75a45181b"},"cell_type":"markdown","source":"Great! Let's move on to machine learning.\n\n## Credit scoring problem setup\n\n#### Problem\n\nPredict whether the customer will repay their credit within 90 days. This is a binary classification problem; we will assign customers into good or bad categories based on our prediction.\n\n#### Data description\n\n| Feature | Variable Type | Value Type | Description |\n|:--------|:--------------|:-----------|:------------|\n| age | Input Feature | integer | Customer age |\n| DebtRatio | Input Feature | real | Total monthly loan payments (loan, alimony, etc.) / Total monthly income percentage |\n| NumberOfTime30-59DaysPastDueNotWorse | Input Feature | integer | The number of cases when client has overdue 30-59 days (not worse) on other loans during the last 2 years |\n| NumberOfTimes90DaysLate | Input Feature | integer | Number of cases when customer had 90+dpd overdue on other credits |\n| NumberOfTime60-89DaysPastDueNotWorse | Input Feature | integer | Number of cased when customer has 60-89dpd (not worse) during the last 2 years |\n| NumberOfDependents | Input Feature | integer | The number of customer dependents |\n| SeriousDlqin2yrs | Target Variable | binary: <br>0 or 1 | Customer hasn't paid the loan debt within 90 days |\n"},{"metadata":{"_uuid":"258acae393c2c65de1b7c1ac581f0917009f58b9"},"cell_type":"markdown","source":"Let's set up our environment:"},{"metadata":{"trusted":true,"_uuid":"958b2cd2a45ac297fb144c47f2e993e0a5fab169"},"cell_type":"code","source":"# Disable warnings in Anaconda\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport numpy as np\nimport pandas as pd\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set()","execution_count":1,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"57846c0c4a90698c2d1e4577a1d338566a0511d2"},"cell_type":"code","source":"from matplotlib import rcParams\nrcParams['figure.figsize'] = 11, 8","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"2e2cb4aaa7a07017c00bfa8621a8c1ff91e980e0"},"cell_type":"markdown","source":"Let's write the function that will replace *NaN* values with the median for each column."},{"metadata":{"trusted":true,"_uuid":"d4b4c073dc7d54908594df01f032b249b2ee461e"},"cell_type":"code","source":"def fill_nan(table):\n    for col in table.columns:\n        table[col] = table[col].fillna(table[col].median())\n    return table   ","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"c0961fa0182a6b2fc8cda93c5a6e2112234e5fba"},"cell_type":"markdown","source":"Now, read the data:"},{"metadata":{"trusted":true,"_uuid":"63c902c7d1d18943bd460ee15c73d8a10e5d8ff3"},"cell_type":"code","source":"data = pd.read_csv('../input/credit_scoring_sample.csv')\ndata.head()","execution_count":4,"outputs":[{"output_type":"execute_result","execution_count":4,"data":{"text/plain":"   SeriousDlqin2yrs  age         ...          MonthlyIncome  NumberOfDependents\n0                 0   64         ...                 8158.0                 0.0\n1                 0   58         ...                    NaN                 0.0\n2                 0   41         ...                 6666.0                 0.0\n3                 0   43         ...                10500.0                 2.0\n4                 1   49         ...                  400.0                 0.0\n\n[5 rows x 8 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>SeriousDlqin2yrs</th>\n      <th>age</th>\n      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n      <th>DebtRatio</th>\n      <th>NumberOfTimes90DaysLate</th>\n      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n      <th>MonthlyIncome</th>\n      <th>NumberOfDependents</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>64</td>\n      <td>0</td>\n      <td>0.249908</td>\n      <td>0</td>\n      <td>0</td>\n      <td>8158.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0</td>\n      <td>58</td>\n      <td>0</td>\n      <td>3870.000000</td>\n      <td>0</td>\n      <td>0</td>\n      <td>NaN</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0</td>\n      <td>41</td>\n      <td>0</td>\n      <td>0.456127</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6666.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0</td>\n      <td>43</td>\n      <td>0</td>\n      <td>0.000190</td>\n      <td>0</td>\n      <td>0</td>\n      <td>10500.0</td>\n      <td>2.0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>49</td>\n      <td>0</td>\n      <td>0.271820</td>\n      <td>0</td>\n      <td>0</td>\n      <td>400.0</td>\n      <td>0.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"474023002b2a33fd2981180400c2f71050577c7d"},"cell_type":"markdown","source":"Look at the variable types:"},{"metadata":{"trusted":true,"_uuid":"8fdf8aa51c7febd4ee2d7914129888b01703b716"},"cell_type":"code","source":"data.dtypes","execution_count":5,"outputs":[{"output_type":"execute_result","execution_count":5,"data":{"text/plain":"SeriousDlqin2yrs                          int64\nage                                       int64\nNumberOfTime30-59DaysPastDueNotWorse      int64\nDebtRatio                               float64\nNumberOfTimes90DaysLate                   int64\nNumberOfTime60-89DaysPastDueNotWorse      int64\nMonthlyIncome                           float64\nNumberOfDependents                      float64\ndtype: object"},"metadata":{}}]},{"metadata":{"_uuid":"4833aaafffd639f230f63c9e42e4919cbbd6210a"},"cell_type":"markdown","source":"Check the class balance:"},{"metadata":{"trusted":true,"_uuid":"35216e0b707d378f851ea1bc9602890998e1ec97"},"cell_type":"code","source":"ax = data['SeriousDlqin2yrs'].hist(orientation='horizontal', color='red')\nax.set_xlabel(\"number_of_observations\")\nax.set_ylabel(\"unique_value\")\nax.set_title(\"Target distribution\")\n\nprint('Distribution of the target:')\ndata['SeriousDlqin2yrs'].value_counts()/data.shape[0]","execution_count":6,"outputs":[{"output_type":"stream","text":"Distribution of the target:\n","name":"stdout"},{"output_type":"execute_result","execution_count":6,"data":{"text/plain":"0    0.777511\n1    0.222489\nName: SeriousDlqin2yrs, dtype: float64"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 792x576 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAqYAAAH1CAYAAAAkiTxbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XucXHV9//HXJgELJBVdF7kYSBHy4acVFaFoFSwWLKK0iiBGBO+CbaHFn7VWK0WU6s+iVkosVKpSLsELFSlFsdZaRKUictPKh6hcYgKyRMEECEJ2f3+cExnW3c1McnbmuzOv5+Oxj5051898Mpm8c873nBkaHx9HkiRJ6rU5vS5AkiRJAoOpJEmSCmEwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSeqyiNgjIh5uef7ViDiyoW0fGBHXtzy/MyKe18S26+39KCKe09T2JKnVvF4XIEmdioi1LU+3Bh4E1tfPj83M87tYy28ADwALM/Mnm7KNzHxBU/vJzK8AT9+UOibZ54XA9zLzfS3bf3IT25akyRhMJc06mTl/w+OIuBV4Yx3IOhYR8zLz4Y0vWb5+ei2SBpPBVFLfiYjnAh8GArgf+DTwF5n5cMuRxz8G3gY8BOwRES8G/h4YAT4FPBs4IzPPq7d5LPDWev63gDdn5krginq3GRHjwKsz8+IJ9cwDPgS8Gvg58NEJ86/asK+I2AM4G3ga8Evgi5l5zGT7AdYBZwDnAH8KXBIRF9Xb2q1lF78bEWcD2wGfA47PzAcj4jjg8Mw8sK7jV0dlgcOAlwMvi4h31HUcERF31utcGRFbAafVy64HlgHvzMyHIuLgurZPACfWr+Xt3TyaLWn2cYyppH70EFVQGwb2Aw4F3jhhmZcAzwKeGRE7UIXXE6mC56p6HgD1+M8/r7fzROBa4Lx69v4bFsvM+RNDae1PgRdQhc3nANONJ30/cDGwLbAzcNZG9rMI2IIqTJ4wxTaX1PsP4JnAX0yzfwAy83TgIuC99f6OmGSx9wB71q/rWcDvAW9vmb8LMATsSNWDMyNiPpI0BYOppL6Tmd/OzKszc31m/ojqCOTzJyx2ambek5kPUAXOqzPz0sx8iOoo4M9blj0OeF9m3lzPfw/wvIh4YpslvQL4UGauysxR4IPTLPsQVdjcPjMfyMxvbGTbD1KFx1/Wr2UyH23Z9/upgmoTjgL+JjPvzsyfAu8Djm6Zfz/w/sx8KDM/D4wDu02yHUkCPJUvqQ9FxFOoTp3vBWxF9Vk3MeCtaHm8Y+vzzByLiJUt83ehOtq3tGXaw8CTgHvbKOlR2wdum2bZE4H3AtdGxF3ABzcMJ5jCnXVYns7Efe+4keU3KiKGgO159Gu5Ddip5floZo61PL8f8IippCl5xFRSP/o48F3gyZn5m8ApVKeUW423PL6DKmQCEBFzeHTAWgG8NjO3bfnZKjOvmbCdqdxBdap9g52nWjAzV2bm64EdqE7NfyIidp5mP+3sf+K+V9WP76O6q8EG27e77cwcB+6kCu2t2145+RqStHEGU0n9aAFwb2aujYinAm/ayPKXAPtGxCH1hUpvBR7XMv9M4K8jIgAi4nER8XKAzHyQ6qjprtNs/zPAiRGxQ0Q8gUePw3yUiDgyInasg9899eT1be5nKie07PsdVONpAa6jGmP71IjYGjhpwno/3cj+lgF/ExHDEbEd8C4eGXsrSR0zmErqRycCb6zvd7qUR4LYpDLzDqpxl6cDd1MdPb2RavwmmbmM6grzf42IX1AFuoNaNnES8NmIuCci/nCSXZwBfB34PvA/VEF1Ks8Brqlr/yyPXP3fzn6mciHwX8Dy+nV9sH5dGx5/HbgJ+NqE9f4J2Kfe34WTbPck4H/r13Ud1XCJ6cbPStK0hsbH2zkLJEmDoz5qeidwaGZ+q9f1SNKg8OInSQIi4kXAN6mOkr6L6kKda3palCQNGE/lS1Jlf+AW4C7g94GXZeYve1uSJA0WT+VLkiSpCP10Kv8xwD5Ut2VZ3+NaJEmSBtlcqtveXU19IWk7+imY7kN1ZakkSZLKsB9wZbsL91MwvQPg5z+/j7ExhydsruHh+axevbbXZfQN+9kce9ks+9kce9ks+9mcXvRyzpwhHve4baDOZ+3qp2C6HmBsbNxg2hD72Cz72Rx72Sz72Rx72Sz72Zwe9rKj4ZVelS9JkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUWY142dRMRpwMuBRcDTMvN7kywzFzgdOBgYBz6QmWd3oz5JkiT1XreOmF4M7A/cNs0yRwG7AbsDzwFOjohFM1+aJEmSStCVYJqZV2bmio0sdiTw8cwcy8xRqjB7xMxXJ0mSpBJ05VR+m3bm0UdUbwcWdrqR4eH5jRU06EZGFvS6hP6xbp39bJC9bJb9bI69bJb9bM5s6WVJwbQZixbBbdONGJB6YHyc0dE1va6iL4yMLLCXDbKfzbGXzbKfzelFL+fMGdqkg4UlXZV/O7BLy/OdgY2d/pckSVKfKOmI6WeBN0XEvwLDwEuB/XpbkiRJkrqlK0dMI+L0iPgJ8CTgKxHx/Xr6ZRGxd73YucCPgeXAVcApmXlLN+qTJElS73XliGlmngCcMMn0Q1oerwfe0o16JEmSVJ6SxphKkiRpgBlMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUhHm9LqBxt97a6wqkX7duXa8rkCSpeH0XTFevXsvY2Hivy5j1RkYWMDq6ptdl9I2RkQWw5qFelyFJUtE8lS9JkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklSEed3aUUQsBs4BhoHVwDGZuXzCMtsBnwQWAlsA/wWckJkPd6tOSZIk9UY3j5ieCSzNzMXAUuCsSZZ5J/CDzNwT2BN4FnBY90qUJElSr3QlmNZHQvcCltWTlgF7RcTIhEXHgQURMQd4DLAlsLIbNUqSJKm3unUqfyGwMjPXA2Tm+ohYVU8fbVnuvcBFwB3ANsAZmfmNTnY0PDy/mYrFyMiCXpfQV+xnc+xls+xnc+xls+xnc2ZLL7s2xrRNRwA3AL8PLAC+GBGHZ+bn2t3A6tVrGRsbn6n6BsbIyAJGR9f0uoy+YT+bYy+bZT+bYy+bZT+b04tezpkztEkHC7s1xnQFsFNEzAWof+9YT291PHB+Zo5l5r3AF4ADulSjJEmSeqgrwTQz7wKuA5bUk5YA12bm6IRFbwEOBoiILYEDge91o0ZJkiT1Vjevyj8OOD4ibqY6MnocQERcFhF718v8ObBfRNxIFWRvBj7exRolSZLUI10bY5qZNwH7TjL9kJbHPwIO6lZNkiRJKoff/CRJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRZjXrR1FxGLgHGAYWA0ck5nLJ1nuFcC7gSFgHDgwM3/arTolSZLUG908YnomsDQzFwNLgbMmLhARewMnAwdl5m8DzwPu7WKNkiRJ6pGuBNOI2A7YC1hWT1oG7BURIxMWPRE4LTPvBMjMezNzXTdqlCRJUm9161T+QmBlZq4HyMz1EbGqnj7astxTgFsi4gpgPvCvwKmZOd7ujoaH5zdX9YAbGVnQ6xL6iv1sjr1slv1sjr1slv1szmzpZdfGmLZpLrAncBCwJfAl4HbgX9rdwOrVaxkbazvHagojIwsYHV3T6zL6hv1sjr1slv1sjr1slv1sTi96OWfO0CYdLOzWGNMVwE4RMReg/r1jPb3V7cDnMvPBzFwDfAH4nS7VKEmSpB7qKJhGxB4R8e6IWNryfM+NrZeZdwHXAUvqSUuAazNzdMKiFwAvjIihiNgC+H3g+k5qlCRJ0uzUdjCNiCOAK4CdgKPryfOBD7e5ieOA4yPiZuD4+jkRcVl9NT7AhcBdwP9SBdnvA//cbo2SJEmavToZY3oK1W2cro+II+tp1wNPb2flzLwJ2HeS6Ye0PB4D3lr/SJIkaYB0cip/O+CG+vF4y2+vNJIkSdJm6ySYXsMjp/A3eCXw7ebKkSRJ0qDq5FT+CcCXI+INwDYRcTmwGHjhjFQmSZKkgdJ2MM3MmyJiD+AlwKVUt3q6NDPXzlRxkiRJGhwd3WA/M+8HPjNDtUiSJGmAtR1MI+LrTHGhU2bu31hFkiRJGkidHDE9e8Lz7YE3AOc1V44kSZIGVSdjTM+ZOC0iLgI+SXWPU0mSJGmTdfSVpJNYCWz0K0klSZKkjelkjOnrJ0zaGjgMuKrRiiRJkjSQOhljOvHm+vcB3wQ+0lw5kiRJGlSdjDE9YCYLkSRJ0mCbNphGxK7tbCQzf9xMOZIkSRpUGzti+kOqe5cOTbPMODC3sYokSZI0kKYNppm5uVftS5IkSW0xeEqSJKkIndwuah7wx8DzgSfQcnrfrySVJEnS5urkiOlHgGOBK4BnARcB2wFfnYG6JEmSNGA6CaaHAS/KzI8CD9e/Xwp4GylJkiRttk6C6dbAivrxAxGxdWbeBDyz+bIkSZI0aDr55qcfAPsA3wa+A5wcEb8AVs5EYZIkSRosnQTTPwPW14/fCvwjsAB4c9NFSZIkafB08pWkV7c8Xg4cOCMVSZIkaSB1cruo64HzgAszc8XGlpckSZI60cnFTydTjTH9QUT8d0QcGxGPn5myJEmSNGjaDqaZ+fnMfAWwA/AJ4GXAioi4ZKaKkyRJ0uDo5OInADJzTURcANwDbAkc0nhVkiRJGjidjDEdAl4AvIrqaOltwAXAa2amNEmSJA2STo6YrgLWAhcCz83MH8xMSZIkSRpEnQTTP8rMb0+3QEQsycxlm1mTJEmSBlAnFz9NG0prZ21GLZIkSRpgndwuqh1DDW9PkiRJA6LpYDre8PYkSZI0IJoOppIkSdImMZhKkiSpCE0H09sb3p4kSZIGREff/BQRewBHANtn5p/Uz7fMzBsAMvO3Z6BGSZIkDYC2j5hGxBHAFcBOwNH15PnAh2egLkmSJA2YTk7lnwIclJnHAevradcDT2+8KkmSJA2cToLpdsAN9ePxlt/eIkqSJEmbrZNgeg2PnMLf4JVAO98IJUmSJE2rk4ufTgC+HBFvALaJiMuBxcALZ6QySZIkDZS2g2lm3lRfhf8S4FJgBXBpZq6dqeIkSZI0ODq6XVRm3g98ZoZqkSRJ0gBrO5hGxNeZ4kKnzNy/sYokSZI0kDo5Ynr2hOfbA28AzmuuHEmSJA2qTsaYnjNxWkRcBHyS6h6nkiRJ0ibr5HZRk1kJ7NlEIZIkSRpsnYwxff2ESVsDhwFXNVqRJEmSBlInY0wn3lz/PuCbwEeaK0eSJEmDqpMxpgfMZCGSJEkabJ2cyt+1neUy88ebXo4kSZIGVSen8n/II/cxHeLR9zQdqn+PA3MbqEuSJEkDppNg+gbgQOBk4DZgF+Ak4D8z81ONVyZJkqSB0kkwfS+we2Y+UD9fHhHHAjcDn2q6MEmSJA2WTu5jOgdYNGHaLnjqXpIkSQ3o5IjpR4CvRsQngRXAQuC1eLsoSZIkNaDtI6aZ+XfA64AnAn8IbA+8PjM/OEO1SZIkaYB0csSUzPwS8KUZqkWSJEkDbNpgGhHvysxT68enTLVcZp7UdGGSJEkaLBs7YvqklscLZ7IQSZIkDbZpg2lmvqXl8etmvhxJkiQNqo7GmEbEY4EA5rdOz8yvNlmUJEmSBk/bwTQiXgssBdYC97fMGgd2bbYsSZIkDZpOjpieChyemV+cqWIkSZI0uDr55qd5wJdnqhBJkiQNtk6C6f8D/joiOllHkiRJaksnp/JPpPq2p7dHxOrWGZm5c6NVSZIkaeB0EkxfPWNVSJIkaeC1HUwz879nshBJkiQNtk5uF7VZX0kaEYuBc4BhYDVwTGYun2LZAK4FPpaZb2u3RkmSJM1enVzItHDCzz7A24Ant7n+mcDSzFxMdT/UsyZbKCLm1vMu7qA2SZIkzXKdnMr/ta8kjYiDgSUbWzcitgP2Ag6qJy0DzoiIkcwcnbD4O4BLqb5daj6SJEkaCB19Jekkvgx8uo3lFgIrM3M9QGauj4hV9fRfBdOIeDrwB8ABwLs3paDhYbNsU0ZGFvS6hL5iP5tjL5tlP5tjL5tlP5szW3rZyRjTiV87ujXwKmBFE4VExBbAPwGvq4PrJm1n9eq1jI2NN1HSQBsZWcDo6Jpel9E37Gdz7GWz7Gdz7GWz7GdzetHLOXOGNulgYSdHTH8IjAND9fP7qS5Qek0b664AdoqIuXXonAvsyKND7Q5U41Uvq0PptsBQRPxmZr65gzolSZI0C3UyxnSTv/EpM++KiOuoxqOeV/++tnV8aWbeDjxhw/OIOBmY71X5kiRJg6GbXy96HHB8RNwMHF8/JyIui4i9u1iHJEmSCrS5Fz+1LTNvAvadZPohUyx/8kzXJEmSpHJ084ipJEmSNCWDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkI87q1o4hYDJwDDAOrgWMyc/mEZd4NvBJYDzwEvDMzL+9WjZIkSeqdbh4xPRNYmpmLgaXAWZMs821gn8zcE3g98OmI2KqLNUqSJKlHuhJMI2I7YC9gWT1pGbBXRIy0LpeZl2fm/fXTG4AhqiOskiRJ6nPdOpW/EFiZmesBMnN9RKyqp49Osc4xwI8y8yed7Gh4eP5mFapHjIws6HUJfcV+NsdeNst+NsdeNst+Nme29LJrY0w7ERHPB94LHNTpuqtXr2VsbLz5ogbMyMgCRkfX9LqMvmE/m2Mvm2U/m2Mvm2U/m9OLXs6ZM7RJBwu7NcZ0BbBTRMwFqH/vWE9/lIh4DnAe8NLMzC7VJ0mSpB7rSjDNzLuA64Al9aQlwLWZ+ajT+BGxD/Bp4PDM/G43apMkSVIZunkq/zjgnIg4Cfg51RhSIuIy4KTM/A7wMWAr4KyI2LDe0Zl5YxfrlCRJUg90LZhm5k3AvpNMP6Tl8T7dqkeSJEll8ZufJEmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSijA0Pj7e6xqasgi4pddFSJIklWbs/vtZfd/6ru1vzpwhhofnA/wWcGu7682bqYJ6ZtEiuO22XlchSZJUjDnj43Dfml6XsVGeypckSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIhhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSpIkqQgGU0mSJBXBYCpJkqQiGEwlSZJUBIOpJEmSimAwlSRJUhEMppIkSSrCvG7tKCIWA+cAw8Bq4JjMXD5hmbnA6cDBwDjwgcw8u1s1SpIkqXe6ecT0TGBpZi4GlgJnTbLMUcBuwO7Ac4CTI2JR1yqUJElSz3QlmEbEdsBewLJ60jJgr4gYmbDokcDHM3MsM0eBi4EjulGjJEmSeqtbR0wXAiszcz1A/XtVPb3VzsBtLc9vn2QZSZIk9aGujTHtmltv7XUFkiRJZVm3jpGRBb2uYqO6FUxXADtFxNzMXF9f5LRjPb3V7cAuwNX184lHUDdq9eq1jI2Nb269A29kZAGjo2t6XUbfsJ/NsZfNsp/NsZfNsp/NGRlZwOiah2DNQ13b55w5QwwPz+98vRmo5ddk5l3AdcCSetIS4Np6HGmrzwJviog59fjTlwKf60aNkiRJ6q1uXpV/HHB8RNwMHF8/JyIui4i962XOBX4MLAeuAk7JzFu6WKMkSZJ6pGtjTDPzJmDfSaYf0vJ4PfCWbtUkSZKkcvjNT5IkSSqCwVSSJElFMJhKkiSpCAZTSZIkFcFgKkmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVwWAqSZKkIszrdQENmgswZ85Qr+voG/ayWfazOfayWfazOfayWfazOd3uZcv+5nay3tD4+Hjz1fTG84Cv97oISZIk/cp+wJXtLtxPwfQxwD7AHcD6HtciSZI0yOYCOwBXAw+2u1I/BVNJkiTNYl78JEmSpCIYTCVJklQEg6kkSZKKYDCVJElSEQymkiRJKoLBVJIkSUUwmEqSJKkIBlNJkiQVYV6vC2hCRCwGzgGGgdXAMZm5vLdVlSUibgXW1T8Af5mZl0fEs4GzgK2AW4FXZ+Zd9TqbNK8fRcRpwMuBRcDTMvN79fQp33szMa8fTNPLW5nkPVrP8306iYgYBs4Fngz8ElgOHJuZozPRswHv5zhwIzBWL350Zt5Yr3co8HdU/6ZeA7wuM+/fnHn9ICIuBn6LqmdrgeMz8zo/Nzs3TS9vpc8+N/vliOmZwNLMXAwspWqoft3hmfmM+ufyiJgDnAf8Sd27K4APAGzqvD52MbA/cNuE6dO992ZiXj+Yqpcw4T0Km/5eHJD36TjwwcyMzHwa8CPgAzPRs0HuZ8v83215f24IpfOBjwOHZuZuwBrgbZszr4+8JjOfnpnPBE4DPlFP93Ozc1P1Evrsc3PWB9OI2A7YC1hWT1oG7BURI72ratZ4FrAuM6+sn58JvGIz5/WlzLwyM1e0TpvuvTcT82bqtXXbZL3cCN+nU8jMn2Xm11omXQXswsz0bJD7OZ0XAd9pOTp3JnDkZs7rC5l5b8vTxwJjfm5umsl6uZFVZu3f81kfTIGFwMrMXA9Q/15VT9ejnR8RN0TExyJiW2BnWo5aZebdwJyIePxmzBsk0733ZmLeIJj4HgXfp22pj3S8BbiEmenZIPdzg69FxHUR8f6IeEw97VF9AW7nkb+vmzqvb0TE2RFxO3Aq8Br83Nxkk/Ryg7763OyHYKr27JeZTwf2AYaAM3pcjzSR79HN8w9UY8/sWzMm9nPnzNybahjKU4B396qw2SQz35iZOwPvpBpPq000RS/77nOzH4LpCmCniJgLUP/esZ6u2oZTp5n5IPAx4LlU/0P/1WmqiHgCMJaZP9uMeYNkuvfeTMzra1O8R8H36UbVF5TtDhyZmWPMTM8GuZ+t789fAGczxfuT6ojTis2c13cy81zgAOAn+LlYpm8TAAAH80lEQVS5WTb0MiKG+/Fzc9YH06yuFLsOWFJPWgJcm5mjvauqLBGxTUQ8tn48BLySqmfXAFtFxPPqRY8DPls/3tR5A2O6995MzJv5V9Q707xHwffptCLib6nGhb20/scJZqZnA9vPiHhcRGxVP54HHM4j788vAftExO718+OAz2zmvFkvIuZHxMKW54cCPwP83OzQNL1c14+fm0Pj4+Pd3N+MiIg9qG4T8Tjg51S3icjeVlWOiNgVuAiYW//8L3BCZt4REb9LdfXib/DIbSF+Wq+3SfP6UUScDhwGbA/cDazOzKdO996biXn9YLJeAocyxXu0Xsf36SQi4qnA94CbgQfqybdk5stmomeD2k/gg1SvexzYAvgm8OeZubZe74/qZeYC1wKvzcz7NmfebBcRTwS+AGwDrKcKUm/LzO/6udmZqXoJ3EMffm72RTCVJEnS7DfrT+VLkiSpPxhMJUmSVASDqSRJkopgMJUkSVIRDKaSJEkqgsFUkiRJRTCYSupLEXFrRBzY4xq2ioh/i4h7I2KTblIdEYsiYry+sfusERFHRcSXe12HpNllVn3QSdIsczjwRGA4Mx/udTEzJSIWUd2IfosNrzMzzwfO72VdkmYfj5hK0jQ280jlLsDNpYfSiBiKCP89kNRzfvOTpK6KiFuBM4BjqILbl4DXUH3P8xsz83kty44Du2fmDyPiU8D9wG8B+wHXAy8H3lGv/1NgSWZe27Kfs4CjgR2Ai4G3ZOa6ev5LgPcBi6i+yu+4zLyhZd1/BI4CAthmqnAZEf+nXvYZwErgrzLzkoh4D/BXwBDwIPBnmfnPU2xjDvBO4E3AVnVPjs/Me1uORh4LnFxv70OZeVq97u8AHwMWU32N5vmZ+dZ63rOBDwNPAW6ra/haPe9rwDeA3wP2qrd9RGbu3VLXicABmfmHEfHiul9PBu4F/jkzT66Xux1YCGz4Os2D6r796s+z/prDj9Z13lzX8s2WWr4OvADYE/gW8KrMvDsifgM4G3gR1dcuLgde0k9fhSrpEf4PWVIvvAI4mCpk7gm8toP1/hp4AlXY+xbw3fr556hCWKujgD+gClOL63WJiGcCn6AKe8NUAfaSiHhMy7pLgBcD204TSrcA/g34MrAdcDxwfkREZv4N8LfApzNz/lShtPba+ucAYFdgPlV4b3UAsDvwQuAvW8bPfhT4aGb+Zv06P1PXthPw71Rh8vFU3619UUSMtGzzaODNwALgzGq12L1l/quAC+rH91H9Z2Lbui9viYiX1vP2r39vW7/Wb03o0+PrWk6n6veHgX+PiOEJ+3odVR+3rOuF6j8dj6UKvsPAcTzyPfaS+ozBVFIvnJ6ZqzLzZ1TB7hltrvf5zLymPur5eWBdZv5LZq4HPg08c8LyZ2Tmino/p1KFTajC2FmZ+T+ZuT4zz6EKus+eUOOKzJwuBD2bKkR+IDN/mZlfBS5t2U+7jgI+nJk/zsy1VEdaXzlhGMF7MvO+zLwR+GTLPh4CdouIJ2Tm2sy8qp7+auCyzLwsM8cy8z+A7wCHtGzzU5n5/cx8ODPvBb6wYbt1QN0DuAQgM7+WmTfW27oBWAY8v83X92JgeWaeW+9rGXATcGjLMp/MzJvrfn+GR94TD1EF0t3qP6trMvMXbe5X0izjxU+SeuHOlsf3Azu2uV7r6dsHJnk+f8LyK1oe39ayn12A10TE8S3zt5xQR+u6U9kRWJGZYxP2s1Mb607czm0TtjGP6sKpyeq5DXha/fgNwCnATRFxC1WAvZTqNR4REa3hbwvgv6bYJlRHRz9Ub+9VwMWZeT9AROwLfAD4bapePQZo904DE1/fhtfQ2qeJ74kNf5bnUh0tvTAitgXOA96VmQ+1uW9Js4jBVFIp7gO23vAkIrZvYJsLWx7vDKyqH68ATs3MU6dZt50B+KuAhRExpyWc7kw1hrITq6iCZGutD1MF7yfV0xZSHWXcMH8VQGYuB5bU41QPAz5XnyJfAZybmW+aZr8TX+N/ACMR8QyqI6cntsy7gGp4wYsyc11E/D3VEIrJtrOx17fhNXxpI+tRB9D3AO+px9teBiQw3dAISbOUwVRSKa4HnlqHopuoLsbZXH8SEZdSHYF7F9XpfoCPA5+PiK8A36YKxL8HXJGZazrY/v/U2357RHwIeC7V6el9OqxzGdW40S8CozwyNvXhiNiwzLsj4k1U43JfR3Wqnoh4NXB5Zo5GxD31smNURxavjog/AL5CdbT02cAPM/MnkxWRmQ/V91v9O6pxqf/RMnsB8LM6lP4O1RHVDfcpHa33uSuTh/LLgH+IiFdRnaZ/OdUFWZdurDERcQBwN9UFar+gOrU/Nu1KkmYtx5hKKkJm3kx1CvkrVFdeX9nAZi+gCk8/Bn5EdSEQmfkdqivgzwB+DvyQ9i/Aaq35l1RB9EVU4eljwDGZedO0K/66T1Cdsr6C6gr8dVQXUrX677rO/wROy8wNofBg4PsRsZbqQqhXZuYDmbkC+COqq/1HqY6g/gUb/9y/ADgQ+OyEi77+GDglItYAJ1FfZAVQn+4/FfhGRNxT3w2AlvmrgZcA/xdYDbyd6sr6uzdSC8D2VBe2/QL4Qd2Hc9tYT9Is5O2iJEmSVASPmEqSJKkIjjGVpGlExM5U4xsn85TMvL3N7ZxJPS50gvMy87hNrU+S+omn8iVJklQET+VLkiSpCAZTSZIkFcFgKkmSpCIYTCVJklSE/w+ub8HmN5+K0wAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"metadata":{"_uuid":"5a4cb242e76c3b015679850caac7b14db8fa1359"},"cell_type":"markdown","source":"Separate the input variable names by excluding the target:"},{"metadata":{"trusted":true,"_uuid":"4d6c4a84ed3fcb2447f11b1d0a08a0b4fefcbaf7"},"cell_type":"code","source":"independent_columns_names = [x for x in data if x != 'SeriousDlqin2yrs']\nindependent_columns_names","execution_count":7,"outputs":[{"output_type":"execute_result","execution_count":7,"data":{"text/plain":"['age',\n 'NumberOfTime30-59DaysPastDueNotWorse',\n 'DebtRatio',\n 'NumberOfTimes90DaysLate',\n 'NumberOfTime60-89DaysPastDueNotWorse',\n 'MonthlyIncome',\n 'NumberOfDependents']"},"metadata":{}}]},{"metadata":{"_uuid":"ad909484de236ecaf28a596fd5570aa11c07ee43"},"cell_type":"markdown","source":"Apply the function to replace *NaN* values:"},{"metadata":{"trusted":true,"_uuid":"4e1e6638b8c50ee094b8d9f3af693fda035aa8c4"},"cell_type":"code","source":"table = fill_nan(data)","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"9b13ec032da4e9b7eab544694132b8f58d9d5adf"},"cell_type":"markdown","source":"Separate the target variable and input features:"},{"metadata":{"trusted":true,"_uuid":"55248eae34b1abf6337dbce543195d9adef29e19"},"cell_type":"code","source":"X = table[independent_columns_names]\ny = table['SeriousDlqin2yrs']","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"42b04e39e04e71871fb2ff1db2295034465d8cbd"},"cell_type":"markdown","source":"## Bootstrapping"},{"metadata":{"_uuid":"245e80eb69b9faba586538ecf6bd0343e0748abb"},"cell_type":"markdown","source":"**Question 2.** Make an interval estimate of the average age for the customers who delayed repayment at the 90% confidence level. Use the example from the article as reference, if needed. Also, use `np.random.seed(0)` as before. What is the resulting interval estimate?\n\n1. 52.59 – 52.86\n2. 45.71 – 46.13\n3. 45.68 – 46.17\n4. 52.56 – 52.88"},{"metadata":{"trusted":true,"_uuid":"0f42f4204cfa87434c31e5e09e270a42bd61850c"},"cell_type":"code","source":"# Your code here\ndef get_bootstrap_samples(data, n_samples):\n    \"\"\"Generate bootstrap samples using the bootstrap method.\"\"\"\n    indices = np.random.randint(0, len(data), (n_samples, len(data)))\n    samples = data[indices]\n    return samples\n\ndef stat_intervals(stat, alpha):\n    \"\"\"Produce an interval estimate.\"\"\"\n    boundaries = np.percentile(stat, [100 * alpha / 2., 100 * (1 - alpha / 2.)])\n    return boundaries\n\ndelayed_age = table[table['SeriousDlqin2yrs'] == 1]['age'].values\n\nnp.random.seed(0)\n\ndelayed_mean_age = [np.mean(sample) for sample in get_bootstrap_samples(delayed_age, 1000)]\n\nprint('Interval estimate of the average age for the customers who delayed repayment at the 90% confidence level: ', stat_intervals(delayed_mean_age, 0.1))\n\n\n","execution_count":10,"outputs":[{"output_type":"stream","text":"Interval estimate of the average age for the customers who delayed repayment at the 90% confidence level:  [45.71379414 46.12700479]\n","name":"stdout"}]},{"metadata":{"_uuid":"0aa59947da5e312e6bb5067bb16ae656a8a8e500"},"cell_type":"markdown","source":"## Logistic regression"},{"metadata":{"_uuid":"4f046164d4d94ed0b2ee0ebe7f4bfff460d0505a"},"cell_type":"markdown","source":"Let's set up to use logistic regression:"},{"metadata":{"trusted":true,"_uuid":"fea9173ba030a164de7d4da05d83bce7a881e92f"},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"c44b5d33184b81809b2747b2d2f5150b13b65e8f"},"cell_type":"markdown","source":"Now, we will create a `LogisticRegression` model and use `class_weight='balanced'` to make up for our unbalanced classes."},{"metadata":{"trusted":true,"_uuid":"92292d0240a272fc6c3948069ef2fc0b9f165926"},"cell_type":"code","source":"lr = LogisticRegression(random_state=5, class_weight='balanced')","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"24e5007b98c985443af75a02a858e85fc72a693a"},"cell_type":"markdown","source":"Let's try to find the best regularization coefficient, which is the coefficient `C` for logistic regression. Then, we will have an optimal model that is not overfit and is a good predictor of the target variable."},{"metadata":{"trusted":true,"_uuid":"77fff55d1042a6b6b906e4f8af2e977966e9ad55"},"cell_type":"code","source":"parameters = {'C': (0.0001, 0.001, 0.01, 0.1, 1, 10)}","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"c350a2bb023dd8ea2261fefee539ae70408586e6"},"cell_type":"markdown","source":"In order to find the optimal value of `C`, let's apply stratified 5-fold validation and look at the *ROC AUC* against different values of the parameter `C`. Use the `StratifiedKFold` function for this: "},{"metadata":{"trusted":true,"_uuid":"296cc7dbf4b702d3170f3515f4c431da8dd02486"},"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=5)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"f0bd594310bed9b8664fccbad367ec40b2494ef9"},"cell_type":"markdown","source":"One of the important metrics of model quality is the *Area Under the Curve (AUC)*. *ROC AUC* varies from 0 to 1. The closer ROC AUC is to 1, the better the quality of the classification model."},{"metadata":{"collapsed":true,"_uuid":"47ff20f56a9496a90057141c671f4534c4d8f435"},"cell_type":"markdown","source":"**Question 3.** Perform a *Grid Search* with the scoring metric \"roc_auc\" for the parameter `C`. Which value of the parameter `C` is optimal? \n\n1. 0.0001\n2. 0.001\n3. 0.01\n4. 0.1\n5. 1\n6. 10"},{"metadata":{"trusted":true,"_uuid":"ea814fc22242f846757a81fda5d63b1e8d71b81c"},"cell_type":"code","source":"# Your code here\n\ngcv = GridSearchCV(estimator = lr, param_grid = parameters, scoring = 'roc_auc', n_jobs = -1, cv = skf, verbose = 1)\ngcv.fit(X,y)","execution_count":15,"outputs":[{"output_type":"stream","text":"Fitting 5 folds for each of 6 candidates, totalling 30 fits\n","name":"stdout"},{"output_type":"stream","text":"[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed:   12.5s finished\n","name":"stderr"},{"output_type":"execute_result","execution_count":15,"data":{"text/plain":"GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=5, shuffle=True),\n       error_score='raise-deprecating',\n       estimator=LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n          fit_intercept=True, intercept_scaling=1, max_iter=100,\n          multi_class='warn', n_jobs=None, penalty='l2', random_state=5,\n          solver='warn', tol=0.0001, verbose=0, warm_start=False),\n       fit_params=None, iid='warn', n_jobs=-1,\n       param_grid={'C': (0.0001, 0.001, 0.01, 0.1, 1, 10)},\n       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n       scoring='roc_auc', verbose=1)"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"gcv.best_params_","execution_count":16,"outputs":[{"output_type":"execute_result","execution_count":16,"data":{"text/plain":"{'C': 0.001}"},"metadata":{}}]},{"metadata":{"_uuid":"ca2e0d2ea8660f676352b0bd60e44e27b621f87a"},"cell_type":"markdown","source":"**Question 4.** Can we consider the best model stable? The model is *stable* if the standard deviation on validation is less than 0.5%. Save the *ROC AUC* value of the best model; it will be useful for the following tasks.\n\n1. Yes\n2. No"},{"metadata":{"trusted":true},"cell_type":"code","source":"gcv.cv_results_['std_test_score'][1]","execution_count":17,"outputs":[{"output_type":"execute_result","execution_count":17,"data":{"text/plain":"0.006357471890470371"},"metadata":{}}]},{"metadata":{"trusted":true,"_uuid":"27e2211b96120fad6ed989c4b55a033184b442ac"},"cell_type":"code","source":"# Your code here\nroc_auc_best_score = gcv.best_score_\nroc_auc_best_score","execution_count":40,"outputs":[{"output_type":"execute_result","execution_count":40,"data":{"text/plain":"0.795409411236062"},"metadata":{}}]},{"metadata":{"_uuid":"eed75eaefb5defeaef3a062d957171c02f94868c"},"cell_type":"markdown","source":"## Feature importance\n\n**Question 5.** *Feature importance* is defined by the absolute value of its corresponding coefficient. First, you need to normalize all of the feature values so that it will be valid to compare them. What is the most important feature for the best logistic regression model?\n\n1. age\n2. NumberOfTime30-59DaysPastDueNotWorse\n3. DebtRatio\n4. NumberOfTimes90DaysLate\n5. NumberOfTime60-89DaysPastDueNotWorse\n6. MonthlyIncome\n7. NumberOfDependents"},{"metadata":{"trusted":true,"_uuid":"42af010963a6d4c1526c3499031dcec274a2b460"},"cell_type":"code","source":"# Your code here\nfrom sklearn.preprocessing import StandardScaler\nbest_lr = LogisticRegression(C = 0.001, random_state = 5, class_weight = 'balanced')\nscale = StandardScaler()\n\nX_scaled = scale.fit_transform(X)\n\nbest_lr.fit(X_scaled, y)\n\npd.DataFrame({'feature': independent_columns_names, 'coef': np.ravel(best_lr.coef_).tolist()}).sort_values(by = 'coef', ascending = False)\n\n\n\n","execution_count":19,"outputs":[{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/preprocessing/data.py:645: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.\n  return self.partial_fit(X, y)\n/opt/conda/lib/python3.6/site-packages/sklearn/base.py:464: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.\n  return self.fit(X, **fit_params).transform(X)\n","name":"stderr"},{"output_type":"execute_result","execution_count":19,"data":{"text/plain":"                                feature      coef\n1  NumberOfTime30-59DaysPastDueNotWorse  0.724004\n3               NumberOfTimes90DaysLate  0.517673\n4  NumberOfTime60-89DaysPastDueNotWorse  0.194732\n6                    NumberOfDependents  0.101326\n2                             DebtRatio -0.024082\n5                         MonthlyIncome -0.162864\n0                                   age -0.416304","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>feature</th>\n      <th>coef</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>NumberOfTime30-59DaysPastDueNotWorse</td>\n      <td>0.724004</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NumberOfTimes90DaysLate</td>\n      <td>0.517673</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NumberOfTime60-89DaysPastDueNotWorse</td>\n      <td>0.194732</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>NumberOfDependents</td>\n      <td>0.101326</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>DebtRatio</td>\n      <td>-0.024082</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>MonthlyIncome</td>\n      <td>-0.162864</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>age</td>\n      <td>-0.416304</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"a59758c51a56ff502021f9aaccd3493f4c848e16"},"cell_type":"markdown","source":"**Question 6.** Calculate how much `DebtRatio` affects our prediction using the [softmax function](https://en.wikipedia.org/wiki/Softmax_function). What is its value?\n\n1. 0.38\n2. -0.02\n3. 0.11\n4. 0.24"},{"metadata":{"trusted":true,"_uuid":"de4195b2f9012f17ad3b2999622404c303d4fd0a"},"cell_type":"code","source":"# Your code here\ndef softmax(x):\n    \"\"\"Compute softmax values for each sets of scores in x.\"\"\"\n    e_x = np.exp(x - np.max(x))\n    return e_x / e_x.sum()\n\nprint(softmax(best_lr.coef_[0])[2])","execution_count":29,"outputs":[{"output_type":"stream","text":"0.11420536719928262\n","name":"stdout"}]},{"metadata":{"_uuid":"82f94fb1ddc235250d7af7156cf69280daa317f5"},"cell_type":"markdown","source":"**Question 7.** Let's see how we can interpret the impact of our features. For this, recalculate the logistic regression with absolute values, that is without scaling. Next, modify the customer's age by adding 20 years, keeping the other features unchanged. How many times will the chance that the customer will not repay their debt increase? You can find an example of the theoretical calculation [here](https://www.unm.edu/~schrader/biostat/bio2/Spr06/lec11.pdf).\n\n1. -0.01\n2. 0.70\n3. 8.32\n4. 0.66"},{"metadata":{"trusted":true,"_uuid":"988fd621a0f8b9cee200d1798a1535c941615dab"},"cell_type":"code","source":"# Your code here\nlogit = LogisticRegression(C = 0.001, random_state = 5, class_weight = 'balanced')\nlogit.fit(X,y)\n\npd.DataFrame({'feature': independent_columns_names, 'coef': np.ravel(best_lr.coef_).tolist()}).sort_values(by = 'coef', ascending = False)\n","execution_count":30,"outputs":[{"output_type":"execute_result","execution_count":30,"data":{"text/plain":"                                feature      coef\n1  NumberOfTime30-59DaysPastDueNotWorse  0.724004\n3               NumberOfTimes90DaysLate  0.517673\n4  NumberOfTime60-89DaysPastDueNotWorse  0.194732\n6                    NumberOfDependents  0.101326\n2                             DebtRatio -0.024082\n5                         MonthlyIncome -0.162864\n0                                   age -0.416304","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>feature</th>\n      <th>coef</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>NumberOfTime30-59DaysPastDueNotWorse</td>\n      <td>0.724004</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NumberOfTimes90DaysLate</td>\n      <td>0.517673</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NumberOfTime60-89DaysPastDueNotWorse</td>\n      <td>0.194732</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>NumberOfDependents</td>\n      <td>0.101326</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>DebtRatio</td>\n      <td>-0.024082</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>MonthlyIncome</td>\n      <td>-0.162864</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>age</td>\n      <td>-0.416304</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.exp(logit.coef_[0][0]*20)","execution_count":32,"outputs":[{"output_type":"execute_result","execution_count":32,"data":{"text/plain":"0.6950957705271931"},"metadata":{}}]},{"metadata":{"_uuid":"99d5aa4076c7064d4834ea32fa63a59da16c4b2d"},"cell_type":"markdown","source":"## Random Forest"},{"metadata":{"_uuid":"02f4dfa9d98b7663df6637e6bfe2ff8e43950905"},"cell_type":"markdown","source":"Import the Random Forest classifier:"},{"metadata":{"trusted":true,"_uuid":"8252905ea996ff08abc33560f7d144af8c53b56d"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier","execution_count":33,"outputs":[]},{"metadata":{"_uuid":"e4e84e4a9edb9ded1e5e5bfcde49c7c116e82cf1"},"cell_type":"markdown","source":"Initialize Random Forest with 100 trees and balance target classes:"},{"metadata":{"trusted":true,"_uuid":"d9fcfad198b9bc5f66c916b8179b2edf7d1d6067"},"cell_type":"code","source":"rf = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=42, \n                            class_weight='balanced')","execution_count":34,"outputs":[]},{"metadata":{"_uuid":"bcea3461d96645b75836cbb547c758036f6af715"},"cell_type":"markdown","source":"We will search for the best parameters among the following values:"},{"metadata":{"trusted":true,"_uuid":"4b04ae1ed65286fbd4e90511f524de20177f580a"},"cell_type":"code","source":"parameters = {'max_features': [1, 2, 4], 'min_samples_leaf': [3, 5, 7, 9], 'max_depth': [5,10,15]}","execution_count":35,"outputs":[]},{"metadata":{"_uuid":"74eb21e19cc3f6b819a438431a7781ffe6c3c9eb"},"cell_type":"markdown","source":"Also, we will use the stratified k-fold validation again. You should still have the `skf` variable."},{"metadata":{"_uuid":"a9a00dd2fc41cb30c83525e6129bfd8c13fa5492"},"cell_type":"markdown","source":"**Question 8.** How much higher is the *ROC AUC* of the best random forest model than that of the best logistic regression on validation?\n\n1. 4%\n2. 3%\n3. 2%\n4. 1%"},{"metadata":{"trusted":true,"_uuid":"f7adb918cb3a5b3b45527e77ca0eb70f78ecce55"},"cell_type":"code","source":"# Your code here\ngcv_rf = GridSearchCV(estimator = rf, param_grid = parameters, scoring = 'roc_auc', n_jobs = -1, cv = skf, verbose = 1)\ngcv_rf.fit(X,y)","execution_count":38,"outputs":[{"output_type":"stream","text":"Fitting 5 folds for each of 36 candidates, totalling 180 fits\n","name":"stdout"},{"output_type":"stream","text":"[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   43.9s\n[Parallel(n_jobs=-1)]: Done 180 out of 180 | elapsed:  4.9min finished\n","name":"stderr"},{"output_type":"execute_result","execution_count":38,"data":{"text/plain":"GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=5, shuffle=True),\n       error_score='raise-deprecating',\n       estimator=RandomForestClassifier(bootstrap=True, class_weight='balanced',\n            criterion='gini', max_depth=None, max_features='auto',\n            max_leaf_nodes=None, min_impurity_decrease=0.0,\n            min_impurity_split=None, min_samples_leaf=1,\n            min_samples_split=2, min_weight_fraction_leaf=0.0,\n            n_estimators=100, n_jobs=-1, oob_score=False, random_state=42,\n            verbose=0, warm_start=False),\n       fit_params=None, iid='warn', n_jobs=-1,\n       param_grid={'max_features': [1, 2, 4], 'min_samples_leaf': [3, 5, 7, 9], 'max_depth': [5, 10, 15]},\n       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n       scoring='roc_auc', verbose=1)"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"gcv_rf.best_score_","execution_count":39,"outputs":[{"output_type":"execute_result","execution_count":39,"data":{"text/plain":"0.8350557986441058"},"metadata":{}}]},{"metadata":{"_uuid":"8be735e7919d87d06dd627ec1885d0ef72f032e0"},"cell_type":"markdown","source":"**Question 9.** What feature has the weakest impact in the Random Forest model?\n\n1. age\n2. NumberOfTime30-59DaysPastDueNotWorse\n3. DebtRatio\n4. NumberOfTimes90DaysLate\n5. NumberOfTime60-89DaysPastDueNotWorse\n6. MonthlyIncome\n7. NumberOfDependents"},{"metadata":{"trusted":true,"_uuid":"bbb93f0ced33c8a65894bfd0bb26c5fbe534394c"},"cell_type":"code","source":"# Your code here\nimportances = gcv_rf.best_estimator_.feature_importances_\nindices = np.argsort(importances)[::-1]\n\n\n\npd.DataFrame({'independent_columns_names':independent_columns_names, 'ranking': indices, 'feature_importances':importances})\n        ","execution_count":47,"outputs":[{"output_type":"execute_result","execution_count":47,"data":{"text/plain":"              independent_columns_names         ...           feature_importances\n0                                   age         ...                      0.115926\n1  NumberOfTime30-59DaysPastDueNotWorse         ...                      0.302461\n2                             DebtRatio         ...                      0.079520\n3               NumberOfTimes90DaysLate         ...                      0.278644\n4  NumberOfTime60-89DaysPastDueNotWorse         ...                      0.148683\n5                         MonthlyIncome         ...                      0.060429\n6                    NumberOfDependents         ...                      0.014337\n\n[7 rows x 3 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>independent_columns_names</th>\n      <th>ranking</th>\n      <th>feature_importances</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>age</td>\n      <td>1</td>\n      <td>0.115926</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>NumberOfTime30-59DaysPastDueNotWorse</td>\n      <td>3</td>\n      <td>0.302461</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>DebtRatio</td>\n      <td>4</td>\n      <td>0.079520</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NumberOfTimes90DaysLate</td>\n      <td>0</td>\n      <td>0.278644</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NumberOfTime60-89DaysPastDueNotWorse</td>\n      <td>2</td>\n      <td>0.148683</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>MonthlyIncome</td>\n      <td>5</td>\n      <td>0.060429</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>NumberOfDependents</td>\n      <td>6</td>\n      <td>0.014337</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"6b4e8cf8f503365cf8d877a8bca477b53ec0bd81"},"cell_type":"markdown","source":"**Question 10.** What is the most significant advantage of using *Logistic Regression* versus *Random Forest* for this problem?\n\n1. Spent less time for model fitting;\n2. Fewer variables to iterate;\n3. Feature interpretability;\n4. Linear properties of the algorithm."},{"metadata":{"_uuid":"e409eaafe9afbebc52c02ba218107f9fde6eb54c"},"cell_type":"markdown","source":"## Bagging"},{"metadata":{"_uuid":"094bd877f40d72cf269e0395fd7d6e070bce8701"},"cell_type":"markdown","source":"Import modules and set up the parameters for bagging:"},{"metadata":{"trusted":true,"_uuid":"ea666be2e57e478428cc553fd7e2ecd5d7ca8bee"},"cell_type":"code","source":"from sklearn.ensemble import BaggingClassifier\nfrom sklearn.model_selection import cross_val_score, RandomizedSearchCV\n\nparameters = {'max_features': [2, 3, 4], 'max_samples': [0.5, 0.7, 0.9], \n              'base_estimator__C': [0.0001, 0.001, 0.01, 1, 10, 100]}","execution_count":48,"outputs":[]},{"metadata":{"_uuid":"98db8aa7f3e6ae450091a61ac187cad6dc0a05aa"},"cell_type":"markdown","source":"**Question 11.** Fit a bagging classifier with `random_state=42`. For the base classifiers, use 100 logistic regressors and use `RandomizedSearchCV` instead of `GridSearchCV`. It will take a lot of time to iterate over all 54 variants, so set the maximum number of iterations for `RandomizedSearchCV` to 20. Don't forget to set the parameters `cv` and `random_state=1`. What is the best *ROC AUC* you achieve?\n\n1. 80.75%\n2. 80.12%\n3. 79.62%\n4. 76.50%"},{"metadata":{"trusted":true,"_uuid":"2c2e9cc7930cc6c3816f8538385d1f78c12b104b"},"cell_type":"code","source":"# Your code here\nbagging = BaggingClassifier(base_estimator = LogisticRegression(class_weight = 'balanced'), n_estimators=100, n_jobs = -1, random_state = 42)\n\nrando = RandomizedSearchCV(estimator = bagging, param_distributions=parameters, n_iter = 20, scoring='roc_auc', n_jobs = -1, random_state = 1, cv=skf)\n\nrando.fit(X,y)","execution_count":50,"outputs":[{"output_type":"execute_result","execution_count":50,"data":{"text/plain":"RandomizedSearchCV(cv=StratifiedKFold(n_splits=5, random_state=5, shuffle=True),\n          error_score='raise-deprecating',\n          estimator=BaggingClassifier(base_estimator=LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n          fit_intercept=True, intercept_scaling=1, max_iter=100,\n          multi_class='warn', n_jobs=None, penalty='l2', random_state=None,\n          solver='warn', tol=0.0001, verbose=0, warm_...n_estimators=100, n_jobs=-1, oob_score=False,\n         random_state=42, verbose=0, warm_start=False),\n          fit_params=None, iid='warn', n_iter=20, n_jobs=-1,\n          param_distributions={'max_features': [2, 3, 4], 'max_samples': [0.5, 0.7, 0.9], 'base_estimator__C': [0.0001, 0.001, 0.01, 1, 10, 100]},\n          pre_dispatch='2*n_jobs', random_state=1, refit=True,\n          return_train_score='warn', scoring='roc_auc', verbose=0)"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"rando.best_score_","execution_count":52,"outputs":[{"output_type":"execute_result","execution_count":52,"data":{"text/plain":"0.8076172570918905"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"rando.best_estimator_","execution_count":53,"outputs":[{"output_type":"execute_result","execution_count":53,"data":{"text/plain":"BaggingClassifier(base_estimator=LogisticRegression(C=0.001, class_weight='balanced', dual=False,\n          fit_intercept=True, intercept_scaling=1, max_iter=100,\n          multi_class='warn', n_jobs=None, penalty='l2', random_state=None,\n          solver='warn', tol=0.0001, verbose=0, warm_start=False),\n         bootstrap=True, bootstrap_features=False, max_features=2,\n         max_samples=0.7, n_estimators=100, n_jobs=-1, oob_score=False,\n         random_state=42, verbose=0, warm_start=False)"},"metadata":{}}]},{"metadata":{"collapsed":true,"_uuid":"09590fe9a20e8ab33fb09dc14ddc2862646e01b5"},"cell_type":"markdown","source":"**Question 12.** Give an interpretation of the best parameters for bagging. Why are these values of `max_features` and `max_samples` the best?\n\n1. For bagging it's important to use as few features as possible;\n2. Bagging works better on small samples;\n3. Less correlation between single models;\n4. The higher the number of features, the lower the loss of information."}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}