a6-linear-models-and-rf-for-regression.ipynb

{"cells":[{"metadata":{"_uuid":"fa68e17d12b944b5925f0f1f3b35de84f88b63e7"},"cell_type":"markdown","source":"<center>\n<img src=\"https://habrastorage.org/files/fd4/502/43d/fd450243dd604b81b9713213a247aa20.jpg\">\n## Open Machine Learning Course\n<center>Author: [Yury Kashnitsky](https://www.linkedin.com/in/festline/), Data Scientist at Mail.ru Group <br>\n    All content is distributed under the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.\nYou may use this material for any purpose (you can edit, correct and use it as example) exept commercial use with mandatory citation of author."},{"metadata":{"_uuid":"d4fba24c2fc97c5651c8b229a864bd3e4fb9c80a"},"cell_type":"markdown","source":"# <center> Assignment #6 (demo).\n## <center>  Exploring OLS, Lasso and Random Forest in a regression task\n    \n<img src=https://habrastorage.org/webt/-h/ns/aa/-hnsaaifymavmmudwip9imcmk58.jpeg width=30%>\n\n**Fill in the missing code and choose answers in [this](https://docs.google.com/forms/d/1aHyK58W6oQmNaqEfvpLTpo6Cb0-ntnvJ18rZcvclkvw/edit) web form.**"},{"metadata":{"trusted":true,"_uuid":"5a5dbe8ccaac5938472fffabd662570bbee580a5"},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\nimport numpy as np\nimport pandas as pd\nfrom sklearn.metrics.regression import mean_squared_error\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.model_selection import cross_val_score, train_test_split\nfrom sklearn.linear_model import LinearRegression, LassoCV, Lasso\nfrom sklearn.ensemble import RandomForestRegressor","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"42c346b2e75d1f1aad91a95e17994c541657c4e5"},"cell_type":"markdown","source":"**We are working with UCI Wine quality dataset (no need to download it – it's already there, in course repo and in Kaggle Dataset).**"},{"metadata":{"trusted":true,"_uuid":"6fbd38aed5fa42e075b5d5a33e8e2da21807ad75"},"cell_type":"code","source":"data = pd.read_csv('../input/winequality-white.csv')","execution_count":2,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ca6a09adeecd4dc07b33e765baa4ea82820fe37"},"cell_type":"code","source":"data.head()","execution_count":3,"outputs":[{"output_type":"execute_result","execution_count":3,"data":{"text/plain":"   fixed acidity  volatile acidity   ...     alcohol  quality\n0            7.0              0.27   ...         8.8        6\n1            6.3              0.30   ...         9.5        6\n2            8.1              0.28   ...        10.1        6\n3            7.2              0.23   ...         9.9        6\n4            7.2              0.23   ...         9.9        6\n\n[5 rows x 12 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>fixed acidity</th>\n      <th>volatile acidity</th>\n      <th>citric acid</th>\n      <th>residual sugar</th>\n      <th>chlorides</th>\n      <th>free sulfur dioxide</th>\n      <th>total sulfur dioxide</th>\n      <th>density</th>\n      <th>pH</th>\n      <th>sulphates</th>\n      <th>alcohol</th>\n      <th>quality</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>7.0</td>\n      <td>0.27</td>\n      <td>0.36</td>\n      <td>20.7</td>\n      <td>0.045</td>\n      <td>45.0</td>\n      <td>170.0</td>\n      <td>1.0010</td>\n      <td>3.00</td>\n      <td>0.45</td>\n      <td>8.8</td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>6.3</td>\n      <td>0.30</td>\n      <td>0.34</td>\n      <td>1.6</td>\n      <td>0.049</td>\n      <td>14.0</td>\n      <td>132.0</td>\n      <td>0.9940</td>\n      <td>3.30</td>\n      <td>0.49</td>\n      <td>9.5</td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>8.1</td>\n      <td>0.28</td>\n      <td>0.40</td>\n      <td>6.9</td>\n      <td>0.050</td>\n      <td>30.0</td>\n      <td>97.0</td>\n      <td>0.9951</td>\n      <td>3.26</td>\n      <td>0.44</td>\n      <td>10.1</td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>7.2</td>\n      <td>0.23</td>\n      <td>0.32</td>\n      <td>8.5</td>\n      <td>0.058</td>\n      <td>47.0</td>\n      <td>186.0</td>\n      <td>0.9956</td>\n      <td>3.19</td>\n      <td>0.40</td>\n      <td>9.9</td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>7.2</td>\n      <td>0.23</td>\n      <td>0.32</td>\n      <td>8.5</td>\n      <td>0.058</td>\n      <td>47.0</td>\n      <td>186.0</td>\n      <td>0.9956</td>\n      <td>3.19</td>\n      <td>0.40</td>\n      <td>9.9</td>\n      <td>6</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true,"_uuid":"8567888d41488b52f8b2d3dff80bd2d16c3df07d"},"cell_type":"code","source":"data.info()","execution_count":4,"outputs":[{"output_type":"stream","text":"<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 4898 entries, 0 to 4897\nData columns (total 12 columns):\nfixed acidity           4898 non-null float64\nvolatile acidity        4898 non-null float64\ncitric acid             4898 non-null float64\nresidual sugar          4898 non-null float64\nchlorides               4898 non-null float64\nfree sulfur dioxide     4898 non-null float64\ntotal sulfur dioxide    4898 non-null float64\ndensity                 4898 non-null float64\npH                      4898 non-null float64\nsulphates               4898 non-null float64\nalcohol                 4898 non-null float64\nquality                 4898 non-null int64\ndtypes: float64(11), int64(1)\nmemory usage: 459.3 KB\n","name":"stdout"}]},{"metadata":{"_uuid":"40b24e1cbe39da5f25f6abdfa6f4dce01052eba2"},"cell_type":"markdown","source":"**Separate the target feature, split data in 7:3 proportion (30% form a holdout set, use random_state=17), and preprocess data with `StandardScaler`.**"},{"metadata":{"trusted":true,"_uuid":"22991d1fcd912df4535ec8e0dd804d9d66b336f6"},"cell_type":"code","source":"y = data['quality']\nX = data.drop('quality', axis = 1)\n\nX_train, X_holdout, y_train, y_holdout = train_test_split(X, y, test_size = 0.3, random_state = 17)\nscaler = StandardScaler()\nX_train_scaled = scaler.fit_transform(X_train)\nX_holdout_scaled = scaler.transform(X_holdout)","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"4323af0a23b5a874c7fb0fc3bcadeebefd5bdce1"},"cell_type":"markdown","source":"## Linear regression"},{"metadata":{"_uuid":"39e8d88d30226a24ea1d62cfdf849b9a9b164a3f"},"cell_type":"markdown","source":"**Train a simple linear regression model (Ordinary Least Squares).**"},{"metadata":{"trusted":true,"_uuid":"d228bb880c81ea10ad104e1f9685c8a2ee95ae97"},"cell_type":"code","source":"linreg = LinearRegression(n_jobs = -1)\nlinreg.fit(X_train_scaled, y_train) ","execution_count":6,"outputs":[{"output_type":"execute_result","execution_count":6,"data":{"text/plain":"LinearRegression(copy_X=True, fit_intercept=True, n_jobs=-1, normalize=False)"},"metadata":{}}]},{"metadata":{"_uuid":"143bd8c03bc75265fe317b7163ea9992fec72a52"},"cell_type":"markdown","source":"**<font color='red'>Question 1:</font> What are mean squared errors of model predictions on train and holdout sets?**"},{"metadata":{"trusted":true,"_uuid":"546f6310409fd0cced8e9cf9b5b9b647bc519b4a"},"cell_type":"code","source":"train_pred = linreg.predict(X_train_scaled)\nholdout_pred = linreg.predict(X_holdout_scaled)\nprint(\"Mean squared error (train): %.3f\" % mean_squared_error(y_train, train_pred))\nprint(\"Mean squared error (test): %.3f\" % mean_squared_error(y_holdout, holdout_pred))","execution_count":7,"outputs":[{"output_type":"stream","text":"Mean squared error (train): 0.558\nMean squared error (test): 0.584\n","name":"stdout"}]},{"metadata":{"_uuid":"eb150b4956b12f57d7a4d2a868b220d56951c70d"},"cell_type":"markdown","source":"**Sort features by their influence on the target feature (wine quality). Beware that both large positive and large negative coefficients mean large influence on target. It's handy to use `pandas.DataFrame` here.**\n\n**<font color='red'>Question 2:</font> Which feature this linear regression model treats as the most influential on wine quality?**"},{"metadata":{"trusted":true,"_uuid":"00f2cd04fdee695e60a0019a095999c1b9d98ff7"},"cell_type":"code","source":"linreg_coef = pd.DataFrame({'coefficient':linreg.coef_, 'absolute_coefficient':np.abs(linreg.coef_)}, index=X.columns)\nlinreg_coef.sort_values(by = 'absolute_coefficient', ascending = False)","execution_count":8,"outputs":[{"output_type":"execute_result","execution_count":8,"data":{"text/plain":"                      coefficient  absolute_coefficient\ndensity                 -0.665720              0.665720\nresidual sugar           0.538164              0.538164\nvolatile acidity        -0.192260              0.192260\npH                       0.150036              0.150036\nalcohol                  0.129533              0.129533\nfixed acidity            0.097822              0.097822\nsulphates                0.062053              0.062053\nfree sulfur dioxide      0.042180              0.042180\ntotal sulfur dioxide     0.014304              0.014304\nchlorides                0.008127              0.008127\ncitric acid             -0.000183              0.000183","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>coefficient</th>\n      <th>absolute_coefficient</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>density</th>\n      <td>-0.665720</td>\n      <td>0.665720</td>\n    </tr>\n    <tr>\n      <th>residual sugar</th>\n      <td>0.538164</td>\n      <td>0.538164</td>\n    </tr>\n    <tr>\n      <th>volatile acidity</th>\n      <td>-0.192260</td>\n      <td>0.192260</td>\n    </tr>\n    <tr>\n      <th>pH</th>\n      <td>0.150036</td>\n      <td>0.150036</td>\n    </tr>\n    <tr>\n      <th>alcohol</th>\n      <td>0.129533</td>\n      <td>0.129533</td>\n    </tr>\n    <tr>\n      <th>fixed acidity</th>\n      <td>0.097822</td>\n      <td>0.097822</td>\n    </tr>\n    <tr>\n      <th>sulphates</th>\n      <td>0.062053</td>\n      <td>0.062053</td>\n    </tr>\n    <tr>\n      <th>free sulfur dioxide</th>\n      <td>0.042180</td>\n      <td>0.042180</td>\n    </tr>\n    <tr>\n      <th>total sulfur dioxide</th>\n      <td>0.014304</td>\n      <td>0.014304</td>\n    </tr>\n    <tr>\n      <th>chlorides</th>\n      <td>0.008127</td>\n      <td>0.008127</td>\n    </tr>\n    <tr>\n      <th>citric acid</th>\n      <td>-0.000183</td>\n      <td>0.000183</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"c8180a8be8a1c16db3b3a4b2f1b045360fffedb8"},"cell_type":"markdown","source":"## Lasso regression"},{"metadata":{"_uuid":"ec78a7a6489c112634ab66ee3c8a64732333b255"},"cell_type":"markdown","source":"**Train a LASSO model with $\\alpha = 0.01$ (weak regularization) and scaled data. Again, set random_state=17.**"},{"metadata":{"trusted":true,"_uuid":"c93c9f0aef5c045104cd170a415f0138ecfde280"},"cell_type":"code","source":"lasso1 = Lasso(alpha = 0.01, random_state = 17)\nlasso1.fit(X_train_scaled, y_train)","execution_count":9,"outputs":[{"output_type":"execute_result","execution_count":9,"data":{"text/plain":"Lasso(alpha=0.01, copy_X=True, fit_intercept=True, max_iter=1000,\n   normalize=False, positive=False, precompute=False, random_state=17,\n   selection='cyclic', tol=0.0001, warm_start=False)"},"metadata":{}}]},{"metadata":{"_uuid":"2ebc826e15035e973f142ea14529e9ea62f0eaae"},"cell_type":"markdown","source":"**Which feature is the least informative in predicting wine quality, according to this LASSO model?**"},{"metadata":{"trusted":true,"_uuid":"a3e0123570f2bfc183e9af04fc09de4aa00a3e33"},"cell_type":"code","source":"lasso1_coef = pd.DataFrame({'coefficient': lasso1.coef_, 'absolute_coefficient': np.abs(lasso1.coef_)}, index = X.columns)\nlasso1_coef.sort_values(by = 'absolute_coefficient', ascending = True)","execution_count":10,"outputs":[{"output_type":"execute_result","execution_count":10,"data":{"text/plain":"                      coefficient  absolute_coefficient\nfixed acidity           -0.000000              0.000000\ncitric acid             -0.000000              0.000000\ntotal sulfur dioxide    -0.000000              0.000000\nchlorides               -0.002747              0.002747\nsulphates                0.029722              0.029722\nfree sulfur dioxide      0.043088              0.043088\npH                       0.067277              0.067277\nvolatile acidity        -0.188479              0.188479\ndensity                 -0.235492              0.235492\nresidual sugar           0.256363              0.256363\nalcohol                  0.322425              0.322425","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>coefficient</th>\n      <th>absolute_coefficient</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>fixed acidity</th>\n      <td>-0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>citric acid</th>\n      <td>-0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>total sulfur dioxide</th>\n      <td>-0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>chlorides</th>\n      <td>-0.002747</td>\n      <td>0.002747</td>\n    </tr>\n    <tr>\n      <th>sulphates</th>\n      <td>0.029722</td>\n      <td>0.029722</td>\n    </tr>\n    <tr>\n      <th>free sulfur dioxide</th>\n      <td>0.043088</td>\n      <td>0.043088</td>\n    </tr>\n    <tr>\n      <th>pH</th>\n      <td>0.067277</td>\n      <td>0.067277</td>\n    </tr>\n    <tr>\n      <th>volatile acidity</th>\n      <td>-0.188479</td>\n      <td>0.188479</td>\n    </tr>\n    <tr>\n      <th>density</th>\n      <td>-0.235492</td>\n      <td>0.235492</td>\n    </tr>\n    <tr>\n      <th>residual sugar</th>\n      <td>0.256363</td>\n      <td>0.256363</td>\n    </tr>\n    <tr>\n      <th>alcohol</th>\n      <td>0.322425</td>\n      <td>0.322425</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"34daf26da8dc65ca9dea1390b1c990bdb1e746fe"},"cell_type":"markdown","source":"**Train LassoCV with random_state=17 to choose the best value of $\\alpha$ in 5-fold cross-validation.**"},{"metadata":{"trusted":true,"_uuid":"1261f3d8b86bf6b32a47f4fbd6421a224a32fb25"},"cell_type":"code","source":"alphas = np.logspace(-6, 2, 200)\nlasso_cv = LassoCV(alphas = alphas, cv = 5, verbose = True, n_jobs = -1, random_state = 17)\nlasso_cv.fit(X_train_scaled, y_train)","execution_count":11,"outputs":[{"output_type":"stream","text":"[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 4 concurrent workers.\n........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................[Parallel(n_jobs=-1)]: Done   5 out of   5 | elapsed:    0.4s finished\n","name":"stderr"},{"output_type":"execute_result","execution_count":11,"data":{"text/plain":"LassoCV(alphas=array([1.00000e-06, 1.09699e-06, ..., 9.11589e+01, 1.00000e+02]),\n    copy_X=True, cv=5, eps=0.001, fit_intercept=True, max_iter=1000,\n    n_alphas=100, n_jobs=-1, normalize=False, positive=False,\n    precompute='auto', random_state=17, selection='cyclic', tol=0.0001,\n    verbose=True)"},"metadata":{}}]},{"metadata":{"trusted":true,"_uuid":"11cfcfa256d3fadcc41c17c4a203980f43270f86"},"cell_type":"code","source":"lasso_cv.alpha_","execution_count":12,"outputs":[{"output_type":"execute_result","execution_count":12,"data":{"text/plain":"0.004150404757850477"},"metadata":{}}]},{"metadata":{"_uuid":"774659494a9e10c5880a061b77f0ad4de3f6f48c"},"cell_type":"markdown","source":"**<font color='red'>Question 3:</font> Which feature is the least informative in predicting wine quality, according to the tuned LASSO model?**"},{"metadata":{"trusted":true,"_uuid":"e48d180e412893f9b510a39edcf612d13a865dc9"},"cell_type":"code","source":"lasso_cv_coef = pd.DataFrame({'coefficient':lasso_cv.coef_, 'absolute_coefficient':np.abs(lasso_cv.coef_)}, index = X.columns)\nlasso_cv_coef.sort_values(by = 'absolute_coefficient', ascending = True)","execution_count":29,"outputs":[{"output_type":"execute_result","execution_count":29,"data":{"text/plain":"                      coefficient  absolute_coefficient\ncitric acid             -0.000000              0.000000\nchlorides               -0.000000              0.000000\ntotal sulfur dioxide     0.000000              0.000000\nfixed acidity            0.035695              0.035695\nsulphates                0.046171              0.046171\nfree sulfur dioxide      0.046644              0.046644\npH                       0.101892              0.101892\nvolatile acidity        -0.190897              0.190897\nalcohol                  0.237681              0.237681\nresidual sugar           0.383264              0.383264\ndensity                 -0.424197              0.424197","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>coefficient</th>\n      <th>absolute_coefficient</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>citric acid</th>\n      <td>-0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>chlorides</th>\n      <td>-0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>total sulfur dioxide</th>\n      <td>0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>fixed acidity</th>\n      <td>0.035695</td>\n      <td>0.035695</td>\n    </tr>\n    <tr>\n      <th>sulphates</th>\n      <td>0.046171</td>\n      <td>0.046171</td>\n    </tr>\n    <tr>\n      <th>free sulfur dioxide</th>\n      <td>0.046644</td>\n      <td>0.046644</td>\n    </tr>\n    <tr>\n      <th>pH</th>\n      <td>0.101892</td>\n      <td>0.101892</td>\n    </tr>\n    <tr>\n      <th>volatile acidity</th>\n      <td>-0.190897</td>\n      <td>0.190897</td>\n    </tr>\n    <tr>\n      <th>alcohol</th>\n      <td>0.237681</td>\n      <td>0.237681</td>\n    </tr>\n    <tr>\n      <th>residual sugar</th>\n      <td>0.383264</td>\n      <td>0.383264</td>\n    </tr>\n    <tr>\n      <th>density</th>\n      <td>-0.424197</td>\n      <td>0.424197</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"6784416717d86f9693adc016ebf2805d82e1f44f"},"cell_type":"markdown","source":"**<font color='red'>Question 4:</font> What are mean squared errors of tuned LASSO predictions on train and holdout sets?**"},{"metadata":{"trusted":true,"_uuid":"8c1e912d29500bdd01232dffb7b7b12708a61354"},"cell_type":"code","source":"print(\"Mean squared error (train): %.3f\" % mean_squared_error(y_train, lasso_cv.predict(X_train_scaled)))\nprint(\"Mean squared error (test): %.3f\" % mean_squared_error(y_holdout, lasso_cv.predict(X_holdout_scaled)))","execution_count":14,"outputs":[{"output_type":"stream","text":"Mean squared error (train): 0.560\nMean squared error (test): 0.575\n","name":"stdout"}]},{"metadata":{"_uuid":"cece55c36b4975456ac808ed60f80c4607bce364"},"cell_type":"markdown","source":"## Random Forest"},{"metadata":{"_uuid":"2b1d919f11b7552d1fd5218469049cbceb0c6b12"},"cell_type":"markdown","source":"**Train a Random Forest with out-of-the-box parameters, setting only random_state to be 17.**"},{"metadata":{"trusted":true,"_uuid":"ef3ab441dd993dbae6326026fa0b685ecb048e43"},"cell_type":"code","source":"forest = RandomForestRegressor(random_state = 17)\nforest.fit(X_train_scaled, y_train)","execution_count":15,"outputs":[{"output_type":"execute_result","execution_count":15,"data":{"text/plain":"RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n           max_features='auto', max_leaf_nodes=None,\n           min_impurity_decrease=0.0, min_impurity_split=None,\n           min_samples_leaf=1, min_samples_split=2,\n           min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=None,\n           oob_score=False, random_state=17, verbose=0, warm_start=False)"},"metadata":{}}]},{"metadata":{"_uuid":"79d151e17dda0fa3667e01f5514edc9fa471a99f"},"cell_type":"markdown","source":"**<font color='red'>Question 5:</font> What are mean squared errors of RF model on the training set, in cross-validation (cross_val_score with scoring='neg_mean_squared_error' and other arguments left with default values) and on holdout set?**"},{"metadata":{"trusted":true,"_uuid":"78cb1562e5250e2c495954ed2f2007db4e9bd4ac"},"cell_type":"code","source":"print(\"Mean squared error (train): %.3f\" % mean_squared_error(y_train, forest.predict(X_train_scaled)))\nprint(\"Mean squared error (cv): %.3f\" % np.mean(cross_val_score(estimator = forest, X = X_train_scaled, y = y_train, scoring = 'neg_mean_squared_error')))\nprint(\"Mean squared error (test): %.3f\" % mean_squared_error(y_holdout, forest.predict(X_holdout_scaled)))\n","execution_count":16,"outputs":[{"output_type":"stream","text":"Mean squared error (train): 0.075\nMean squared error (cv): -0.460\nMean squared error (test): 0.422\n","name":"stdout"}]},{"metadata":{"_uuid":"cbfda28eb1ad4d6d0a8b640e6bcee1a6c2a0d83a"},"cell_type":"markdown","source":"**Tune the `max_features` and `max_depth` hyperparameters with GridSearchCV and again check mean cross-validation MSE and MSE on holdout set.**"},{"metadata":{"trusted":true,"_uuid":"b8fcca3ab63a14e9ebe4e05f1b706affee19ec15"},"cell_type":"code","source":"forest_params = {'max_depth': list(range(10, 25)), \n                  'min_samples_leaf': list(range(1, 8)),\n                  'max_features': list(range(6,12))}\n\nlocally_best_forest = GridSearchCV(estimator = RandomForestRegressor(n_jobs=-1, random_state=17), param_grid = forest_params, \n                                   n_jobs = -1, cv = 5, verbose = True, scoring = 'neg_mean_squared_error')\nlocally_best_forest.fit(X_train_scaled, y_train)","execution_count":17,"outputs":[{"output_type":"stream","text":"Fitting 5 folds for each of 630 candidates, totalling 3150 fits\n","name":"stdout"},{"output_type":"stream","text":"[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   11.9s\n[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   24.3s\n[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:   44.7s\n[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed:  1.2min\n[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed:  1.8min\n[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed:  2.6min\n[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed:  3.5min\n[Parallel(n_jobs=-1)]: Done 3150 out of 3150 | elapsed:  4.5min finished\n","name":"stderr"},{"output_type":"execute_result","execution_count":17,"data":{"text/plain":"GridSearchCV(cv=5, error_score='raise-deprecating',\n       estimator=RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n           max_features='auto', max_leaf_nodes=None,\n           min_impurity_decrease=0.0, min_impurity_split=None,\n           min_samples_leaf=1, min_samples_split=2,\n           min_weight_fraction_leaf=0.0, n_estimators='warn', n_jobs=-1,\n           oob_score=False, random_state=17, verbose=0, warm_start=False),\n       fit_params=None, iid='warn', n_jobs=-1,\n       param_grid={'max_depth': [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], 'min_samples_leaf': [1, 2, 3, 4, 5, 6, 7], 'max_features': [6, 7, 8, 9, 10, 11]},\n       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n       scoring='neg_mean_squared_error', verbose=True)"},"metadata":{}}]},{"metadata":{"trusted":true,"_uuid":"daa31046bca781b63e8b08d941d270ed39cca694"},"cell_type":"code","source":"locally_best_forest.best_params_, locally_best_forest.best_score_","execution_count":21,"outputs":[{"output_type":"execute_result","execution_count":21,"data":{"text/plain":"({'max_depth': 19, 'max_features': 7, 'min_samples_leaf': 1},\n -0.4346519750542217)"},"metadata":{}}]},{"metadata":{"_uuid":"f3b5f128ac9d25d6a2fe001e1b945d890e4e125a"},"cell_type":"markdown","source":"**<font color='red'>Question 6:</font> What are mean squared errors of tuned RF model in cross-validation (cross_val_score with scoring='neg_mean_squared_error' and other arguments left with default values) and on holdout set?**"},{"metadata":{"trusted":true,"_uuid":"8b1c78727baa4caa5e5de4fc99289f9c38a78eca"},"cell_type":"code","source":"print(\"Mean squared error (cv): %.3f\" % np.mean(cross_val_score(estimator = locally_best_forest.best_estimator_, X = X_train_scaled,\n                                                                y = y_train, scoring = 'neg_mean_squared_error',\n                                                        n_jobs = -1)))\nprint(\"Mean squared error (test): %.3f\" % mean_squared_error(y_holdout, locally_best_forest.predict(X_holdout_scaled)))","execution_count":26,"outputs":[{"output_type":"stream","text":"Mean squared error (cv): -0.457\nMean squared error (test): 0.410\n","name":"stdout"}]},{"metadata":{"_uuid":"2edaca791a764dcbd8c255c622046a3262c9d343"},"cell_type":"markdown","source":"**Output RF's feature importance. Again, it's nice to present it as a DataFrame.**<br>\n**<font color='red'>Question 7:</font> What is the most important feature, according to the Random Forest model?**"},{"metadata":{"trusted":true,"_uuid":"c5fc1d874ba41c152e0c1dd11eba469320a91613"},"cell_type":"code","source":"rf_importance = pd.DataFrame({'feature_importance':locally_best_forest.best_estimator_.feature_importances_}, index = X.columns)\nrf_importance.sort_values(by = 'feature_importance', ascending = False)","execution_count":27,"outputs":[{"output_type":"execute_result","execution_count":27,"data":{"text/plain":"                      feature_importance\nalcohol                         0.224432\nvolatile acidity                0.119393\nfree sulfur dioxide             0.116147\npH                              0.072806\ntotal sulfur dioxide            0.071318\nresidual sugar                  0.070160\ndensity                         0.069367\nchlorides                       0.067982\nfixed acidity                   0.064268\ncitric acid                     0.062945\nsulphates                       0.061184","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>feature_importance</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>alcohol</th>\n      <td>0.224432</td>\n    </tr>\n    <tr>\n      <th>volatile acidity</th>\n      <td>0.119393</td>\n    </tr>\n    <tr>\n      <th>free sulfur dioxide</th>\n      <td>0.116147</td>\n    </tr>\n    <tr>\n      <th>pH</th>\n      <td>0.072806</td>\n    </tr>\n    <tr>\n      <th>total sulfur dioxide</th>\n      <td>0.071318</td>\n    </tr>\n    <tr>\n      <th>residual sugar</th>\n      <td>0.070160</td>\n    </tr>\n    <tr>\n      <th>density</th>\n      <td>0.069367</td>\n    </tr>\n    <tr>\n      <th>chlorides</th>\n      <td>0.067982</td>\n    </tr>\n    <tr>\n      <th>fixed acidity</th>\n      <td>0.064268</td>\n    </tr>\n    <tr>\n      <th>citric acid</th>\n      <td>0.062945</td>\n    </tr>\n    <tr>\n      <th>sulphates</th>\n      <td>0.061184</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"_uuid":"bc3c0e87a735ef74b1d1de869dd8ceae9fa42f9a"},"cell_type":"markdown","source":"**Make conclusions about the perdormance of the explored 3 models in this particular prediction task.**"}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}