deep_learning/keras_regression_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression Tutorial with the Keras\n",
    "\n",
    "The problem that we will look at in this tutorial is the Boston house price dataset. The dataset describes 13 numerical properties of houses in Boston suburbs and is concerned with modeling the price of houses in those suburbs in thousands of dollars. As such, this is a regression predictive modeling problem. Input attributes include things like crime rate, proportion of nonretail business acres, chemical concentrations and more. It is convenient to work with because all of the input and output attributes are numerical and there are 506 instances to work with.\n",
    "\n",
    "Reasonable performance for models evaluated using Mean Squared Error (MSE) are around 20 in squared thousands of dollars (or $4,500 if you take the square root). This is a nice target to aim for with our neural network model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "import pandas\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.wrappers.scikit_learn import KerasRegressor\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load dataset\n",
    "dataframe = pandas.read_csv(\"../data/housing.csv\", delim_whitespace=True, header=None)\n",
    "dataset = dataframe.values\n",
    "# split into input (X) and output (Y) variables\n",
    "X = dataset[:,0:13]\n",
    "Y = dataset[:,13]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using scikit-learn to evaluate model\n",
    "\n",
    "We can create Keras models and evaluate them with scikit-learn by using handy wrapper objects provided by the Keras library. This is desirable, because scikit-learn excels at evaluating models and will allow us to use powerful data preparation and model evaluation schemes with very few lines of code.\n",
    "\n",
    "The Keras wrappers require a function as an argument. This function that we must define is responsible for creating the neural network model to be evaluated.\n",
    "\n",
    "Below we define the function to create the baseline model to be evaluated. It is a simple model that has a single fully connected hidden layer with the same number of neurons as input attributes (13). The network uses good practices such as the rectifier activation function for the hidden layer. No activation function is used for the output layer because it is a regression problem and we are interested in predicting numerical values directly without transform.\n",
    "\n",
    "The efficient ADAM optimization algorithm is used and a mean squared error loss function is optimized. This will be the same metric that we will use to evaluate the performance of the model. It is a desirable metric because by taking the square root gives us an error value we can directly understand in the context of the problem (thousands of dollars)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def baseline_model():\n",
    "    # create model\n",
    "    model = Sequential()\n",
    "    model.add(Dense(13, input_dim=13, init='normal', activation='relu'))\n",
    "    model.add(Dense(1, init='normal'))\n",
    "    # Compile model\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fix random seed for reproducibility\n",
    "seed = 7\n",
    "numpy.random.seed(seed)\n",
    "# evaluate model with standardized dataset\n",
    "estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results: 32.80 (28.08) MSE\n"
     ]
    }
   ],
   "source": [
    "kfold = KFold(n_splits=10, random_state=seed)\n",
    "results = cross_val_score(estimator, X, Y, cv=kfold)\n",
    "print(\"Results: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling The Standardized Dataset\n",
    "\n",
    "An important concern with the Boston house price dataset is that the input attributes all vary in their scales because they measure different quantities. It is almost always good practice to prepare your data before modeling it using a neural network model. Continuing on from the above baseline model, we can re-evaluate the same model using a standardized version of the input dataset.\n",
    "\n",
    "We can use scikit-learn’s Pipeline framework to perform the standardization during the model evaluation process, within each fold of the cross validation. This ensures that there is no data leakage from each testset cross validation fold into the training data. The code below creates a scikit-learn Pipeline that first standardizes the dataset then creates and evaluate the baseline neural network model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Standardized: 30.05 (26.20) MSE\n"
     ]
    }
   ],
   "source": [
    "# evaluate model with standardized dataset\n",
    "numpy.random.seed(seed)\n",
    "estimators = []\n",
    "estimators.append(('standardize', StandardScaler()))\n",
    "estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, nb_epoch=50, batch_size=5, verbose=0)))\n",
    "pipeline = Pipeline(estimators)\n",
    "kfold = KFold(n_splits=10, random_state=seed)\n",
    "results = cross_val_score(pipeline, X, Y, cv=kfold)\n",
    "print(\"Standardized: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tune The Neural Network Topology\n",
    "\n",
    "There are many concerns that can be optimized for a neural network model. Perhaps the point of biggest leverage is the structure of the network itself, including the number of layers and the number of neurons in each layer. In this section we will evaluate two additional network topologies in an effort to further improve the performance of the model. We will look at both a deeper and a wider network topology.\n",
    "\n",
    "### Deeper Network\n",
    "\n",
    "One way to improve the performance a neural network is to add more layers. This might allow the model to extract and recombine higher order features embedded in the data. In this section we will evaluate the effect of adding one more hidden layer to the model. This is as easy as defining a new function that will create this deeper model, copied from our baseline model above. We can then insert a new line after the first hidden layer. In this case with about half the number of neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Larger: 103.31 (236.28) MSE\n"
     ]
    }
   ],
   "source": [
    "def larger_model():\n",
    "    # create model\n",
    "    model = Sequential()\n",
    "    model.add(Dense(13, input_dim=13, init='normal', activation='relu'))\n",
    "    model.add(Dense(6, init='normal', activation='relu'))\n",
    "    model.add(Dense(1, init='normal'))\n",
    "    # Compile model\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    return model\n",
    "\n",
    "numpy.random.seed(seed)\n",
    "estimators = []\n",
    "estimators.append(('standardize', StandardScaler()))\n",
    "estimators.append(('mlp', KerasRegressor(build_fn=larger_model, nb_epoch=50, batch_size=5, verbose=0)))\n",
    "pipeline = Pipeline(estimators)\n",
    "kfold = KFold(n_splits=10, random_state=seed)\n",
    "results = cross_val_score(pipeline, X, Y, cv=kfold)\n",
    "print(\"Larger: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate a Wider Network Topology\n",
    "\n",
    "Another approach to increasing the representational capability of the model is to create a wider network. In this section we evaluate the effect of keeping a shallow network architecture and nearly doubling the number of neurons in the one hidden layer. Again, all we need to do is define a new function that creates our neural network model. Here, we have increased the number of neurons in the hidden layer compared to the baseline model from 13 to 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wider: 21.50 (23.43) MSE\n"
     ]
    }
   ],
   "source": [
    "def wider_model():\n",
    "    # create model\n",
    "    model = Sequential()\n",
    "    model.add(Dense(20, input_dim=13, init='normal', activation='relu'))\n",
    "    model.add(Dense(1, init='normal'))\n",
    "    # Compile model\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    return model\n",
    "\n",
    "numpy.random.seed(seed)\n",
    "estimators = []\n",
    "estimators.append(('standardize', StandardScaler()))\n",
    "estimators.append(('mlp', KerasRegressor(build_fn=wider_model, nb_epoch=100, batch_size=5, verbose=0)))\n",
    "pipeline = Pipeline(estimators)\n",
    "kfold = KFold(n_splits=10, random_state=seed)\n",
    "results = cross_val_score(pipeline, X, Y, cv=kfold)\n",
    "print(\"Wider: %.2f (%.2f) MSE\" % (results.mean(), results.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building the model does see a further drop in error to about 21 thousand squared dollars. This is not a bad result for this problem. It would have been hard to guess that a wider network would outperform a deeper network on this problem. The results demonstrate the importance of empirical testing when it comes to developing neural network models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}