deep_learning/intro-tensorflow-datacamp.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Introduction to TensorFlow in Python\n",
    "### Defining data as constants\n",
    "Throughout this course, we will use tensorflow version 2.3 and will exclusively import the submodules needed to complete each exercise. This will usually be done for you, but you will do it in this exercise by importing constant from tensorflow.\n",
    "\n",
    "After you have imported constant, you will use it to transform a numpy array, credit_numpy, into a tensorflow constant, credit_constant. This array contains feature columns from a dataset on credit card holders and is previewed in the image below. We will return to this dataset in later chapters.\n",
    "\n",
    "Note that tensorflow 2 allows you to use data as either a numpy array or a tensorflow constant object. Using a constant will ensure that any operations performed with that object are done in tensorflow."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import constant from TensorFlow\n",
    "from tensorflow import constant\n",
    "\n",
    "# Convert the credit_numpy array into a tensorflow constant\n",
    "credit_constant = constant(credit_numpy)\n",
    "\n",
    "# Print constant datatype\n",
    "print('The datatype is:', credit_constant.dtype)\n",
    "\n",
    "# Print constant shape\n",
    "print('The shape is:', credit_constant.shape)"
   ]
  },
  {
   "source": [
    "### Defining variables\n",
    "Unlike a constant, a variable's value can be modified. This will be quite useful when we want to train a model by updating its parameters. Constants can't be used for this purpose, so variables are the natural choice.\n",
    "\n",
    "Let's try defining and working with a variable. Note that Variable(), which is used to create a variable tensor, has been imported from tensorflow and is available to use in the exercise."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 1-dimensional variable A1\n",
    "A1 = Variable([1, 2, 3, 4])\n",
    "\n",
    "# Print the variable A1\n",
    "print(A1)\n",
    "\n",
    "# Convert A1 to a numpy array and assign it to B1\n",
    "B1 = A1.numpy()\n",
    "\n",
    "# Print B1\n",
    "print(B1)"
   ]
  },
  {
   "source": [
    "### Performing element-wise multiplication\n",
    "Element-wise multiplication in TensorFlow is performed using two tensors with identical shapes. This is because the operation multiplies elements in corresponding positions in the two tensors. An example of an element-wise multiplication, denoted by the ⊙ symbol, is shown below:\n",
    "\n",
    "![](../resources/element-wise-multiplication.jpg)"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define tensors A1 and A23 as constants\n",
    "A1 = constant([1, 2, 3, 4])\n",
    "A23 = constant([[1, 2, 3], [1, 6, 4]])\n",
    "\n",
    "# Define B1 and B23 to have the correct shape\n",
    "B1 = ones_like(A1)\n",
    "B23 = ones_like(A23)\n",
    "\n",
    "# Perform element-wise multiplication\n",
    "C1 = multiply(A1, B1)\n",
    "C23 = multiply(A23, B23)\n",
    "\n",
    "# Print the tensors C1 and C23\n",
    "print('C1: {}'.format(C1.numpy()))\n",
    "print('C23: {}'.format(C23.numpy()))"
   ]
  },
  {
   "source": [
    "### Making predictions with matrix multiplication\n",
    "In later chapters, you will learn to train linear regression models. This process will yield a vector of parameters that can be multiplied by the input data to generate predictions. In this exercise, you will use input data, features, and a target vector, bill, which are taken from a credit card dataset we will use later in the course.\n",
    "\n",
    "![](../resources/matrix-mult.jpg)\n",
    "\n",
    "The matrix of input data, features, contains two columns: education level and age. The target vector, bill, is the size of the credit card borrower's bill.\n",
    "\n",
    "Since we have not trained the model, you will enter a guess for the values of the parameter vector, params. You will then use matmul() to perform matrix multiplication of features by params to generate predictions, billpred, which you will compare with bill. Note that we have imported matmul() and constant()."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define features, params, and bill as constants\n",
    "features = constant([[2, 24], [2, 26], [2, 57], [1, 37]])\n",
    "params = constant([[1000], [150]])\n",
    "bill = constant([[3913], [2682], [8617], [64400]])\n",
    "\n",
    "# Compute billpred using features and params\n",
    "billpred = matmul(features, params)\n",
    "\n",
    "# Compute and print the error\n",
    "error = bill - billpred\n",
    "print(error.numpy())"
   ]
  },
  {
   "source": [
    "### Reshaping tensors\n",
    "Later in the course, you will classify images of sign language letters using a neural network. In some cases, the network will take 1-dimensional tensors as inputs, but your data will come in the form of images, which will either be either 2- or 3-dimensional tensors, depending on whether they are grayscale or color images.\n",
    "\n",
    "The figure below shows grayscale and color images of the sign language letter A. The two images have been imported for you and converted to the numpy arrays gray_tensor and color_tensor. Reshape these arrays into 1-dimensional vectors using the reshape operation, which has been imported for you from tensorflow. Note that the shape of gray_tensor is 28x28 and the shape of color_tensor is 28x28x3."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape the grayscale image tensor into a vector\n",
    "gray_vector = reshape(gray_tensor, (784, 1))\n",
    "\n",
    "# Reshape the color image tensor into a vector\n",
    "color_vector = reshape(color_tensor, (2352, 1))"
   ]
  },
  {
   "source": [
    "### Optimizing with gradients\n",
    "You are given a loss function, y=x2, which you want to minimize. You can do this by computing the slope using the GradientTape() operation at different values of x. If the slope is positive, you can decrease the loss by lowering x. If it is negative, you can decrease it by increasing x.\n",
    "\n",
    "In practice, you will use a high level tensorflow operation to perform gradient descent automatically. In this exercise, however, you will compute the slope at x values of -1, 1, and 0. The following operations are available: GradientTape(), multiply(), and Variable()."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_gradient(x0):\n",
    "  \t# Define x as a variable with an initial value of x0\n",
    "\tx = Variable(x0)\n",
    "\twith GradientTape() as tape:\n",
    "\t\ttape.watch(x)\n",
    "        # Define y using the multiply operation\n",
    "\t\ty = multiply(x, x)\n",
    "    # Return the gradient of y with respect to x\n",
    "\treturn tape.gradient(y, x).numpy()\n",
    "\n",
    "# Compute and print gradients at x = -1, 1, and 0\n",
    "print(compute_gradient(-1.0))\n",
    "print(compute_gradient(1.0))\n",
    "print(compute_gradient(0.0))"
   ]
  },
  {
   "source": [
    "### Working with image data\n",
    "You are given a black-and-white image of a letter, which has been encoded as a tensor, letter. You want to determine whether the letter is an X or a K. You don't have a trained neural network, but you do have a simple model, model, which can be used to classify letter.\n",
    "\n",
    "The 3x3 tensor, letter, and the 1x3 tensor, model, are available in the Python shell. You can determine whether letter is a K by multiplying letter by model, summing over the result, and then checking if it is equal to 1. As with more complicated models, such as neural networks, model is a collection of weights, arranged in a tensor.\n",
    "\n",
    "Note that the functions reshape(), matmul(), and reduce_sum() have been imported from tensorflow and are available for use."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape model from a 1x3 to a 3x1 tensor\n",
    "model = reshape(model, (3, 1))\n",
    "\n",
    "# Multiply letter by model\n",
    "output = matmul(letter, model)\n",
    "\n",
    "# Sum over output and print prediction using the numpy method\n",
    "prediction = sum(output)\n",
    "print(prediction.numpy())"
   ]
  },
  {
   "source": [
    "### Load data using pandas\n",
    "Before you can train a machine learning model, you must first import data. There are several valid ways to do this, but for now, we will use a simple one-liner from pandas: pd.read_csv(). Recall from the video that the first argument specifies the path or URL. All other arguments are optional.\n",
    "\n",
    "In this exercise, you will import the King County housing dataset, which we will use to train a linear model later in the chapter."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas under the alias pd\n",
    "import pandas as pd\n",
    "\n",
    "# Assign the path to a string variable named data_path\n",
    "data_path = 'kc_house_data.csv'\n",
    "\n",
    "# Load the dataset as a dataframe named housing\n",
    "housing = pd.read_csv(data_path)\n",
    "\n",
    "# Print the price column of housing\n",
    "print(housing['price'])"
   ]
  },
  {
   "source": [
    "### Setting the data type\n",
    "In this exercise, you will both load data and set its type. Note that housing is available and pandas has been imported as pd. You will import numpy and tensorflow, and define tensors that are usable in tensorflow using columns in housing with a given data type. Recall that you can select the price column, for instance, from housing using housing['price']."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import numpy and tensorflow with their standard aliases\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# Use a numpy array to define price as a 32-bit float\n",
    "price = np.array(housing['price'], np.float32)\n",
    "\n",
    "# Define waterfront as a Boolean using cast\n",
    "waterfront = tf.cast(housing['waterfront'], tf.bool)\n",
    "\n",
    "# Print price and waterfront\n",
    "print(price)\n",
    "print(waterfront)"
   ]
  },
  {
   "source": [
    "### Loss functions in TensorFlow\n",
    "In this exercise, you will compute the loss using data from the King County housing dataset. You are given a target, price, which is a tensor of house prices, and predictions, which is a tensor of predicted house prices. You will evaluate the loss function and print out the value of the loss."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the keras module from tensorflow\n",
    "from tensorflow import keras\n",
    "\n",
    "# Compute the mean absolute error (mae)\n",
    "loss = keras.losses.mae(price, predictions)\n",
    "\n",
    "# Print the mean absolute error (mae)\n",
    "print(loss.numpy())"
   ]
  },
  {
   "source": [
    "### Modifying the loss function\n",
    "In the previous exercise, you defined a tensorflow loss function and then evaluated it once for a set of actual and predicted values. In this exercise, you will compute the loss within another function called loss_function(), which first generates predicted values from the data and variables. The purpose of this is to construct a function of the trainable model variables that returns the loss. You can then repeatedly evaluate this function for different variable values until you find the minimum. In practice, you will pass this function to an optimizer in tensorflow. Note that features and targets have been defined and are available. Additionally, Variable, float32, and keras are available."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a variable named scalar\n",
    "scalar = Variable(1.0, float32)\n",
    "\n",
    "# Define the model\n",
    "def model(scalar, features = features):\n",
    "  \treturn scalar * features\n",
    "\n",
    "# Define a loss function\n",
    "def loss_function(scalar, features = features, targets = targets):\n",
    "\t# Compute the predicted values\n",
    "\tpredictions = model(scalar, features)\n",
    "    \n",
    "\t# Return the mean absolute error loss\n",
    "\treturn keras.losses.mae(targets, predictions)\n",
    "\n",
    "# Evaluate the loss function and print the loss\n",
    "print(loss_function(scalar).numpy())"
   ]
  },
  {
   "source": [
    "### Set up a linear regression\n",
    "A univariate linear regression identifies the relationship between a single feature and the target tensor. In this exercise, we will use a property's lot size and price. Just as we discussed in the video, we will take the natural logarithms of both tensors, which are available as price_log and size_log.\n",
    "\n",
    "In this exercise, you will define the model and the loss function. You will then evaluate the loss function for two different values of intercept and slope. Remember that the predicted values are given by intercept + features*slope. Additionally, note that keras.losses.mse() is available for you. Furthermore, slope and intercept have been defined as variables."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a linear regression model\n",
    "def linear_regression(intercept, slope, features = size_log):\n",
    "\treturn intercept + features * slope\n",
    "\n",
    "# Set loss_function() to take the variables as arguments\n",
    "def loss_function(intercept, slope, features = size_log, targets = price_log):\n",
    "\t# Set the predicted values\n",
    "\tpredictions = linear_regression(intercept, slope, features)\n",
    "    \n",
    "    # Return the mean squared error loss\n",
    "\treturn keras.losses.mse(targets, predictions)\n",
    "\n",
    "# Compute the loss for different slope and intercept values\n",
    "print(loss_function(0.1, 0.1).numpy())\n",
    "print(loss_function(0.1, 0.5).numpy())"
   ]
  },
  {
   "source": [
    "### Train a linear model\n",
    "In this exercise, we will pick up where the previous exercise ended. The intercept and slope, intercept and slope, have been defined and initialized. Additionally, a function has been defined, loss_function(intercept, slope), which computes the loss using the data and model variables.\n",
    "\n",
    "You will now define an optimization operation as opt. You will then train a univariate linear model by minimizing the loss to find the optimal values of intercept and slope. Note that the opt operation will try to move closer to the optimum with each step, but will require many steps to find it. Thus, you must repeatedly execute the operation."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize an adam optimizer\n",
    "opt = keras.optimizers.Adam(0.5)\n",
    "\n",
    "for j in range(100):\n",
    "\t# Apply minimize, pass the loss function, and supply the variables\n",
    "\topt.minimize(lambda: loss_function(intercept, slope), var_list=[intercept, slope])\n",
    "\n",
    "\t# Print every 10th value of the loss\n",
    "\tif j % 10 == 0:\n",
    "\t\tprint(loss_function(intercept, slope).numpy())\n",
    "\n",
    "# Plot data and regression line\n",
    "plot_results(intercept, slope)"
   ]
  },
  {
   "source": [
    "### Multiple linear regression\n",
    "In most cases, performing a univariate linear regression will not yield a model that is useful for making accurate predictions. In this exercise, you will perform a multiple regression, which uses more than one feature.\n",
    "\n",
    "You will use price_log as your target and size_log and bedrooms as your features. Each of these tensors has been defined and is available. You will also switch from using the the mean squared error loss to the mean absolute error loss: keras.losses.mae(). Finally, the predicted values are computed as follows: params[0] + feature1*params[1] + feature2*params[2]. Note that we've defined a vector of parameters, params, as a variable, rather than using three variables. Here, params[0] is the intercept and params[1] and params[2] are the slopes."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the linear regression model\n",
    "def linear_regression(params, feature1 = size_log, feature2 = bedrooms):\n",
    "\treturn params[0] + feature1*params[1] + feature2*params[2]\n",
    "\n",
    "# Define the loss function\n",
    "def loss_function(params, targets = price_log, feature1 = size_log, feature2 = bedrooms):\n",
    "\t# Set the predicted values\n",
    "\tpredictions = linear_regression(params, feature1, feature2)\n",
    "  \n",
    "\t# Use the mean absolute error loss\n",
    "\treturn keras.losses.mae(targets, predictions)\n",
    "\n",
    "# Define the optimize operation\n",
    "opt = keras.optimizers.Adam()\n",
    "\n",
    "# Perform minimization and print trainable variables\n",
    "for j in range(10):\n",
    "\topt.minimize(lambda: loss_function(params), var_list=[params])\n",
    "\tprint_results(params)"
   ]
  },
  {
   "source": [
    "### Preparing to batch train\n",
    "Before we can train a linear model in batches, we must first define variables, a loss function, and an optimization operation. In this exercise, we will prepare to train a model that will predict price_batch, a batch of house prices, using size_batch, a batch of lot sizes in square feet. In contrast to the previous lesson, we will do this by loading batches of data using pandas, converting it to numpy arrays, and then using it to minimize the loss function in steps.\n",
    "\n",
    "Variable(), keras(), and float32 have been imported for you. Note that you should not set default argument values for either the model or loss function, since we will generate the data in batches during the training process."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the intercept and slope\n",
    "intercept = Variable(10.0, float32)\n",
    "slope = Variable(0.5, float32)\n",
    "\n",
    "# Define the model\n",
    "def linear_regression(intercept, slope, features):\n",
    "\t# Define the predicted values\n",
    "\treturn intercept + features*slope\n",
    "\n",
    "# Define the loss function\n",
    "def loss_function(intercept, slope, targets, features):\n",
    "\t# Define the predicted values\n",
    "\tpredictions = linear_regression(intercept, slope, features)\n",
    "    \n",
    " \t# Define the MSE loss\n",
    "\treturn keras.losses.mse(targets, predictions)"
   ]
  },
  {
   "source": [
    "### Training a linear model in batches\n",
    "In this exercise, we will train a linear regression model in batches, starting where we left off in the previous exercise. We will do this by stepping through the dataset in batches and updating the model's variables, intercept and slope, after each step. This approach will allow us to train with datasets that are otherwise too large to hold in memory.\n",
    "\n",
    "Note that the loss function,loss_function(intercept, slope, targets, features), has been defined for you. Additionally, keras has been imported for you and numpy is available as np. The trainable variables should be entered into var_list in the order in which they appear as loss function arguments."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize adam optimizer\n",
    "opt = keras.optimizers.Adam()\n",
    "\n",
    "# Load data in batches\n",
    "for batch in pd.read_csv('kc_house_data.csv', chunksize=100):\n",
    "\tsize_batch = np.array(batch['sqft_lot'], np.float32)\n",
    "\n",
    "\t# Extract the price values for the current batch\n",
    "\tprice_batch = np.array(batch['price'], np.float32)\n",
    "\n",
    "\t# Complete the loss, fill in the variable list, and minimize\n",
    "\topt.minimize(lambda: loss_function(intercept, slope, price_batch, size_batch), var_list=[intercept, slope])\n",
    "\n",
    "# Print trained parameters\n",
    "print(intercept.numpy(), slope.numpy())"
   ]
  },
  {
   "source": [
    "### The linear algebra of dense layers\n",
    "There are two ways to define a dense layer in tensorflow. The first involves the use of low-level, linear algebraic operations. The second makes use of high-level keras operations. In this exercise, we will use the first method to construct the network shown in the image below.\n",
    "\n",
    "![](../resources/3_2_1_network2.png)\n",
    "\n",
    "The input layer contains 3 features -- education, marital status, and age -- which are available as borrower_features. The hidden layer contains 2 nodes and the output layer contains a single node.\n",
    "\n",
    "For each layer, you will take the previous layer as an input, initialize a set of weights, compute the product of the inputs and weights, and then apply an activation function. Note that Variable(), ones(), matmul(), and keras() have been imported from tensorflow."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize bias1\n",
    "bias1 = Variable(1.0)\n",
    "\n",
    "# Initialize weights1 as 3x2 variable of ones\n",
    "weights1 = Variable(ones((3, 2)))\n",
    "\n",
    "# Perform matrix multiplication of borrower_features and weights1\n",
    "product1 = matmul(borrower_features, weights1)\n",
    "\n",
    "# Apply sigmoid activation function to product1 + bias1\n",
    "dense1 = keras.activations.sigmoid(product1  + bias1)\n",
    "\n",
    "# Print shape of dense1\n",
    "print(\"\\n dense1's output shape: {}\".format(dense1.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From previous step\n",
    "bias1 = Variable(1.0)\n",
    "weights1 = Variable(ones((3, 2)))\n",
    "product1 = matmul(borrower_features, weights1)\n",
    "dense1 = keras.activations.sigmoid(product1 + bias1)\n",
    "\n",
    "# Initialize bias2 and weights2\n",
    "bias2 = Variable(1.0)\n",
    "weights2 = Variable(ones((2, 1)))\n",
    "\n",
    "# Perform matrix multiplication of dense1 and weights2\n",
    "product2 = matmul(dense1, weights2)\n",
    "\n",
    "# Apply activation to product2 + bias2 and print the prediction\n",
    "prediction = keras.activations.sigmoid(product2 + bias2)\n",
    "print('\\n prediction: {}'.format(prediction.numpy()[0,0]))\n",
    "print('\\n actual: 1')"
   ]
  },
  {
   "source": [
    "### The low-level approach with multiple examples\n",
    "In this exercise, we'll build further intuition for the low-level approach by constructing the first dense hidden layer for the case where we have multiple examples. We'll assume the model is trained and the first layer weights, weights1, and bias, bias1, are available. We'll then perform matrix multiplication of the borrower_features tensor by the weights1 variable. Recall that the borrower_features tensor includes education, marital status, and age. Finally, we'll apply the sigmoid function to the elements of products1 + bias1, yielding dense1."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the product of borrower_features and weights1\n",
    "products1 = matmul(borrower_features, weights1)\n",
    "\n",
    "# Apply a sigmoid activation function to products1 + bias1\n",
    "dense1 = keras.activations.sigmoid(products1 + bias1)\n",
    "\n",
    "# Print the shapes of borrower_features, weights1, bias1, and dense1\n",
    "print('\\n shape of borrower_features: ', borrower_features.shape)\n",
    "print('\\n shape of weights1: ', weights1.shape)\n",
    "print('\\n shape of bias1: ', bias1.shape)\n",
    "print('\\n shape of dense1: ', dense1.shape)"
   ]
  },
  {
   "source": [
    "### Using the dense layer operation\n",
    "We've now seen how to define dense layers in tensorflow using linear algebra. In this exercise, we'll skip the linear algebra and let keras work out the details. This will allow us to construct the network below, which has 2 hidden layers and 10 features, using less code than we needed for the network with 1 hidden layer and 3 features.\n",
    "\n",
    "![](../resources/10_7_3_1_network.png)\n",
    "\n",
    "To construct this network, we'll need to define three dense layers, each of which takes the previous layer as an input, multiplies it by weights, and applies an activation function. Note that input data has been defined and is available as a 100x10 tensor: borrower_features. Additionally, the keras.layers module is available."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the first dense layer\n",
    "dense1 = keras.layers.Dense(7, activation='sigmoid')(borrower_features)\n",
    "\n",
    "# Define a dense layer with 3 output nodes\n",
    "dense2 = keras.layers.Dense(3, activation='sigmoid')(dense1)\n",
    "\n",
    "# Define a dense layer with 1 output node\n",
    "predictions = keras.layers.Dense(1, activation='sigmoid')(dense2)\n",
    "\n",
    "# Print the shapes of dense1, dense2, and predictions\n",
    "print('\\n shape of dense1: ', dense1.shape)\n",
    "print('\\n shape of dense2: ', dense2.shape)\n",
    "print('\\n shape of predictions: ', predictions.shape)"
   ]
  },
  {
   "source": [
    "### Binary classification problems\n",
    "In this exercise, you will again make use of credit card data. The target variable, default, indicates whether a credit card holder defaults on his or her payment in the following period. Since there are only two options--default or not--this is a binary classification problem. While the dataset has many features, you will focus on just three: the size of the three latest credit card bills. Finally, you will compute predictions from your untrained network, outputs, and compare those the target variable, default.\n",
    "\n",
    "The tensor of features has been loaded and is available as bill_amounts. Additionally, the constant(), float32, and keras.layers.Dense() operations are available."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct input layer from features\n",
    "inputs = constant(bill_amounts, float32)\n",
    "\n",
    "# Define first dense layer\n",
    "dense1 = keras.layers.Dense(3, activation='relu')(inputs)\n",
    "\n",
    "# Define second dense layer\n",
    "dense2 = keras.layers.Dense(2, activation='relu')(dense1)\n",
    "\n",
    "# Define output layer\n",
    "outputs = keras.layers.Dense(1, activation='sigmoid')(dense2)\n",
    "\n",
    "# Print error for first five examples\n",
    "error = default[:5] - outputs.numpy()[:5]\n",
    "print(error)"
   ]
  },
  {
   "source": [
    "### Multiclass classification problems\n",
    "In this exercise, we expand beyond binary classification to cover multiclass problems. A multiclass problem has targets that can take on three or more values. In the credit card dataset, the education variable can take on 6 different values, each corresponding to a different level of education. We will use that as our target in this exercise and will also expand the feature set from 3 to 10 columns.\n",
    "\n",
    "As in the previous problem, you will define an input layer, dense layers, and an output layer. You will also print the untrained model's predictions, which are probabilities assigned to the classes. The tensor of features has been loaded and is available as borrower_features. Additionally, the constant(), float32, and keras.layers.Dense() operations are available."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct input layer from borrower features\n",
    "inputs = constant(borrower_features, float32)\n",
    "\n",
    "# Define first dense layer\n",
    "dense1 = keras.layers.Dense(10, activation='sigmoid')(inputs)\n",
    "\n",
    "# Define second dense layer\n",
    "dense2 = keras.layers.Dense(8, activation='relu')(dense1)\n",
    "\n",
    "# Define output layer\n",
    "outputs = keras.layers.Dense(6, activation='softmax')(dense2)\n",
    "\n",
    "# Print first five predictions\n",
    "print(outputs.numpy()[:5])"
   ]
  },
  {
   "source": [
    "### The dangers of local minima\n",
    "In this exercise, you will try to find the global minimum of loss_function() using keras.optimizers.SGD(). You will do this twice, each time with a different initial value of the input to loss_function(). First, you will use x_1, which is a variable with an initial value of 6.0. Second, you will use x_2, which is a variable with an initial value of 0.3. Note that loss_function() has been defined and is available."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize x_1 and x_2\n",
    "x_1 = Variable(6.0,float32)\n",
    "x_2 = Variable(0.3,float32)\n",
    "\n",
    "# Define the optimization operation\n",
    "opt = keras.optimizers.SGD(learning_rate=0.01)\n",
    "\n",
    "for j in range(100):\n",
    "\t# Perform minimization using the loss function and x_1\n",
    "\topt.minimize(lambda: loss_function(x_1), var_list=[x_1])\n",
    "\t# Perform minimization using the loss function and x_2\n",
    "\topt.minimize(lambda: loss_function(x_2), var_list=[x_2])\n",
    "\n",
    "# Print x_1 and x_2 as numpy arrays\n",
    "print(x_1.numpy(), x_2.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Avoiding local minima\n",
    "The previous problem showed how easy it is to get stuck in local minima. We had a simple optimization problem in one variable and gradient descent still failed to deliver the global minimum when we had to travel through local minima first. One way to avoid this problem is to use momentum, which allows the optimizer to break through local minima. We will again use the loss function from the previous problem, which has been defined and is available for you as loss_function().\n",
    "\n",
    "Several optimizers in tensorflow have a momentum parameter, including SGD and RMSprop. You will make use of RMSprop in this exercise. Note that x_1 and x_2 have been initialized to the same value this time. Furthermore, keras.optimizers.RMSprop() has also been imported for you from tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize x_1 and x_2\n",
    "x_1 = Variable(0.05,float32)\n",
    "x_2 = Variable(0.05,float32)\n",
    "\n",
    "# Define the optimization operation for opt_1 and opt_2\n",
    "opt_1 = keras.optimizers.RMSprop(learning_rate=0.01, momentum=0.99)\n",
    "opt_2 = keras.optimizers.RMSprop(learning_rate=0.01, momentum=0.00)\n",
    "\n",
    "for j in range(100):\n",
    "\topt_1.minimize(lambda: loss_function(x_1), var_list=[x_1])\n",
    "    # Define the minimization operation for opt_2\n",
    "\topt_2.minimize(lambda: loss_function(x_2), var_list=[x_2])\n",
    "\n",
    "# Print x_1 and x_2 as numpy arrays\n",
    "print(x_1.numpy(), x_2.numpy())"
   ]
  },
  {
   "source": [
    "### Initialization in TensorFlow\n",
    "A good initialization can reduce the amount of time needed to find the global minimum. In this exercise, we will initialize weights and biases for a neural network that will be used to predict credit card default decisions. To build intuition, we will use the low-level, linear algebraic approach, rather than making use of convenience functions and high-level keras operations. We will also expand the set of input features from 3 to 23. Several operations have been imported from tensorflow: Variable(), random(), and ones()."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the layer 1 weights\n",
    "w1 = Variable(random.normal([23, 7]))\n",
    "\n",
    "# Initialize the layer 1 bias\n",
    "b1 = Variable(ones([7]))\n",
    "\n",
    "# Define the layer 2 weights\n",
    "w2 = Variable(random.normal([7, 1]))\n",
    "\n",
    "# Define the layer 2 bias\n",
    "b2 = Variable(0.0)"
   ]
  },
  {
   "source": [
    "### Defining the model and loss function\n",
    "In this exercise, you will train a neural network to predict whether a credit card holder will default. The features and targets you will use to train your network are available in the Python shell as borrower_features and default. You defined the weights and biases in the previous exercise.\n",
    "\n",
    "Note that the predictions layer is defined as σ(layer1∗w2+b2), where σ is the sigmoid activation, layer1 is a tensor of nodes for the first hidden dense layer, w2 is a tensor of weights, and b2 is the bias tensor.\n",
    "\n",
    "The trainable variables are w1, b1, w2, and b2. Additionally, the following operations have been imported for you: keras.activations.relu() and keras.layers.Dropout()."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the model\n",
    "def model(w1, b1, w2, b2, features = borrower_features):\n",
    "\t# Apply relu activation functions to layer 1\n",
    "\tlayer1 = keras.activations.relu(matmul(features, w1) + b1)\n",
    "    # Apply dropout\n",
    "\tdropout = keras.layers.Dropout(0.25)(layer1)\n",
    "\treturn keras.activations.sigmoid(matmul(dropout, w2) + b2)\n",
    "\n",
    "# Define the loss function\n",
    "def loss_function(w1, b1, w2, b2, features = borrower_features, targets = default):\n",
    "\tpredictions = model(w1, b1, w2, b2)\n",
    "\t# Pass targets and predictions to the cross entropy loss\n",
    "\treturn keras.losses.binary_crossentropy(targets, predictions)"
   ]
  },
  {
   "source": [
    "### Training neural networks with TensorFlow\n",
    "In the previous exercise, you defined a model, model(w1, b1, w2, b2, features), and a loss function, loss_function(w1, b1, w2, b2, features, targets), both of which are available to you in this exercise. You will now train the model and then evaluate its performance by predicting default outcomes in a test set, which consists of test_features and test_targets and is available to you. The trainable variables are w1, b1, w2, and b2. Additionally, the following operations have been imported for you: keras.activations.relu() and keras.layers.Dropout()."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model\n",
    "for j in range(100):\n",
    "    # Complete the optimizer\n",
    "\topt.minimize(lambda: loss_function(w1, b1, w2, b2), \n",
    "                 var_list=[w1, b1, w2, b2])\n",
    "\n",
    "# Make predictions with model\n",
    "model_predictions = model(w1, b1, w2, b2, test_features)\n",
    "\n",
    "# Construct the confusion matrix\n",
    "confusion_matrix(test_targets, model_predictions)"
   ]
  },
  {
   "source": [
    "### The sequential model in Keras\n",
    "In chapter 3, we used components of the keras API in tensorflow to define a neural network, but we stopped short of using its full capabilities to streamline model definition and training. In this exercise, you will use the keras sequential model API to define a neural network that can be used to classify images of sign language letters. You will also use the .summary() method to print the model's architecture, including the shape and number of parameters associated with each layer.\n",
    "\n",
    "Note that the images were reshaped from (28, 28) to (784,), so that they could be used as inputs to a dense layer. Additionally, note that keras has been imported from tensorflow for you."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a Keras sequential model\n",
    "model = keras.Sequential()\n",
    "\n",
    "# Define the first dense layer\n",
    "model.add(keras.layers.Dense(16, activation='relu', input_shape=(784,)))\n",
    "\n",
    "# Define the second dense layer\n",
    "model.add(keras.layers.Dense(8, activation='relu'))\n",
    "\n",
    "# Define the output layer\n",
    "model.add(keras.layers.Dense(4, activation='softmax'))\n",
    "\n",
    "# Print the model architecture\n",
    "print(model.summary())"
   ]
  },
  {
   "source": [
    "### Compiling a sequential model\n",
    "In this exercise, you will work towards classifying letters from the Sign Language MNIST dataset; however, you will adopt a different network architecture than what you used in the previous exercise. There will be fewer layers, but more nodes. You will also apply dropout to prevent overfitting. Finally, you will compile the model to use the adam optimizer and the categorical_crossentropy loss. You will also use a method in keras to summarize your model's architecture. Note that keras has been imported from tensorflow for you and a sequential keras model has been defined as model."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the first dense layer\n",
    "model.add(keras.layers.Dense(16, activation='sigmoid', input_shape=(784,)))\n",
    "\n",
    "# Apply dropout to the first layer's output\n",
    "model.add(keras.layers.Dropout(0.25))\n",
    "\n",
    "# Define the output layer\n",
    "model.add(keras.layers.Dense(4, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile('adam', loss='categorical_crossentropy')\n",
    "\n",
    "# Print a model summary\n",
    "print(model.summary())"
   ]
  },
  {
   "source": [
    "### Defining a multiple input model\n",
    "In some cases, the sequential API will not be sufficiently flexible to accommodate your desired model architecture and you will need to use the functional API instead. If, for instance, you want to train two models with different architectures jointly, you will need to use the functional API to do this. In this exercise, we will see how to do this. We will also use the .summary() method to examine the joint model's architecture.\n",
    "\n",
    "Note that keras has been imported from tensorflow for you. Additionally, the input layers of the first and second models have been defined as m1_inputs and m2_inputs, respectively. Note that the two models have the same architecture, but one of them uses a sigmoid activation in the first layer and the other uses a relu."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For model 1, pass the input layer to layer 1 and layer 1 to layer 2\n",
    "m1_layer1 = keras.layers.Dense(12, activation='sigmoid')(m1_inputs)\n",
    "m1_layer2 = keras.layers.Dense(4, activation='softmax')(m1_layer1)\n",
    "\n",
    "# For model 2, pass the input layer to layer 1 and layer 1 to layer 2\n",
    "m2_layer1 = keras.layers.Dense(12, activation='relu')(m2_inputs)\n",
    "m2_layer2 = keras.layers.Dense(4, activation='softmax')(m2_layer1)\n",
    "\n",
    "# Merge model outputs and define a functional model\n",
    "merged = keras.layers.add([m1_layer2, m2_layer2])\n",
    "model = keras.Model(inputs=[m1_inputs, m2_inputs], outputs=merged)\n",
    "\n",
    "# Print a model summary\n",
    "print(model.summary())"
   ]
  },
  {
   "source": [
    "### Training with Keras\n",
    "In this exercise, we return to our sign language letter classification problem. We have 2000 images of four letters--A, B, C, and D--and we want to classify them with a high level of accuracy. We will complete all parts of the problem, including the model definition, compilation, and training.\n",
    "\n",
    "Note that keras has been imported from tensorflow for you. Additionally, the features are available as sign_language_features and the targets are available as sign_language_labels."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a sequential model\n",
    "model = keras.Sequential()\n",
    "\n",
    "# Define a hidden layer\n",
    "model.add(keras.layers.Dense(16, activation='relu', input_shape=(784,)))\n",
    "\n",
    "# Define the output layer\n",
    "model.add(keras.layers.Dense(4, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile('SGD', loss='categorical_crossentropy')\n",
    "\n",
    "# Complete the fitting operation\n",
    "model.fit(sign_language_features, sign_language_labels, epochs=5)"
   ]
  },
  {
   "source": [
    "### Metrics and validation with Keras\n",
    "We trained a model to predict sign language letters in the previous exercise, but it is unclear how successful we were in doing so. In this exercise, we will try to improve upon the interpretability of our results. Since we did not use a validation split, we only observed performance improvements within the training set; however, it is unclear how much of that was due to overfitting. Furthermore, since we did not supply a metric, we only saw decreases in the loss function, which do not have any clear interpretation.\n",
    "\n",
    "Note that keras has been imported for you from tensorflow."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define sequential model\n",
    "model = keras.Sequential()\n",
    "\n",
    "# Define the first layer\n",
    "model.add(keras.layers.Dense(32, activation='sigmoid', input_shape=(784,)))\n",
    "\n",
    "# Add activation function to classifier\n",
    "model.add(keras.layers.Dense(4, activation='softmax'))\n",
    "\n",
    "# Set the optimizer, loss function, and metrics\n",
    "model.compile(optimizer='RMSprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Add the number of epochs and the validation split\n",
    "model.fit(sign_language_features, sign_language_labels, epochs=10, validation_split=0.1)"
   ]
  },
  {
   "source": [
    "### Overfitting detection\n",
    "In this exercise, we'll work with a small subset of the examples from the original sign language letters dataset. A small sample, coupled with a heavily-parameterized model, will generally lead to overfitting. This means that your model will simply memorize the class of each example, rather than identifying features that generalize to many examples.\n",
    "\n",
    "You will detect overfitting by checking whether the validation sample loss is substantially higher than the training sample loss and whether it increases with further training. With a small sample and a high learning rate, the model will struggle to converge on an optimum. You will set a low learning rate for the optimizer, which will make it easier to identify overfitting."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define sequential model\n",
    "model = keras.Sequential()\n",
    "\n",
    "# Define the first layer\n",
    "model.add(keras.layers.Dense(1024, activation='relu', input_shape=(784,)))\n",
    "\n",
    "# Add activation function to classifier\n",
    "model.add(keras.layers.Dense(4, activation='softmax'))\n",
    "\n",
    "# Finish the model compilation\n",
    "model.compile(optimizer=keras.optimizers.Adam(lr=0.001), \n",
    "              loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Complete the model fit operation\n",
    "model.fit(sign_language_features, sign_language_labels, epochs=50, validation_split=0.5)"
   ]
  },
  {
   "source": [
    "### Evaluating models\n",
    "Two models have been trained and are available: large_model, which has many parameters; and small_model, which has fewer parameters. Both models have been trained using train_features and train_labels, which are available to you. A separate test set, which consists of test_features and test_labels, is also available.\n",
    "\n",
    "Your goal is to evaluate relative model performance and also determine whether either model exhibits signs of overfitting. You will do this by evaluating large_model and small_model on both the train and test sets. For each model, you can do this by applying the .evaluate(x, y) method to compute the loss for features x and labels y. You will then compare the four losses generated."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the small model using the train data\n",
    "small_train = small_model.evaluate(train_features, train_labels)\n",
    "\n",
    "# Evaluate the small model using the test data\n",
    "small_test = small_model.evaluate(test_features, test_labels)\n",
    "\n",
    "# Evaluate the large model using the train data\n",
    "large_train = large_model.evaluate(train_features, train_labels)\n",
    "\n",
    "# Evaluate the large model using the test data\n",
    "large_test = large_model.evaluate(test_features, test_labels)\n",
    "\n",
    "# Print losses\n",
    "print('\\n Small - Train: {}, Test: {}'.format(small_train, small_test))\n",
    "print('Large - Train: {}, Test: {}'.format(large_train, large_test))"
   ]
  },
  {
   "source": [
    "### Preparing to train with Estimators\n",
    "For this exercise, we'll return to the King County housing transaction dataset from chapter 2. We will again develop and train a machine learning model to predict house prices; however, this time, we'll do it using the estimator API.\n",
    "\n",
    "Rather than completing everything in one step, we'll break this procedure down into parts. We'll begin by defining the feature columns and loading the data. In the next exercise, we'll define and train a premade estimator. Note that feature_column has been imported for you from tensorflow. Additionally, numpy has been imported as np, and the Kings County housing dataset is available as a pandas DataFrame: housing."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define feature columns for bedrooms and bathrooms\n",
    "bedrooms = feature_column.numeric_column(\"bedrooms\")\n",
    "bathrooms = feature_column.numeric_column(\"bathrooms\")\n",
    "\n",
    "# Define the list of feature columns\n",
    "feature_list = [bedrooms, bathrooms]\n",
    "\n",
    "def input_fn():\n",
    "\t# Define the labels\n",
    "\tlabels = np.array(housing['price'])\n",
    "\t# Define the features\n",
    "\tfeatures = {'bedrooms':np.array(housing['bedrooms']), \n",
    "                'bathrooms':housing['bathrooms']}\n",
    "\treturn features, labels"
   ]
  },
  {
   "source": [
    "### Defining Estimators\n",
    "In the previous exercise, you defined a list of feature columns, feature_list, and a data input function, input_fn(). In this exercise, you will build on that work by defining an estimator that makes use of input data."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a deep neural network regressor with 2 nodes in both the first and second hidden layers and 1 training step\n",
    "# Define the model and set the number of steps\n",
    "model = estimator.DNNRegressor(feature_columns=feature_list, hidden_units=[2,2])\n",
    "model.train(input_fn, steps=1)\n",
    "\n",
    "# Modify the code to use a LinearRegressor(), remove the hidden_units, and set the number of steps to 2\n",
    "# Define the model and set the number of steps\n",
    "model = estimator.LinearRegressor(feature_columns=feature_list)\n",
    "model.train(input_fn, steps=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}