deep_learning/advanced-keras-datacamp.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Advanced Deep Learning with Keras"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### Input layers\n",
    "The first step in creating a neural network model is to define the Input layer. This layer takes in raw data, usually in the form of numpy arrays. The shape of the Input layer defines how many variables your neural network will use. For example, if the input data has 10 columns, you define an Input layer with a shape of (10,).\n",
    "\n",
    "### Dense layers\n",
    "Once you have an Input layer, the next step is to add a Dense layer.\n",
    "\n",
    "Dense layers learn a weight matrix, where the first dimension of the matrix is the dimension of the input data, and the second dimension is the dimension of the output data. Recall that your Input layer has a shape of 1. In this case, your output layer will also have a shape of 1. This means that the Dense layer will learn a 1x1 weight matrix."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load layers\n",
    "from keras.layers import Input, Dense\n",
    "\n",
    "# Input layer\n",
    "input_tensor = Input(shape=(1,))\n",
    "\n",
    "# Dense layer\n",
    "output_layer = Dense(1)\n",
    "\n",
    "# Connect the dense layer to the input_tensor\n",
    "output_tensor = output_layer(input_tensor)"
   ]
  },
  {
   "source": [],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### Output layers\n",
    "Output layers are simply Dense layers! Output layers are used to reduce the dimension of the inputs to the dimension of the outputs. You'll learn more about output dimensions in chapter 4, but for now, you'll always use a single output in your neural networks, which is equivalent to Dense(1) or a dense layer with a single unit."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load layers\n",
    "from keras.layers import Input, Dense\n",
    "\n",
    "# Input layer\n",
    "input_tensor = Input(shape=(1,))\n",
    "\n",
    "# Create a dense layer and connect the dense layer to the input_tensor in one step\n",
    "# Note that we did this in 2 steps in the previous exercise, but are doing it in one step now\n",
    "output_tensor = Dense(1)(input_tensor)"
   ]
  },
  {
   "source": [
    "### Build a model\n",
    "Once you've defined an input layer and an output layer, you can build a Keras model. The model object is how you tell Keras where the model starts and stops: where data comes in and where predictions come out."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input/dense/output layers\n",
    "from keras.layers import Input, Dense\n",
    "input_tensor = Input(shape=(1,))\n",
    "output_tensor = Dense(1)(input_tensor)\n",
    "\n",
    "# Build the model\n",
    "from keras.models import Model\n",
    "model = Model(input_tensor, output_tensor)"
   ]
  },
  {
   "source": [
    "### Compile a model\n",
    "The final step in creating a model is compiling it. Now that you've created a model, you have to compile it before you can fit it to data. This finalizes your model, freezes all its settings, and prepares it to meet some data!\n",
    "\n",
    "During compilation, you specify the optimizer to use for fitting the model to the data, and a loss function. 'adam' is a good default optimizer to use, and will generally work well. Loss function depends on the problem at hand. Mean squared error is a common loss function and will optimize for predicting the mean, as is done in least squares regression.\n",
    "\n",
    "Mean absolute error optimizes for the median and is used in quantile regression. For this dataset, 'mean_absolute_error' works pretty well, so use it as your loss function.\n",
    "\n",
    "### Visualize a model\n",
    "Now that you've compiled the model, take a look a the result of your hard work! You can do this by looking at the model summary, as well as its plot.\n",
    "\n",
    "The summary will tell you the names of the layers, as well as how many units they have and how many parameters are in the model. The plot will show how the layers connect to each other."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_absolute_error')\n",
    "\n",
    "# Import the plotting function\n",
    "from keras.utils import plot_model\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Summarize the model\n",
    "model.summary()\n",
    "\n",
    "# Plot the model\n",
    "plot_model(model, to_file='model.png')\n",
    "\n",
    "# Display the image\n",
    "data = plt.imread('model.png')\n",
    "plt.imshow(data)\n",
    "plt.show()"
   ]
  },
  {
   "source": [
    "### Fit the model to the tournament basketball data\n",
    "Now that the model is compiled, you are ready to fit it to some data!\n",
    "\n",
    "In this exercise, you'll use a dataset of scores from US College Basketball tournament games. Each row of the dataset has the team ids: team_1 and team_2, as integers. It also has the seed difference between the teams (seeds are assigned by the tournament committee and represent a ranking of how strong the teams are) and the score difference of the game (e.g. if team_1 wins by 5 points, the score difference is 5).\n",
    "\n",
    "To fit the model, you provide a matrix of X variables (in this case one column: the seed difference) and a matrix of Y variables (in this case one column: the score difference).\n",
    "\n",
    "### Evaluate the model on a test set\n",
    "After fitting the model, you can evaluate it on new data. You will give the model a new X matrix (also called test data), allow it to make predictions, and then compare to the known y variable (also called target data).\n",
    "\n",
    "In this case, you'll use data from the post-season tournament to evaluate your model. The tournament games happen after the regular season games you used to train our model, and are therefore a good evaluation of how well your model performs out-of-sample."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now fit the model\n",
    "model.fit(games_tourney_train['seed_diff'], games_tourney_train['score_diff'],\n",
    "          epochs=1,\n",
    "          batch_size=128,\n",
    "          validation_split=0.1,\n",
    "          verbose=True)\n",
    "\n",
    "# Load the X variable from the test data\n",
    "X_test = games_tourney_test['seed_diff']\n",
    "\n",
    "# Load the y variable from the test data\n",
    "y_test = games_tourney_test['score_diff']\n",
    "\n",
    "# Evaluate the model on the test data\n",
    "print(model.evaluate(X_test, y_test, verbose=False))"
   ]
  },
  {
   "source": [
    "## Categorical Embeddings\n",
    "### Define team lookup\n",
    "Shared layers allow a model to use the same weight matrix for multiple steps. In this exercise, you will build a \"team strength\" layer that represents each team by a single number. You will use this number for both teams in the model. The model will learn a number for each team that works well both when the team is team_1 and when the team is team_2 in the input data.\n",
    "\n",
    "- Count the number of unique teams.\n",
    "- Create an embedding layer that maps each team ID to a single number representing that team's strength.\n",
    "- The output shape should be 1 dimension (as we want to represent the teams by a single number).\n",
    "- The input length should be 1 dimension (as each team is represented by exactly one id)."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "from keras.layers import Embedding\n",
    "from numpy import unique\n",
    "\n",
    "# Count the unique number of teams\n",
    "n_teams = unique(games_season['team_1']).shape[0]\n",
    "\n",
    "# Create an embedding layer\n",
    "team_lookup = Embedding(input_dim=n_teams,\n",
    "                        output_dim=1,\n",
    "                        input_length=1,\n",
    "                        name='Team-Strength')"
   ]
  },
  {
   "source": [
    "### Define team model\n",
    "The team strength lookup has three components: an input, an embedding layer, and a flatten layer that creates the output.\n",
    "\n",
    "If you wrap these three layers in a model with an input and output, you can re-use that stack of three layers at multiple places.\n",
    "\n",
    "Note again that the weights for all three layers will be shared everywhere we use them.\n",
    "\n",
    "- Create a 1D input layer for the team ID (which will be an integer). Be sure to set the correct input shape!\n",
    "- Pass this input to the team strength lookup layer you created previously.\n",
    "- Flatten the output of the team strength lookup.\n",
    "- Create a model that uses the 1D input as input and flattened team strength as output."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "from keras.layers import Input, Embedding, Flatten\n",
    "from keras.models import Model\n",
    "\n",
    "# Create an input layer for the team ID\n",
    "teamid_in = Input(shape=(1,))\n",
    "\n",
    "# Lookup the input in the team strength embedding layer\n",
    "strength_lookup = team_lookup(teamid_in)\n",
    "\n",
    "# Flatten the output\n",
    "strength_lookup_flat = Flatten()(strength_lookup)\n",
    "\n",
    "# Combine the operations into a single, re-usable model\n",
    "team_strength_model = Model(teamid_in, strength_lookup_flat, name='Team-Strength-Model')"
   ]
  },
  {
   "source": [
    "### Defining two inputs\n",
    "In this exercise, you will define two input layers for the two teams in your model. This allows you to specify later in the model how the data from each team will be used differently.\n",
    "\n",
    "- Create an input layer to use for team 1. Recall that our input dimension is 1.\n",
    "- Name the input \"Team-1-In\" so you can later distinguish it from team 2.\n",
    "- Create an input layer to use for team 2, named \"Team-2-In\"."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the input layer from keras.layers\n",
    "from keras.layers import Input\n",
    "\n",
    "# Input layer for team 1\n",
    "team_in_1 = Input(shape=(1,), name='Team-1-In')\n",
    "\n",
    "# Separate input layer for team 2\n",
    "team_in_2 = Input(shape=(1,), name='Team-2-In')"
   ]
  },
  {
   "source": [
    "### Lookup both inputs in the same model\n",
    "Now that you have a team strength model and an input layer for each team, you can lookup the team inputs in the shared team strength model. The two inputs will share the same weights.\n",
    "\n",
    "In this dataset, you have 10,888 unique teams. You want to learn a strength rating for each team, such that if any pair of teams plays each other, you can predict the score, even if those two teams have never played before. Furthermore, you want the strength rating to be the same, regardless of whether the team is the home team or the away team.\n",
    "\n",
    "To achieve this, you use a shared layer, defined by the re-usable model (team_strength_model()) you built in exercise 3 and the two input layers (team_in_1 and team_in_2) from the previous exercise, all of which are available in your workspace."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lookup team 1 in the team strength model\n",
    "team_1_strength = team_strength_model(team_in_1)\n",
    "\n",
    "# Lookup team 2 in the team strength model\n",
    "team_2_strength = team_strength_model(team_in_2)"
   ]
  },
  {
   "source": [
    "### Output layer using shared layer\n",
    "Now that you've looked up how \"strong\" each team is, subtract the team strengths to determine which team is expected to win the game.\n",
    "\n",
    "This is a bit like the seeds that the tournament committee uses, which are also a measure of team strength. But rather than using seed differences to predict score differences, you'll use the difference of your own team strength model to predict score differences.\n",
    "\n",
    "The subtract layer will combine the weights from the two layers by subtracting them."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Subtract layer from keras\n",
    "from keras.layers import Subtract\n",
    "\n",
    "# Create a subtract layer using the inputs from the previous exercise\n",
    "score_diff = Subtract()([team_1_strength, team_2_strength])"
   ]
  },
  {
   "source": [
    "### Model using two inputs and one output\n",
    "Now that you have your two inputs (team id 1 and team id 2) and output (score difference), you can wrap them up in a model so you can use it later for fitting to data and evaluating on new data."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "from keras.layers import Subtract\n",
    "from keras.models import Model\n",
    "\n",
    "# Subtraction layer from previous exercise\n",
    "score_diff = Subtract()([team_1_strength, team_2_strength])\n",
    "\n",
    "# Create the model\n",
    "model = Model([team_in_1, team_in_2], score_diff)\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_absolute_error')"
   ]
  },
  {
   "source": [
    "### Fit the model to the regular season training data\n",
    "Now that you've defined a complete team strength model, you can fit it to the basketball data! Since your model has two inputs now, you need to pass the input data as a list."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the team_1 column from the regular season data\n",
    "input_1 = games_season['team_1']\n",
    "\n",
    "# Get the team_2 column from the regular season data\n",
    "input_2 = games_season['team_2']\n",
    "\n",
    "# Fit the model to input 1 and 2, using score diff as a target\n",
    "model.fit([input_1,input_2],\n",
    "          games_season['score_diff'],\n",
    "          epochs=1,\n",
    "          batch_size=2048,\n",
    "          validation_split=0.1,\n",
    "          verbose=True)"
   ]
  },
  {
   "source": [
    "### Evaluate the model on the tournament test data\n",
    "The model you fit to the regular season data (model) in the previous exercise and the tournament dataset (games_tourney) are available in your workspace.\n",
    "\n",
    "In this exercise, you will evaluate the model on this new dataset. This evaluation will tell you how well you can predict the tournament games, based on a model trained with the regular season data. This is interesting because many teams play each other in the tournament that did not play in the regular season, so this is a very good check that your model is not overfitting."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get team_1 from the tournament data\n",
    "input_1 = games_tourney['team_1']\n",
    "\n",
    "# Get team_2 from the tournament data\n",
    "input_2 = games_tourney['team_2']\n",
    "\n",
    "# Evaluate the model using these inputs\n",
    "print(model.evaluate([input_1, input_2], games_tourney['score_diff'], verbose=False))"
   ]
  },
  {
   "source": [
    "### Make an input layer for home vs. away\n",
    "Now you will make an improvement to the model you used in the previous chapter for regular season games. You know there is a well-documented home-team advantage in basketball, so you will add a new input to your model to capture this effect.\n",
    "\n",
    "This model will have three inputs: team_id_1, team_id_2, and home. The team IDs will be integers that you look up in your team strength model from the previous chapter, and home will be a binary variable, 1 if team_1 is playing at home, 0 if they are not.\n",
    "\n",
    "The team_strength_model you used in the previous chapter has been loaded into your workspace. After applying it to each input, use a Concatenate layer to join the two team strengths and with the home vs away variable, and pass the result to a Dense layer."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an Input for each team\n",
    "team_in_1 = Input(shape=(1,), name='Team-1-In')\n",
    "team_in_2 = Input(shape=(1,), name='Team-2-In')\n",
    "\n",
    "# Create an input for home vs away\n",
    "home_in = Input(shape=(1,), name='Home-In')\n",
    "\n",
    "# Lookup the team inputs in the team strength model\n",
    "team_1_strength = team_strength_model(team_in_1)\n",
    "team_2_strength = team_strength_model(team_in_2)\n",
    "\n",
    "# Combine the team strengths with the home input using a Concatenate layer, then add a Dense layer\n",
    "out = Concatenate()([team_1_strength, team_2_strength, home_in])\n",
    "out = Dense(1)(out)"
   ]
  },
  {
   "source": [
    "### Make a model and compile it\n",
    "Now that you've input and output layers for the 3-input model, wrap them up in a Keras model class, and then compile the model, so you can fit it to data and use it to make predictions on new data."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the model class\n",
    "from keras.models import Model\n",
    "\n",
    "# Make a Model\n",
    "model = Model([team_in_1, team_in_2, home_in], out)\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_absolute_error')"
   ]
  },
  {
   "source": [
    "### Fit the model and evaluate\n",
    "Now that you've defined a new model, fit it to the regular season basketball data.\n",
    "\n",
    "Use the model you fit in the previous exercise (which was trained on the regular season data) and evaluate the model on data for tournament games (games_tourney)."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model to the games_season dataset\n",
    "model.fit([games_season['team_1'], games_season['team_2'], games_season['home']],\n",
    "          games_season['score_diff'],\n",
    "          epochs=1,\n",
    "          verbose=True,\n",
    "          validation_split=.10,\n",
    "          batch_size=2048)\n",
    "\n",
    "# Evaluate the model on the games_tourney dataset\n",
    "print(model.evaluate([games_tourney['team_1'], games_tourney['team_2'], games_tourney['home']],\n",
    "               games_tourney['score_diff'], verbose=False))"
   ]
  },
  {
   "source": [
    "### Plotting models\n",
    "In addition to summarizing your model, you can also plot your model to get a more intuitive sense of it. Your model is available in the workspace."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import matplotlib.pyplot as plt\n",
    "from keras.utils import plot_model\n",
    "\n",
    "# Plot the model\n",
    "plot_model(model, to_file='model.png')\n",
    "\n",
    "# Display the image\n",
    "data = plt.imread('model.png')\n",
    "plt.imshow(data)\n",
    "plt.show()"
   ]
  },
  {
   "source": [
    "### Add the model predictions to the tournament data\n",
    "In lesson 1 of this chapter, you used the regular season model to make predictions on the tournament dataset, and got pretty good results! Try to improve your predictions for the tournament by modeling it specifically.\n",
    "\n",
    "You'll use the prediction from the regular season model as an input to the tournament model. This is a form of \"model stacking.\"\n",
    "\n",
    "To start, take the regular season model from the previous lesson, and predict on the tournament data. Add this prediction to the tournament data as a new column."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict\n",
    "games_tourney['pred'] = model.predict([games_tourney['team_1'],\n",
    "                                             games_tourney['team_2'],\n",
    "                                             games_tourney['home']])"
   ]
  },
  {
   "source": [
    "### Create an input layer with multiple columns\n",
    "In this exercise, you will look at a different way to create models with multiple inputs. This method only works for purely numeric data, but its a much simpler approach to making multi-variate neural networks.\n",
    "\n",
    "Now you have three numeric columns in the tournament dataset: 'seed_diff', 'home', and 'pred'. In this exercise, you will create a neural network that uses a single input layer to process all three of these numeric inputs.\n",
    "\n",
    "This model should have a single output to predict the tournament game score difference."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an input layer with 3 columns\n",
    "input_tensor = Input((3,))\n",
    "\n",
    "# Pass it to a Dense layer with 1 unit\n",
    "output_tensor = Dense(1)(input_tensor)\n",
    "\n",
    "# Create a model\n",
    "model = Model(input_tensor, output_tensor)\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_absolute_error')"
   ]
  },
  {
   "source": [
    "### Fit the model\n",
    "Now that you've enriched the tournament dataset and built a model to make use of the new data, fit that model to the tournament data.\n",
    "\n",
    "Note that this model has only one input layer that is capable of handling all 3 inputs, so it's inputs and outputs do not need to be a list.\n",
    "\n",
    "Tournament games are split into a training set and a test set. The tournament games before 2010 are in the training set, and the ones after 2010 are in the test set."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "model.fit(games_tourney_train[['home', 'seed_diff', 'pred']],\n",
    "          games_tourney_train['score_diff'],\n",
    "          epochs=1,\n",
    "          verbose=True)"
   ]
  },
  {
   "source": [
    "### Evaluate the model\n",
    "Now that you've fit your model to the tournament training data, evaluate it on the tournament test data. Recall that the tournament test data contains games from after 2010."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the model on the games_tourney_test dataset\n",
    "print(model.evaluate(games_tourney_test[['home', 'seed_diff', 'prediction']],\n",
    "               games_tourney_test['score_diff'], verbose=False))"
   ]
  },
  {
   "source": [
    "### Simple two-output model\n",
    "In this exercise, you will use the tournament data to build one model that makes two predictions: the scores of both teams in a given game. Your inputs will be the seed difference of the two teams, as well as the predicted score difference from the model you built in chapter 3.\n",
    "\n",
    "The output from your model will be the predicted score for team 1 as well as team 2. This is called \"multiple target regression\": one model making more than one prediction."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the input\n",
    "input_tensor = Input((2,))\n",
    "\n",
    "# Define the output\n",
    "output_tensor = Dense(2)(input_tensor)\n",
    "\n",
    "# Create a model\n",
    "model = Model(input_tensor, output_tensor)\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_absolute_error')"
   ]
  },
  {
   "source": [
    "### Fit a model with two outputs\n",
    "Now that you've defined your 2-output model, fit it to the tournament data. I've split the data into games_tourney_train and games_tourney_test, so use the training set to fit for now.\n",
    "\n",
    "This model will use the pre-tournament seeds, as well as your pre-tournament predictions from the regular season model you built previously in this course.\n",
    "\n",
    "As a reminder, this model will predict the scores of both teams."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "model.fit(games_tourney_train[['seed_diff', 'pred']],\n",
    "  \t\t  games_tourney_train[['score_1', 'score_2']],\n",
    "  \t\t  verbose=True,\n",
    "  \t\t  epochs=100,\n",
    "  \t\t  batch_size=16384)"
   ]
  },
  {
   "source": [
    "### Inspect the model (I)\n",
    "Now that you've fit your model, let's take a look at it. You can use the .get_weights() method to inspect your model's weights.\n",
    "\n",
    "The input layer will have 4 weights: 2 for each input times 2 for each output.\n",
    "\n",
    "The output layer will have 2 weights, one for each output."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the model's weights\n",
    "print(model.get_weights())\n",
    "\n",
    "# Print the column means of the training data\n",
    "print(games_tourney_train.mean())"
   ]
  },
  {
   "source": [
    "### Exercise\n",
    "Exercise\n",
    "Evaluate the model\n",
    "Now that you've fit your model and inspected it's weights to make sure it makes sense, evaluate it on the tournament test set to see how well it performs on new data."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the model on the tournament test data\n",
    "print(model.evaluate(games_tourney_test[['seed_diff', 'pred']], games_tourney_test[['score_1', 'score_2']], verbose=False))"
   ]
  },
  {
   "source": [
    "### Classification and regression in one model\n",
    "Now you will create a different kind of 2-output model. This time, you will predict the score difference, instead of both team's scores and then you will predict the probability that team 1 won the game. This is a pretty cool model: it is going to do both classification and regression!\n",
    "\n",
    "In this model, turn off the bias, or intercept for each layer. Your inputs (seed difference and predicted score difference) have a mean of very close to zero, and your outputs both have means that are close to zero, so your model shouldn't need the bias term to fit the data well."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an input layer with 2 columns\n",
    "input_tensor = Input((2,))\n",
    "\n",
    "# Create the first output\n",
    "output_tensor_1 = Dense(1, activation='linear', use_bias=False)(input_tensor)\n",
    "\n",
    "# Create the second output (use the first output as input here)\n",
    "output_tensor_2 = Dense(1, activation='sigmoid', use_bias=False)(output_tensor_1)\n",
    "\n",
    "# Create a model with 2 outputs\n",
    "model = Model(input_tensor, [output_tensor_1, output_tensor_2])"
   ]
  },
  {
   "source": [
    "### Compile and fit the model\n",
    "Now that you have a model with 2 outputs, compile it with 2 loss functions: mean absolute error (MAE) for 'score_diff' and binary cross-entropy (also known as logloss) for 'won'. Then fit the model with 'seed_diff' and 'pred' as inputs. For outputs, predict 'score_diff' and 'won'.\n",
    "\n",
    "This model can use the scores of the games to make sure that close games (small score diff) have lower win probabilities than blowouts (large score diff).\n",
    "\n",
    "The regression problem is easier than the classification problem because MAE punishes the model less for a loss due to random chance. For example, if score_diff is -1 and won is 0, that means team_1 had some bad luck and lost by a single free throw. The data for the easy problem helps the model find a solution to the hard problem."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Adam optimizer\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "# Compile the model with 2 losses and the Adam optimzer with a higher learning rate\n",
    "model.compile(loss=['mean_absolute_error', 'binary_crossentropy'], optimizer=Adam(0.01))\n",
    "\n",
    "# Fit the model to the tournament training data, with 2 inputs and 2 outputs\n",
    "model.fit(games_tourney_train[['seed_diff', 'pred']],\n",
    "          [games_tourney_train[['score_diff']], games_tourney_train[['won']]],\n",
    "          epochs=10,\n",
    "          verbose=True,\n",
    "          batch_size=16384)"
   ]
  },
  {
   "source": [
    "### Inspect the model (II)\n",
    "Now you should take a look at the weights for this model. In particular, note the last weight of the model. This weight converts the predicted score difference to a predicted win probability. If you multiply the predicted score difference by the last weight of the model and then apply the sigmoid function, you get the win probability of the game."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the model weights\n",
    "print(model.get_weights())\n",
    "\n",
    "# Print the training data means\n",
    "print(games_tourney_train.mean())\n",
    "\n",
    "# Import the sigmoid function from scipy\n",
    "from scipy.special import expit as sigmoid\n",
    "\n",
    "# Weight from the model\n",
    "weight = 0.14\n",
    "\n",
    "# Print the approximate win probability of a predicted close game\n",
    "print(sigmoid(1 * weight))\n",
    "\n",
    "# Print the approximate win probability of a predicted blowout game\n",
    "print(sigmoid(10 * weight))"
   ]
  },
  {
   "source": [
    "### Evaluate on new data with two metrics\n",
    "Now that you've fit your model and inspected its weights to make sure they make sense, evaluate your model on the tournament test set to see how well it does on new data.\n",
    "\n",
    "Note that in this case, Keras will return 3 numbers: the first number will be the sum of both the loss functions, and then the next 2 numbers will be the loss functions you used when defining the model."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the model on new data\n",
    "print(model.evaluate(games_tourney_test[['seed_diff', 'pred']], [games_tourney_test[['score_diff']], games_tourney_test[['won']]], verbose=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}