BookRecommender.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notebook: BookRecommender-Submitted\n",
    "# Author: George Tohme\n",
    "# Date: Sep 27, 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Approach\n",
    "This notebook outlines my attempts at tackling the book recommendation problem. \n",
    "\n",
    "At a high level, since we don't have actual ratings of books by users but rather the interaction with the books, I opted to treat the 6 types of interactions as scaled up rating imitating regulr 1-5 star ratings with an interaction of 'dislike getting rated as a '0', all the way to an interaction of 'checkout' being rated as as a '5'.\n",
    "\n",
    "Anther potential approach would be to treat this problem as a classification problem. I have not attempted to do so.\n",
    "\n",
    "As you can see below, I have taken a couple of paths to the solution:\n",
    "1. Considering the problem production-related constraints, I attemted using a simple **Matrix Factorization** method using standard libraries and functions namely **scipy.sparse.linalg's SVDS**. This a simple collaborative filtering exercise where we know some of the ratings (in our case, pseudo-ratings) and we're trying to uncover latent features. The big question here is determining the 'k' factor. As noted in the code below, I have not attempted to optimize the algorithm to find an optimal 'k'.\n",
    "\n",
    "2. Having worked on a similar problems using **Neural Net** (using the **Keras** library with **Tensorflow** as the backend) with pretty reasonable performance, I attempted to apply that approach to this problem by providing multiple models including a simple NNMF.\n",
    "\n",
    "  \n",
    "With the exception of one NN, I have not used any explicite data features. One NN model, I complemented the latent features with the inclusion of the books' **Author and YearOfPublication**.\n",
    "\n",
    "In a real-life scenario, we're always be faced with the problem of providing recommendations to new users to avoid the 'cold start problem'. This could be mitigated using a number of methods from beefing up the user profile with preferences to finding recommendations based on similar users, best sellers, new books, etc...\n",
    "\n",
    "#### Results\n",
    "The results were a mixed bag. \n",
    "\n",
    "Why? Generally, one should start with looking at the data, the approach and then the algorithms/models.\n",
    "\n",
    "In the case of the NN models, overfitting seemed to be a problem across the different model architecture.\n",
    "\n",
    "I think the data (and its sparcity) played a huge role. Taking the same models and applying them to similiar-ish data such as the MovieLens dataset would yield a more consistent results. So that could mean that the data is odd is some to-be-determined way or the approach (treating the interactions as ratings) is not a good one.\n",
    "\n",
    "It also could be that the model parameters need further tuning or simply that there is something wrong with code. In both case, more time for further investigation could yield better results.\n",
    "\n",
    "The best result (although the numbers look very suspicious) was obtained using Model 0 (SVD) with a resulting predictions table shape (17252, 21128) which will create a massibe multi-GB file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General notes\n",
    "As noted above, the results are a mixed bag which could attributed to one or many reasons:\n",
    "- limited time for testing and finetuning\n",
    "- available processing power \n",
    "- data quality & consistency\n",
    "\n",
    "Throughout, I attempted to simulate a clean-data environment. To that end, I limited my dataset to a normalized subset (from the events.csv) satisfying the following 2 conditions:\n",
    "- the books must exist in the books.csv\n",
    "- the users must exist in the users.csv\n",
    "\n",
    "I could blame lack of time for a number of things that could and should have been better with the code below, but in all honesty i only had about a day to work on this. This is not the most elegant code: inefficient use of memeory (lots of pd dataframes), df joinging/merging is a bit convoluted. However, I thouroughly enjoyed it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from scipy.sparse.linalg import svds\n",
    "from scipy import linalg\n",
    "\n",
    "from math import sqrt\n",
    "\n",
    "from IPython.display import SVG, display, HTML\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "from keras.utils import plot_model\n",
    "\n",
    "import pydot\n",
    "\n",
    "import keras\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Embedding\n",
    "from keras.layers.core import Flatten, Dense, Dropout\n",
    "from keras.layers.merge import dot, add, concatenate\n",
    "\n",
    "from keras.regularizers import l2\n",
    "from keras.optimizers import Adam, SGD\n",
    "from keras.constraints import non_neg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "\n",
    "SVD_k = 150\n",
    "MAX_RECOMMENDATIONS = 10\n",
    "\n",
    "MM_LATENT_FACTORS = 5\n",
    "MM_BATCH_SIZE = 64\n",
    "MM_TRAIN_EPOCHS = 50\n",
    "\n",
    "TRAIN_VERBOSE = 1\n",
    "TRAIN_SPLIT_RATIO = 0.2\n",
    "\n",
    "NN_USER_LATENT_FACTORS = 5\n",
    "NN_MOVIE_LATENT_FACTORS = 5\n",
    "NN_TRAIN_EPOCHS = 50\n",
    "NN_BATCH_SIZE = 64\n",
    "\n",
    "# Non-negative Matrix factorisation\n",
    "NNMF = True\n",
    "\n",
    "SGD_OPTIMIZER = False\n",
    "\n",
    "# Adam Parameters\n",
    "Adam_lr=0.02\n",
    "Adam_beta_1=0.9\n",
    "Adam_beta_2=0.999\n",
    "Adam_epsilon=None\n",
    "Adam_decay=0.0\n",
    "Adam_amsgrad=False\n",
    "\n",
    "#SGD Parameters\n",
    "SGD_lr=0.02\n",
    "SGD_momentum=0.0\n",
    "SGD_decay=0.0\n",
    "SGD_nesterov=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Translate the events action to ratings. \n",
    "# That's the approach taken in this Notebook.\n",
    "\n",
    "# Another idea is to one-hot encode the actions and then \n",
    "# treat the problem as a classification problem. \n",
    "# Have not tried this to see if it yields better predictions.\n",
    "\n",
    "impressionRatings = {\n",
    "  \"dislike\" : 0,\n",
    "  \"view\" : 1,\n",
    "  \"interact\" : 2,\n",
    "  \"like\" : 3,\n",
    "  \"add to cart\" : 4,\n",
    "  \"checkout\" : 5\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A function to display tables & plots side-by-side\n",
    "\n",
    "from IPython.display import display_html\n",
    "def displaySideBySide(*args):\n",
    "    htmlStr=''\n",
    "    for df in args:\n",
    "        htmlStr += df.to_html()\n",
    "    display_html(htmlStr.replace('table','table style=\"display:inline\"'), raw=True)\n",
    "#enddef"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read and show the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "usersDataFrame = pd.read_csv('../Data/Users.csv', encoding = 'latin-1')\n",
    "booksDataFrame = pd.read_csv('../Data/Books.csv', encoding = 'latin-1')\n",
    "eventsDataFrame = pd.read_csv('../Data/UserEvents.csv', encoding = 'latin-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 149998, 400000)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preliminary dimensions\n",
    "len(usersDataFrame.index), len(booksDataFrame.index), len(eventsDataFrame.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>user</th>\n",
       "      <th>location</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>177121</td>\n",
       "      <td>177122.0</td>\n",
       "      <td>winter park, florida, usa</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>174021</td>\n",
       "      <td>174022.0</td>\n",
       "      <td>fairview park, ohio, usa</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>179441</td>\n",
       "      <td>179442.0</td>\n",
       "      <td>farmville, north carolina, usa</td>\n",
       "      <td>45.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44391</td>\n",
       "      <td>44392.0</td>\n",
       "      <td>weston, florida, usa</td>\n",
       "      <td>65.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>89250</td>\n",
       "      <td>89251.0</td>\n",
       "      <td>duesseldorf, n/a, germany</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0      user                        location   age\n",
       "0      177121  177122.0       winter park, florida, usa  50.0\n",
       "1      174021  174022.0        fairview park, ohio, usa   NaN\n",
       "2      179441  179442.0  farmville, north carolina, usa  45.0\n",
       "3       44391   44392.0            weston, florida, usa  65.0\n",
       "4       89250   89251.0       duesseldorf, n/a, germany   NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usersDataFrame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>bookISBN</th>\n",
       "      <th>bookName</th>\n",
       "      <th>author</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "      <th>urlId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>58005</td>\n",
       "      <td>773730982</td>\n",
       "      <td>Stretch, Swallow &amp;amp; Stare</td>\n",
       "      <td>Veronika Martenova Charles</td>\n",
       "      <td>1999</td>\n",
       "      <td>Stoddart Kids</td>\n",
       "      <td>8040.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>158211</td>\n",
       "      <td>916620867</td>\n",
       "      <td>The Two Faces of Religion: A Psychiatrists View</td>\n",
       "      <td>N.S. Xavier</td>\n",
       "      <td>1987</td>\n",
       "      <td>Portals Pr</td>\n",
       "      <td>46584.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>127812</td>\n",
       "      <td>380978598</td>\n",
       "      <td>Roma Eterna</td>\n",
       "      <td>Robert Silverberg</td>\n",
       "      <td>2003</td>\n",
       "      <td>Eos</td>\n",
       "      <td>39681.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>47598</td>\n",
       "      <td>802727719</td>\n",
       "      <td>For Everything a Season: Simple Musings on Liv...</td>\n",
       "      <td>Philip Gulley</td>\n",
       "      <td>2001</td>\n",
       "      <td>Walker Large Print</td>\n",
       "      <td>29478.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>131723</td>\n",
       "      <td>446604178</td>\n",
       "      <td>Mountain, Get Out of My Way: Life Lessons and ...</td>\n",
       "      <td>Montel Williams</td>\n",
       "      <td>1997</td>\n",
       "      <td>Warner Books (Mm)</td>\n",
       "      <td>104573.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0   bookISBN                                           bookName  \\\n",
       "0       58005  773730982                       Stretch, Swallow &amp; Stare   \n",
       "1      158211  916620867    The Two Faces of Religion: A Psychiatrists View   \n",
       "2      127812  380978598                                        Roma Eterna   \n",
       "3       47598  802727719  For Everything a Season: Simple Musings on Liv...   \n",
       "4      131723  446604178  Mountain, Get Out of My Way: Life Lessons and ...   \n",
       "\n",
       "                       author  yearOfPublication           publisher     urlId  \n",
       "0  Veronika Martenova Charles               1999       Stoddart Kids    8040.0  \n",
       "1                 N.S. Xavier               1987          Portals Pr   46584.0  \n",
       "2           Robert Silverberg               2003                 Eos   39681.0  \n",
       "3               Philip Gulley               2001  Walker Large Print   29478.0  \n",
       "4             Montel Williams               1997   Warner Books (Mm)  104573.0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "booksDataFrame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>user</th>\n",
       "      <th>bookId</th>\n",
       "      <th>impression</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>523113</td>\n",
       "      <td>126736</td>\n",
       "      <td>0843946806</td>\n",
       "      <td>dislike</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>861298</td>\n",
       "      <td>208406</td>\n",
       "      <td>0345353145</td>\n",
       "      <td>like</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37104</td>\n",
       "      <td>8890</td>\n",
       "      <td>2020213508</td>\n",
       "      <td>add to cart</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>328497</td>\n",
       "      <td>78553</td>\n",
       "      <td>0451402383</td>\n",
       "      <td>add to cart</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>121368</td>\n",
       "      <td>27875</td>\n",
       "      <td>0307129659</td>\n",
       "      <td>add to cart</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0    user      bookId   impression\n",
       "0      523113  126736  0843946806      dislike\n",
       "1      861298  208406  0345353145         like\n",
       "2       37104    8890  2020213508  add to cart\n",
       "3      328497   78553  0451402383  add to cart\n",
       "4      121368   27875  0307129659  add to cart"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eventsDataFrame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400000, 4)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eventsDataFrame.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preliminary cleanup..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the user column for consistency\n",
    "usersDataFrame.rename(columns={\"user\": \"userId\"}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the bookId column for consistency\n",
    "booksDataFrame.rename(columns={\"bookId\": \"bookISBN\"}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some housekeeping column names updates for consistency and clarity...\n",
    "eventsDataFrame.rename(columns={\"user\": \"userId\"}, inplace = True)\n",
    "eventsDataFrame.rename(columns={\"bookId\": \"bookISBN\"}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the impression to a rating (think sales funnel ratings)\n",
    "eventsDataFrame[\"impressionRating\"] = eventsDataFrame[\"impression\"].map(impressionRatings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the old text impression columns and the first column (event row id)\n",
    "eventsDataFrame.drop(eventsDataFrame.columns[0], axis=1, inplace=True)\n",
    "eventsDataFrame.drop(eventsDataFrame.columns[2], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Merge the dataframes and remove the nulls..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join the events DF to the books DF using the books' ISBN.\n",
    "# We're doing this, to ensure we're counting the books and users \n",
    "# that exist in the other tables (files)\n",
    "\n",
    "# Building the unique indices based on the full events file will \n",
    "# yield incorrect data and will cause the models to crash as some ids will \n",
    "# larger than what max count is.\n",
    "\n",
    "filledEventsDataFrame = pd.merge(eventsDataFrame,\n",
    "             booksDataFrame[['author', 'yearOfPublication', 'bookISBN']], \n",
    "             on='bookISBN', \n",
    "             how='inner')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove all nulls...\n",
    "filledEventsDataFrame = filledEventsDataFrame.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create unique ids for come columns..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Build an index for the books' authors' names\n",
    "authors = filledEventsDataFrame['author'].unique()\n",
    "authorsDict = {name: authorId for authorId, name in enumerate(authors)}\n",
    "\n",
    "# add it to the books dataframe\n",
    "booksDataFrame[\"authorIDX\"] = booksDataFrame[\"author\"].map(authorsDict)\n",
    "# and to ....\n",
    "filledEventsDataFrame[\"authorIDX\"] = filledEventsDataFrame[\"author\"].map(authorsDict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a book id based on the users in the events dataset\n",
    "bookIds = filledEventsDataFrame[\"bookISBN\"].unique()\n",
    "bookIdsDict = {bookId: bookIDX for bookIDX, bookId in enumerate(bookIds)}\n",
    "\n",
    "# add it to the books dataframe\n",
    "booksDataFrame[\"bookIDX\"] = booksDataFrame[\"bookISBN\"].map(bookIdsDict)\n",
    "\n",
    "# and to ...\n",
    "filledEventsDataFrame[\"bookIDX\"] = filledEventsDataFrame[\"bookISBN\"].map(bookIdsDict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a unique user id based on the users in the events dataset\n",
    "userIds = filledEventsDataFrame[\"userId\"].unique()\n",
    "userIdsDict = {userId: userIDX for userIDX, userId in enumerate(userIds)}\n",
    "\n",
    "# Add it to the users dataframe\n",
    "usersDataFrame[\"userIDX\"] = usersDataFrame[\"userId\"].map(userIdsDict)\n",
    "\n",
    "# and to ...\n",
    "filledEventsDataFrame[\"userIDX\"] = filledEventsDataFrame[\"userId\"].map(userIdsDict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>bookISBN</th>\n",
       "      <th>impressionRating</th>\n",
       "      <th>author</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>authorIDX</th>\n",
       "      <th>bookIDX</th>\n",
       "      <th>userIDX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>850</td>\n",
       "      <td>3426616262</td>\n",
       "      <td>2</td>\n",
       "      <td>Alexander Solschenizyn</td>\n",
       "      <td>1999</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>177458</td>\n",
       "      <td>1565112318</td>\n",
       "      <td>2</td>\n",
       "      <td>Neil Gaiman</td>\n",
       "      <td>1997</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>151420</td>\n",
       "      <td>8445071416</td>\n",
       "      <td>4</td>\n",
       "      <td>J. R. R. Tolkien</td>\n",
       "      <td>1991</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>229501</td>\n",
       "      <td>8445071416</td>\n",
       "      <td>3</td>\n",
       "      <td>J. R. R. Tolkien</td>\n",
       "      <td>1991</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>93755</td>\n",
       "      <td>8445071416</td>\n",
       "      <td>4</td>\n",
       "      <td>J. R. R. Tolkien</td>\n",
       "      <td>1991</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId    bookISBN  impressionRating                  author  \\\n",
       "0     850  3426616262                 2  Alexander Solschenizyn   \n",
       "1  177458  1565112318                 2             Neil Gaiman   \n",
       "2  151420  8445071416                 4        J. R. R. Tolkien   \n",
       "3  229501  8445071416                 3        J. R. R. Tolkien   \n",
       "4   93755  8445071416                 4        J. R. R. Tolkien   \n",
       "\n",
       "   yearOfPublication  authorIDX  bookIDX  userIDX  \n",
       "0               1999          0        0        0  \n",
       "1               1997          1        1        1  \n",
       "2               1991          2        2        2  \n",
       "3               1991          2        2        3  \n",
       "4               1991          2        2        4  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filledEventsDataFrame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41048, 8)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filledEventsDataFrame.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ensure there are no nulls in the dataframe we will be using and save it (just in case) for offline inspection..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>bookISBN</th>\n",
       "      <th>impressionRating</th>\n",
       "      <th>author</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>authorIDX</th>\n",
       "      <th>bookIDX</th>\n",
       "      <th>userIDX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [userId, bookISBN, impressionRating, author, yearOfPublication, authorIDX, bookIDX, userIDX]\n",
       "Index: []"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filledEventsDataFrame[filledEventsDataFrame.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "filledEventsDataFrame.to_csv(\"filledEventsDataFrame.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split the data into training and test sets..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Do the split...\n",
    "trainDataFrame, testDataFrame = train_test_split(filledEventsDataFrame, test_size=TRAIN_SPLIT_RATIO)\n",
    "\n",
    "# Grab the actual impression to check the results later\n",
    "actualImpressionRatings = testDataFrame[\"impressionRating\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(41048, 8)\n",
      "Number of users: 17252\n",
      "Number of books: 21128\n",
      "Matrix sparcity: 99.9887%\n"
     ]
    }
   ],
   "source": [
    "# dig into the data...\n",
    "numOfUsers = len(filledEventsDataFrame['userIDX'].unique())\n",
    "numOfBooks = len(filledEventsDataFrame['bookIDX'].unique())\n",
    "dataSparcity = 100.0 - (100.0 * len(filledEventsDataFrame) / (numOfUsers * numOfBooks))\n",
    "\n",
    "print(filledEventsDataFrame.shape)\n",
    "print(\"Number of users: \" + str(numOfUsers))\n",
    "print(\"Number of books: \" + str(numOfBooks))\n",
    "print(\"Matrix sparcity: {0:2.4f}%\".format(dataSparcity))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show a crosstab of the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>bookIDX</th>\n",
       "      <th>30</th>\n",
       "      <th>34</th>\n",
       "      <th>41</th>\n",
       "      <th>46</th>\n",
       "      <th>60</th>\n",
       "      <th>63</th>\n",
       "      <th>87</th>\n",
       "      <th>156</th>\n",
       "      <th>171</th>\n",
       "      <th>206</th>\n",
       "      <th>292</th>\n",
       "      <th>420</th>\n",
       "      <th>472</th>\n",
       "      <th>518</th>\n",
       "      <th>774</th>\n",
       "      <th>869</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userIDX</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>414</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>520</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1156</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1244</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1348</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3401</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3887</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "bookIDX  30   34   41   46   60   63   87   156  171  206  292  420  472  518  \\\n",
       "userIDX                                                                         \n",
       "21       3.0  NaN  0.0  4.0  5.0  1.0  5.0  NaN  NaN  5.0  NaN  3.0  3.0  NaN   \n",
       "107      4.0  NaN  NaN  NaN  5.0  NaN  NaN  5.0  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "260      4.0  NaN  NaN  NaN  NaN  NaN  NaN  4.0  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "414      NaN  1.0  NaN  2.0  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "520      NaN  NaN  1.0  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "938      NaN  NaN  NaN  NaN  5.0  5.0  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "1156     NaN  NaN  NaN  NaN  5.0  NaN  NaN  5.0  5.0  NaN  NaN  5.0  NaN  NaN   \n",
       "1244     NaN  NaN  NaN  NaN  NaN  2.0  NaN  NaN  NaN  NaN  5.0  NaN  NaN  1.0   \n",
       "1348     NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   \n",
       "3401     NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  3.0  NaN  NaN   \n",
       "3887     NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  2.0  2.0   \n",
       "\n",
       "bookIDX  774  869  \n",
       "userIDX            \n",
       "21       5.0  5.0  \n",
       "107      NaN  NaN  \n",
       "260      NaN  NaN  \n",
       "414      NaN  NaN  \n",
       "520      NaN  NaN  \n",
       "938      NaN  NaN  \n",
       "1156     4.0  5.0  \n",
       "1244     NaN  NaN  \n",
       "1348     NaN  4.0  \n",
       "3401     NaN  NaN  \n",
       "3887     NaN  NaN  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the crosstab of the top users/books events (ratings)\n",
    "\n",
    "g = filledEventsDataFrame.groupby('userIDX')['impressionRating'].count()\n",
    "topUsers = g.sort_values(ascending=False)[:20]\n",
    "\n",
    "g = filledEventsDataFrame.groupby('bookIDX')['impressionRating'].count()\n",
    "topBooks = g.sort_values(ascending=False)[:20]\n",
    "\n",
    "topRatings = filledEventsDataFrame.join(topUsers, how='inner', rsuffix='r_', on='userIDX')\n",
    "topRatings = topRatings.join(topBooks, how='inner', rsuffix='r_', on='bookIDX')\n",
    "\n",
    "pd.crosstab(topRatings.userIDX, topRatings.bookIDX, topRatings.impressionRating, aggfunc=np.sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show the heads..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>bookISBN</th>\n",
       "      <th>impressionRating</th>\n",
       "      <th>author</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>authorIDX</th>\n",
       "      <th>bookIDX</th>\n",
       "      <th>userIDX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4822</th>\n",
       "      <td>94865</td>\n",
       "      <td>3442309174</td>\n",
       "      <td>5</td>\n",
       "      <td>Natsuo Kirino</td>\n",
       "      <td>2003</td>\n",
       "      <td>441</td>\n",
       "      <td>467</td>\n",
       "      <td>3726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28284</th>\n",
       "      <td>265077</td>\n",
       "      <td>1555834337</td>\n",
       "      <td>2</td>\n",
       "      <td>Paula Martinac</td>\n",
       "      <td>1997</td>\n",
       "      <td>7206</td>\n",
       "      <td>9856</td>\n",
       "      <td>13625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24343</th>\n",
       "      <td>221901</td>\n",
       "      <td>1551666952</td>\n",
       "      <td>0</td>\n",
       "      <td>Karen Harper</td>\n",
       "      <td>2003</td>\n",
       "      <td>332</td>\n",
       "      <td>7414</td>\n",
       "      <td>12397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28212</th>\n",
       "      <td>126248</td>\n",
       "      <td>1894869494</td>\n",
       "      <td>4</td>\n",
       "      <td>Savannah Michaels</td>\n",
       "      <td>2002</td>\n",
       "      <td>7168</td>\n",
       "      <td>9809</td>\n",
       "      <td>13601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23199</th>\n",
       "      <td>197659</td>\n",
       "      <td>1903116155</td>\n",
       "      <td>4</td>\n",
       "      <td>Edward J. Cowan</td>\n",
       "      <td>2000</td>\n",
       "      <td>5105</td>\n",
       "      <td>6755</td>\n",
       "      <td>1924</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       userId    bookISBN  impressionRating             author  \\\n",
       "4822    94865  3442309174                 5      Natsuo Kirino   \n",
       "28284  265077  1555834337                 2     Paula Martinac   \n",
       "24343  221901  1551666952                 0       Karen Harper   \n",
       "28212  126248  1894869494                 4  Savannah Michaels   \n",
       "23199  197659  1903116155                 4    Edward J. Cowan   \n",
       "\n",
       "       yearOfPublication  authorIDX  bookIDX  userIDX  \n",
       "4822                2003        441      467     3726  \n",
       "28284               1997       7206     9856    13625  \n",
       "24343               2003        332     7414    12397  \n",
       "28212               2002       7168     9809    13601  \n",
       "23199               2000       5105     6755     1924  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDataFrame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32838, 8)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDataFrame.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>bookISBN</th>\n",
       "      <th>impressionRating</th>\n",
       "      <th>author</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>authorIDX</th>\n",
       "      <th>bookIDX</th>\n",
       "      <th>userIDX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>33904</th>\n",
       "      <td>38832</td>\n",
       "      <td>1890451053</td>\n",
       "      <td>5</td>\n",
       "      <td>A. Juno</td>\n",
       "      <td>2000</td>\n",
       "      <td>9793</td>\n",
       "      <td>13883</td>\n",
       "      <td>11390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7022</th>\n",
       "      <td>184465</td>\n",
       "      <td>1850651558</td>\n",
       "      <td>2</td>\n",
       "      <td>Stevan Pavlowitch</td>\n",
       "      <td>1993</td>\n",
       "      <td>748</td>\n",
       "      <td>821</td>\n",
       "      <td>5132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19385</th>\n",
       "      <td>255546</td>\n",
       "      <td>067086191X</td>\n",
       "      <td>3</td>\n",
       "      <td>Eileen Goudge</td>\n",
       "      <td>1996</td>\n",
       "      <td>3830</td>\n",
       "      <td>4876</td>\n",
       "      <td>1362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40365</th>\n",
       "      <td>28938</td>\n",
       "      <td>1586630997</td>\n",
       "      <td>5</td>\n",
       "      <td>John Haywood Ph.D.</td>\n",
       "      <td>2000</td>\n",
       "      <td>13167</td>\n",
       "      <td>19386</td>\n",
       "      <td>3977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8321</th>\n",
       "      <td>130049</td>\n",
       "      <td>1558744606</td>\n",
       "      <td>3</td>\n",
       "      <td>Jack Canfield</td>\n",
       "      <td>1997</td>\n",
       "      <td>176</td>\n",
       "      <td>1090</td>\n",
       "      <td>5825</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       userId    bookISBN  impressionRating              author  \\\n",
       "33904   38832  1890451053                 5             A. Juno   \n",
       "7022   184465  1850651558                 2   Stevan Pavlowitch   \n",
       "19385  255546  067086191X                 3       Eileen Goudge   \n",
       "40365   28938  1586630997                 5  John Haywood Ph.D.   \n",
       "8321   130049  1558744606                 3       Jack Canfield   \n",
       "\n",
       "       yearOfPublication  authorIDX  bookIDX  userIDX  \n",
       "33904               2000       9793    13883    11390  \n",
       "7022                1993        748      821     5132  \n",
       "19385               1996       3830     4876     1362  \n",
       "40365               2000      13167    19386     3977  \n",
       "8321                1997        176     1090     5825  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testDataFrame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8210, 8)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testDataFrame.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33904    5\n",
       "7022     2\n",
       "19385    3\n",
       "40365    5\n",
       "8321     3\n",
       "Name: impressionRating, dtype: int64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actualImpressionRatings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8210,)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actualImpressionRatings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>bookISBN</th>\n",
       "      <th>impressionRating</th>\n",
       "      <th>author</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>authorIDX</th>\n",
       "      <th>bookIDX</th>\n",
       "      <th>userIDX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>850</td>\n",
       "      <td>3426616262</td>\n",
       "      <td>2</td>\n",
       "      <td>Alexander Solschenizyn</td>\n",
       "      <td>1999</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>177458</td>\n",
       "      <td>1565112318</td>\n",
       "      <td>2</td>\n",
       "      <td>Neil Gaiman</td>\n",
       "      <td>1997</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>151420</td>\n",
       "      <td>8445071416</td>\n",
       "      <td>4</td>\n",
       "      <td>J. R. R. Tolkien</td>\n",
       "      <td>1991</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>229501</td>\n",
       "      <td>8445071416</td>\n",
       "      <td>3</td>\n",
       "      <td>J. R. R. Tolkien</td>\n",
       "      <td>1991</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>93755</td>\n",
       "      <td>8445071416</td>\n",
       "      <td>4</td>\n",
       "      <td>J. R. R. Tolkien</td>\n",
       "      <td>1991</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId    bookISBN  impressionRating                  author  \\\n",
       "0     850  3426616262                 2  Alexander Solschenizyn   \n",
       "1  177458  1565112318                 2             Neil Gaiman   \n",
       "2  151420  8445071416                 4        J. R. R. Tolkien   \n",
       "3  229501  8445071416                 3        J. R. R. Tolkien   \n",
       "4   93755  8445071416                 4        J. R. R. Tolkien   \n",
       "\n",
       "   yearOfPublication  authorIDX  bookIDX  userIDX  \n",
       "0               1999          0        0        0  \n",
       "1               1997          1        1        1  \n",
       "2               1991          2        2        2  \n",
       "3               1991          2        2        3  \n",
       "4               1991          2        2        4  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filledEventsDataFrame.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ---------Model 0: Matrix Multiplication - SVD Implementation (No Optimization)--------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the full ratings dataframe\n",
    "R_DataFrame = filledEventsDataFrame.pivot(index = 'userIDX', \n",
    "                                          columns ='bookIDX', \n",
    "                                          values = 'impressionRating').fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17252, 21128)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R_DataFrame.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Convert it to a matrix (array)\n",
    "R_Matrix = R_DataFrame.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "meanUserRatings = np.mean(R_Matrix, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "R_Demeaned = R_Matrix - meanUserRatings.reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Optimize  'k'\n",
    "\n",
    "# Ideally, we could optimize svds by creating a training and validation set to find the best 'k'\n",
    "# by automatically minimizing the resulting RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Assess adding a bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In the interest of time, this is not done here. Instead, I checked the outside temperature and\n",
    "# I picked k=150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Crunch and factorize by calling on scipy...\n",
    "U, sigma, Vt = svds(R_Demeaned, k=SVD_k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Convert sigma into a diag matrix to be used for predictions\n",
    "sigma = np.diag(sigma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recreate the ratings by multiplying the 3 matrices...\n",
    "allPredictedRatings = np.dot(np.dot(U, sigma), Vt) + meanUserRatings.reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17252, 21128)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Format into a dataframe and add column names\n",
    "allPredictedRatingsDataFrame = pd.DataFrame(allPredictedRatings, columns = R_DataFrame.columns)\n",
    "allPredictedRatingsDataFrame.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>bookIDX</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>21118</th>\n",
       "      <th>21119</th>\n",
       "      <th>21120</th>\n",
       "      <th>21121</th>\n",
       "      <th>21122</th>\n",
       "      <th>21123</th>\n",
       "      <th>21124</th>\n",
       "      <th>21125</th>\n",
       "      <th>21126</th>\n",
       "      <th>21127</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>-0.0030</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0012</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0012</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>-0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>1.7517</td>\n",
       "      <td>0.0207</td>\n",
       "      <td>0.0578</td>\n",
       "      <td>0.0024</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>-0.0548</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0014</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>-0.0092</td>\n",
       "      <td>-0.0107</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0035</td>\n",
       "      <td>-0.0229</td>\n",
       "      <td>-0.0020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>5.7406</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0108</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0028</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>-0.0003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>1.1758</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.0006</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>1.1626</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-0.0028</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>2.5230</td>\n",
       "      <td>-0.0031</td>\n",
       "      <td>-0.0022</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>-0.0072</td>\n",
       "      <td>-0.0030</td>\n",
       "      <td>-0.0028</td>\n",
       "      <td>-0.0080</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>-0.0025</td>\n",
       "      <td>-0.0028</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>-0.0025</td>\n",
       "      <td>-0.0040</td>\n",
       "      <td>-0.0027</td>\n",
       "      <td>-0.0022</td>\n",
       "      <td>-0.0035</td>\n",
       "      <td>0.0003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0031</td>\n",
       "      <td>2.8727</td>\n",
       "      <td>-0.0053</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0023</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0070</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0013</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>-0.0100</td>\n",
       "      <td>0.0006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.0007</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>1.4561</td>\n",
       "      <td>-0.0009</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0032</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0024</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.0006</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>1.1626</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-0.0015</td>\n",
       "      <td>0.0025</td>\n",
       "      <td>4.7807</td>\n",
       "      <td>-0.0014</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0147</td>\n",
       "      <td>-0.0016</td>\n",
       "      <td>-0.0014</td>\n",
       "      <td>-0.0060</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0033</td>\n",
       "      <td>-0.0031</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>3.2578</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0053</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0011</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0013</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>1.7836</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0051</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.5813</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.6294</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.0018</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>3.7579</td>\n",
       "      <td>0.0029</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>-0.0011</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>0.0033</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.0004</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.8719</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0020</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>-0.0004</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.8719</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0020</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.5895</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0009</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-0.0006</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>1.1626</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0026</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-0.0007</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>1.4532</td>\n",
       "      <td>-0.0009</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0033</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0024</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0009</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0382</td>\n",
       "      <td>-0.0109</td>\n",
       "      <td>0.1028</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0310</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0626</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0063</td>\n",
       "      <td>-0.0043</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0377</td>\n",
       "      <td>-0.0005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0026</td>\n",
       "      <td>0.0025</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>1.9947</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0068</td>\n",
       "      <td>-0.0018</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0045</td>\n",
       "      <td>0.0062</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>-0.0007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0021</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>-0.0002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0022</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0178</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>-0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0020</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0268</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0016</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0020</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0268</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0016</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0012</td>\n",
       "      <td>-0.0023</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0292</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0034</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0054</td>\n",
       "      <td>-0.0004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0264</td>\n",
       "      <td>-0.0068</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0021</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.1536</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>0.0023</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>-0.0086</td>\n",
       "      <td>0.0007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>-0.0046</td>\n",
       "      <td>0.0024</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0012</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0316</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0093</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0025</td>\n",
       "      <td>-0.0085</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0740</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.0017</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>-0.0003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0016</td>\n",
       "      <td>0.0014</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0140</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0093</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>-0.0016</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>-0.0420</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0886</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>-0.0073</td>\n",
       "      <td>-0.0033</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>0.0022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0024</td>\n",
       "      <td>-0.0014</td>\n",
       "      <td>-0.0009</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0188</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0199</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0030</td>\n",
       "      <td>-0.0016</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0007</td>\n",
       "      <td>-0.0067</td>\n",
       "      <td>-0.0019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0013</td>\n",
       "      <td>-0.0016</td>\n",
       "      <td>0.0124</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0101</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0123</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0006</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>-0.0002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0093</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0015</td>\n",
       "      <td>0.0013</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0139</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0016</td>\n",
       "      <td>0.0014</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0140</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>-0.0075</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0020</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0225</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>-0.0003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0018</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0142</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0019</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0186</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>-0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.0032</td>\n",
       "      <td>-0.0093</td>\n",
       "      <td>0.0715</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>-0.0035</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.2182</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>-0.0023</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0038</td>\n",
       "      <td>0.0006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0064</td>\n",
       "      <td>0.0016</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0022</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0088</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0138</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>0.0025</td>\n",
       "      <td>-0.0022</td>\n",
       "      <td>-0.0003</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0004</td>\n",
       "      <td>-0.0007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0005</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0046</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>-0.0000</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0093</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>-0.0010</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0159</td>\n",
       "      <td>-0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0151</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0039</td>\n",
       "      <td>0.0013</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>-0.0008</td>\n",
       "      <td>-0.0006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 21128 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "bookIDX   0       1       2       3       4       5       6       7      \\\n",
       "0        0.0011  0.0006 -0.0030  0.0010  0.0006  0.0011  0.0012  0.0011   \n",
       "1        0.0000  1.7517  0.0207  0.0578  0.0024  0.0003 -0.0548 -0.0006   \n",
       "2        0.0000  0.0001  5.7406  0.0008  0.0002  0.0002 -0.0108 -0.0001   \n",
       "3        0.0000 -0.0000  1.1758 -0.0004 -0.0007  0.0001 -0.0013  0.0001   \n",
       "4       -0.0006  0.0002  1.1626 -0.0007 -0.0003 -0.0005 -0.0026 -0.0006   \n",
       "5       -0.0028  0.0005  2.5230 -0.0031 -0.0022 -0.0026 -0.0072 -0.0030   \n",
       "6       -0.0004 -0.0031  2.8727 -0.0053 -0.0010 -0.0003  0.0023 -0.0007   \n",
       "7       -0.0007  0.0003  1.4561 -0.0009 -0.0003 -0.0006 -0.0032 -0.0007   \n",
       "8       -0.0006  0.0002  1.1626 -0.0007 -0.0003 -0.0005 -0.0026 -0.0006   \n",
       "9       -0.0015  0.0025  4.7807 -0.0014 -0.0004 -0.0012 -0.0147 -0.0016   \n",
       "10      -0.0001  0.0005  3.2578 -0.0000 -0.0002 -0.0000 -0.0053 -0.0002   \n",
       "11      -0.0003  0.0001  1.7836 -0.0002  0.0000 -0.0002 -0.0051 -0.0004   \n",
       "12      -0.0003  0.0001  0.5813 -0.0004 -0.0001 -0.0003 -0.0013 -0.0003   \n",
       "13      -0.0002  0.0004  0.6294 -0.0003 -0.0001 -0.0002 -0.0019 -0.0003   \n",
       "14       0.0018  0.0006  3.7579  0.0029  0.0005  0.0019 -0.0010  0.0018   \n",
       "15      -0.0004  0.0001  0.8719 -0.0005 -0.0002 -0.0004 -0.0020 -0.0004   \n",
       "16      -0.0004  0.0001  0.8719 -0.0005 -0.0002 -0.0004 -0.0020 -0.0004   \n",
       "17      -0.0003  0.0001  0.5895 -0.0003 -0.0002 -0.0003 -0.0015 -0.0003   \n",
       "18      -0.0006  0.0002  1.1626 -0.0007 -0.0003 -0.0005 -0.0026 -0.0006   \n",
       "19      -0.0007  0.0002  1.4532 -0.0009 -0.0003 -0.0007 -0.0033 -0.0007   \n",
       "20      -0.0001  0.0382 -0.0109  0.1028 -0.0001 -0.0000  0.0310  0.0001   \n",
       "21      -0.0002  0.0026  0.0025  0.0002  1.9947  0.0002  0.0068 -0.0018   \n",
       "22       0.0004  0.0001  0.0009  0.0004  0.0021  0.0004  0.0002  0.0004   \n",
       "23       0.0001  0.0001 -0.0003  0.0001  0.0022  0.0001 -0.0001  0.0001   \n",
       "24       0.0001  0.0000 -0.0001  0.0001  0.0011  0.0001 -0.0000  0.0001   \n",
       "25       0.0002  0.0001 -0.0004  0.0002  0.0001  0.0002  0.0002  0.0002   \n",
       "26       0.0001 -0.0005 -0.0013  0.0004 -0.0000  0.0001  0.0178  0.0001   \n",
       "27       0.0002 -0.0008 -0.0020  0.0005 -0.0000  0.0002  0.0268  0.0001   \n",
       "28       0.0002 -0.0008 -0.0020  0.0005 -0.0000  0.0002  0.0268  0.0001   \n",
       "29       0.0002 -0.0012 -0.0023  0.0008 -0.0001  0.0002  0.0292  0.0002   \n",
       "..          ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "70       0.0007  0.0009  0.0264 -0.0068  0.0001  0.0007  0.0021  0.0007   \n",
       "71       0.0010  0.0005 -0.0046  0.0024  0.0004  0.0010  0.0012  0.0009   \n",
       "72       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "73       0.0001  0.0001 -0.0010  0.0009 -0.0000  0.0001 -0.0002  0.0001   \n",
       "74       0.0001 -0.0025 -0.0085  0.0019  0.0002  0.0001 -0.0019  0.0001   \n",
       "75       0.0002  0.0002 -0.0016  0.0014 -0.0000  0.0002 -0.0003  0.0002   \n",
       "76       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "77       0.0001  0.0001 -0.0010  0.0009 -0.0000  0.0001 -0.0002  0.0001   \n",
       "78       0.0010  0.0019 -0.0016  0.0003 -0.0013  0.0010 -0.0420  0.0009   \n",
       "79       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "80      -0.0005  0.0004  0.0024 -0.0014 -0.0009 -0.0005 -0.0188 -0.0005   \n",
       "81      -0.0001 -0.0013 -0.0016  0.0124 -0.0002 -0.0001  0.0101 -0.0001   \n",
       "82       0.0001  0.0001 -0.0010  0.0009 -0.0000  0.0001 -0.0002  0.0001   \n",
       "83       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "84       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "85      -0.0000  0.0000 -0.0000  0.0000  0.0000 -0.0000 -0.0000  0.0000   \n",
       "86       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "87       0.0001  0.0002 -0.0015  0.0013 -0.0001  0.0001 -0.0003  0.0001   \n",
       "88       0.0002  0.0002 -0.0016  0.0014 -0.0000  0.0002 -0.0003  0.0002   \n",
       "89       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "90       0.0001  0.0006 -0.0075 -0.0001 -0.0003  0.0001 -0.0020  0.0001   \n",
       "91       0.0002  0.0001 -0.0008  0.0006  0.0046  0.0002 -0.0001  0.0002   \n",
       "92       0.0000 -0.0003 -0.0018  0.0007 -0.0000  0.0000  0.0015  0.0000   \n",
       "93       0.0002  0.0002 -0.0019  0.0018 -0.0001  0.0002 -0.0004  0.0002   \n",
       "94       0.0000 -0.0000  0.0000 -0.0000 -0.0000  0.0000  0.0000 -0.0000   \n",
       "95       0.0005  0.0032 -0.0093  0.0715  0.0008  0.0006 -0.0035  0.0006   \n",
       "96      -0.0003  0.0064  0.0016  0.0006  0.0022 -0.0003 -0.0088 -0.0003   \n",
       "97       0.0000  0.0001 -0.0005  0.0004 -0.0000  0.0000 -0.0001  0.0000   \n",
       "98       0.0001  0.0001 -0.0010  0.0009 -0.0000  0.0001 -0.0002  0.0001   \n",
       "99       0.0001 -0.0010 -0.0010  0.0003  0.0027  0.0001  0.0159 -0.0001   \n",
       "\n",
       "bookIDX   8       9       ...     21118   21119   21120   21121   21122  \\\n",
       "0        0.0011  0.0009   ...    0.0011  0.0011  0.0011  0.0011  0.0011   \n",
       "1       -0.0002  0.0014   ...    0.0002  0.0004  0.0000  0.0003 -0.0092   \n",
       "2       -0.0000 -0.0007   ...    0.0001  0.0002  0.0000  0.0002 -0.0019   \n",
       "3        0.0000 -0.0026   ...    0.0001  0.0001  0.0000  0.0001 -0.0003   \n",
       "4       -0.0006 -0.0019   ...   -0.0005 -0.0005 -0.0006 -0.0005 -0.0007   \n",
       "5       -0.0028 -0.0080   ...   -0.0026 -0.0025 -0.0028 -0.0026 -0.0025   \n",
       "6       -0.0003  0.0070   ...   -0.0003 -0.0002 -0.0004 -0.0003  0.0013   \n",
       "7       -0.0007 -0.0024   ...   -0.0006 -0.0006 -0.0007 -0.0006 -0.0008   \n",
       "8       -0.0006 -0.0019   ...   -0.0005 -0.0005 -0.0006 -0.0005 -0.0007   \n",
       "9       -0.0014 -0.0060   ...   -0.0012 -0.0012 -0.0015 -0.0012 -0.0033   \n",
       "10      -0.0001 -0.0019   ...   -0.0001 -0.0000 -0.0002 -0.0000 -0.0012   \n",
       "11      -0.0003 -0.0015   ...   -0.0002 -0.0002 -0.0003 -0.0002 -0.0004   \n",
       "12      -0.0003 -0.0010   ...   -0.0003 -0.0003 -0.0003 -0.0003 -0.0004   \n",
       "13      -0.0002 -0.0012   ...   -0.0002 -0.0002 -0.0002 -0.0002 -0.0004   \n",
       "14       0.0017 -0.0011   ...    0.0018  0.0019  0.0017  0.0019 -0.0000   \n",
       "15      -0.0004 -0.0015   ...   -0.0004 -0.0004 -0.0004 -0.0004 -0.0005   \n",
       "16      -0.0004 -0.0015   ...   -0.0004 -0.0004 -0.0004 -0.0004 -0.0005   \n",
       "17      -0.0003 -0.0009   ...   -0.0003 -0.0003 -0.0003 -0.0003 -0.0004   \n",
       "18      -0.0006 -0.0019   ...   -0.0005 -0.0005 -0.0006 -0.0005 -0.0007   \n",
       "19      -0.0007 -0.0024   ...   -0.0007 -0.0007 -0.0007 -0.0007 -0.0009   \n",
       "20      -0.0001  0.0626   ...   -0.0000 -0.0000 -0.0001 -0.0000 -0.0063   \n",
       "21      -0.0002  0.0018   ...    0.0001  0.0002 -0.0004  0.0002  0.0045   \n",
       "22       0.0004  0.0002   ...    0.0004  0.0004  0.0004  0.0004  0.0004   \n",
       "23       0.0001 -0.0001   ...    0.0001  0.0001  0.0001  0.0001  0.0001   \n",
       "24       0.0001 -0.0000   ...    0.0001  0.0001  0.0001  0.0001  0.0001   \n",
       "25       0.0002  0.0001   ...    0.0002  0.0002  0.0002  0.0002  0.0002   \n",
       "26       0.0001 -0.0002   ...    0.0001  0.0001  0.0001  0.0001  0.0004   \n",
       "27       0.0002 -0.0003   ...    0.0002  0.0002  0.0002  0.0002  0.0006   \n",
       "28       0.0002 -0.0003   ...    0.0002  0.0002  0.0002  0.0002  0.0006   \n",
       "29       0.0002 -0.0002   ...    0.0002  0.0002  0.0002  0.0002  0.0034   \n",
       "..          ...     ...   ...       ...     ...     ...     ...     ...   \n",
       "70       0.0007  0.1536   ...    0.0007  0.0007  0.0007  0.0007  0.0019   \n",
       "71       0.0009  0.0316   ...    0.0010  0.0009  0.0009  0.0010  0.0009   \n",
       "72       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "73       0.0001  0.0093   ...    0.0001  0.0001  0.0001  0.0001  0.0000   \n",
       "74       0.0001  0.0740   ...    0.0001  0.0001  0.0002  0.0001  0.0008   \n",
       "75       0.0002  0.0140   ...    0.0002  0.0002  0.0002  0.0002  0.0001   \n",
       "76       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "77       0.0001  0.0093   ...    0.0001  0.0001  0.0001  0.0001  0.0000   \n",
       "78       0.0010  0.0886   ...    0.0010  0.0010  0.0009  0.0010 -0.0073   \n",
       "79       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "80      -0.0005  0.0199   ...   -0.0005 -0.0004 -0.0005 -0.0005 -0.0030   \n",
       "81      -0.0001  0.0123   ...   -0.0001 -0.0001 -0.0001 -0.0001 -0.0006   \n",
       "82       0.0001  0.0093   ...    0.0001  0.0001  0.0001  0.0001  0.0000   \n",
       "83       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "84       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "85      -0.0000  0.0000   ...   -0.0000 -0.0000 -0.0000 -0.0000 -0.0000   \n",
       "86       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "87       0.0001  0.0139   ...    0.0001  0.0001  0.0001  0.0001  0.0000   \n",
       "88       0.0002  0.0140   ...    0.0002  0.0002  0.0002  0.0002  0.0001   \n",
       "89       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "90       0.0001  0.0225   ...    0.0001  0.0001  0.0001  0.0001 -0.0002   \n",
       "91       0.0002  0.0046   ...    0.0002  0.0002  0.0002  0.0002  0.0002   \n",
       "92       0.0000  0.0142   ...    0.0000  0.0000  0.0000  0.0000 -0.0001   \n",
       "93       0.0002  0.0186   ...    0.0002  0.0002  0.0002  0.0002  0.0000   \n",
       "94       0.0000 -0.0000   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "95       0.0006  0.2182   ...    0.0006  0.0006  0.0005  0.0006 -0.0023   \n",
       "96      -0.0003  0.0138   ...   -0.0003 -0.0003 -0.0004 -0.0003  0.0025   \n",
       "97       0.0000  0.0046   ...    0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "98       0.0001  0.0093   ...    0.0001  0.0001  0.0001  0.0001  0.0000   \n",
       "99       0.0001  0.0151   ...    0.0001  0.0001  0.0002  0.0001  0.0039   \n",
       "\n",
       "bookIDX   21123   21124   21125   21126   21127  \n",
       "0        0.0012  0.0011  0.0011  0.0010 -0.0001  \n",
       "1       -0.0107  0.0001 -0.0035 -0.0229 -0.0020  \n",
       "2       -0.0019  0.0001  0.0028  0.0008 -0.0003  \n",
       "3        0.0003  0.0000  0.0004 -0.0007  0.0000  \n",
       "4       -0.0010 -0.0005 -0.0001 -0.0010 -0.0000  \n",
       "5       -0.0040 -0.0027 -0.0022 -0.0035  0.0003  \n",
       "6       -0.0001 -0.0003  0.0009 -0.0100  0.0006  \n",
       "7       -0.0012 -0.0006 -0.0000 -0.0012 -0.0000  \n",
       "8       -0.0010 -0.0005 -0.0001 -0.0010 -0.0000  \n",
       "9       -0.0031 -0.0013  0.0010 -0.0003  0.0002  \n",
       "10      -0.0011 -0.0001  0.0013 -0.0006 -0.0002  \n",
       "11      -0.0015 -0.0003  0.0004 -0.0007 -0.0001  \n",
       "12      -0.0005 -0.0003 -0.0000 -0.0005 -0.0000  \n",
       "13      -0.0004 -0.0002  0.0001 -0.0002  0.0001  \n",
       "14       0.0011  0.0018  0.0033  0.0004 -0.0005  \n",
       "15      -0.0008 -0.0004 -0.0000 -0.0008 -0.0000  \n",
       "16      -0.0008 -0.0004 -0.0000 -0.0008 -0.0000  \n",
       "17      -0.0006 -0.0003 -0.0000 -0.0004 -0.0000  \n",
       "18      -0.0010 -0.0005 -0.0001 -0.0010 -0.0000  \n",
       "19      -0.0013 -0.0007 -0.0001 -0.0013 -0.0000  \n",
       "20      -0.0043 -0.0001 -0.0005  0.0377 -0.0005  \n",
       "21       0.0062 -0.0000 -0.0002  0.0015 -0.0007  \n",
       "22       0.0005  0.0004  0.0004  0.0003 -0.0002  \n",
       "23       0.0002  0.0001  0.0001 -0.0000 -0.0002  \n",
       "24       0.0001  0.0001  0.0001 -0.0000 -0.0001  \n",
       "25       0.0002  0.0002  0.0002  0.0002 -0.0000  \n",
       "26       0.0011  0.0001  0.0001  0.0003 -0.0001  \n",
       "27       0.0016  0.0002  0.0002  0.0005 -0.0001  \n",
       "28       0.0016  0.0002  0.0002  0.0005 -0.0001  \n",
       "29       0.0018  0.0002  0.0002  0.0054 -0.0004  \n",
       "..          ...     ...     ...     ...     ...  \n",
       "70       0.0023  0.0007  0.0006 -0.0086  0.0007  \n",
       "71       0.0015  0.0010  0.0009  0.0000  0.0000  \n",
       "72       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "73       0.0002  0.0001  0.0001 -0.0002  0.0001  \n",
       "74       0.0017  0.0001 -0.0002  0.0008 -0.0003  \n",
       "75       0.0003  0.0002  0.0002 -0.0002  0.0001  \n",
       "76       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "77       0.0002  0.0001  0.0001 -0.0002  0.0001  \n",
       "78      -0.0033  0.0010  0.0009  0.0015  0.0022  \n",
       "79       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "80      -0.0016 -0.0005 -0.0007 -0.0067 -0.0019  \n",
       "81      -0.0004 -0.0001 -0.0001  0.0006 -0.0002  \n",
       "82       0.0002  0.0001  0.0001 -0.0002  0.0001  \n",
       "83       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "84       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "85      -0.0000 -0.0000 -0.0000  0.0000  0.0000  \n",
       "86       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "87       0.0003  0.0001  0.0001 -0.0002  0.0001  \n",
       "88       0.0003  0.0002  0.0002 -0.0002  0.0001  \n",
       "89       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "90       0.0006  0.0001  0.0001 -0.0010  0.0000  \n",
       "91       0.0004  0.0002  0.0002 -0.0005 -0.0003  \n",
       "92       0.0008  0.0000  0.0000 -0.0002  0.0001  \n",
       "93       0.0003  0.0002  0.0002 -0.0003  0.0001  \n",
       "94       0.0000  0.0000  0.0000 -0.0000 -0.0000  \n",
       "95      -0.0001  0.0006  0.0004 -0.0038  0.0006  \n",
       "96      -0.0022 -0.0003 -0.0004 -0.0004 -0.0007  \n",
       "97       0.0001  0.0000  0.0000 -0.0001  0.0000  \n",
       "98       0.0002  0.0001  0.0001 -0.0002  0.0001  \n",
       "99       0.0013  0.0001  0.0001 -0.0008 -0.0006  \n",
       "\n",
       "[100 rows x 21128 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allPredictedRatingsDataFrame.round(decimals=4).head(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# allPredictedRatingsDataFrame.to_csv('RecommendedBooks-SVD.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>bookIDX</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>21118</th>\n",
       "      <th>21119</th>\n",
       "      <th>21120</th>\n",
       "      <th>21121</th>\n",
       "      <th>21122</th>\n",
       "      <th>21123</th>\n",
       "      <th>21124</th>\n",
       "      <th>21125</th>\n",
       "      <th>21126</th>\n",
       "      <th>21127</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userIDX</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 21128 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "bookIDX  0      1      2      3      4      5      6      7      8      9      \\\n",
       "userIDX                                                                         \n",
       "0          2.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "1          0.0    2.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "2          0.0    0.0    4.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "3          0.0    0.0    3.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "4          0.0    0.0    4.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "5          0.0    0.0    5.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "6          0.0    0.0    5.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "7          0.0    0.0    5.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "8          0.0    0.0    4.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "9          0.0    0.0    5.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "10         0.0    0.0    3.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "11         0.0    0.0    5.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "12         0.0    0.0    2.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "13         0.0    0.0    2.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "14         0.0    0.0    3.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "15         0.0    0.0    3.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "16         0.0    0.0    3.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "17         0.0    0.0    2.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "18         0.0    0.0    4.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "19         0.0    0.0    5.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "20         0.0    0.0    0.0    3.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "21         0.0    0.0    0.0    0.0    2.0    0.0    0.0    0.0    0.0    0.0   \n",
       "22         0.0    0.0    0.0    0.0    1.0    0.0    0.0    0.0    0.0    0.0   \n",
       "23         0.0    0.0    0.0    0.0    4.0    0.0    0.0    0.0    0.0    0.0   \n",
       "24         0.0    0.0    0.0    0.0    2.0    0.0    0.0    0.0    0.0    0.0   \n",
       "25         0.0    0.0    0.0    0.0    0.0    3.0    0.0    0.0    0.0    0.0   \n",
       "26         0.0    0.0    0.0    0.0    0.0    0.0    2.0    0.0    0.0    0.0   \n",
       "27         0.0    0.0    0.0    0.0    0.0    0.0    3.0    0.0    0.0    0.0   \n",
       "28         0.0    0.0    0.0    0.0    0.0    0.0    3.0    0.0    0.0    0.0   \n",
       "29         0.0    0.0    0.0    0.0    0.0    0.0    3.0    0.0    0.0    0.0   \n",
       "...        ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "70         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    3.0   \n",
       "71         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    4.0   \n",
       "72         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "73         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "74         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    5.0   \n",
       "75         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    3.0   \n",
       "76         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "77         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "78         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "79         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "80         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "81         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "82         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "83         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "84         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "85         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "86         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "87         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    3.0   \n",
       "88         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    3.0   \n",
       "89         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "90         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "91         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "92         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "93         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    4.0   \n",
       "94         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "95         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "96         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "97         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "98         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    2.0   \n",
       "99         0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    3.0   \n",
       "\n",
       "bookIDX  ...    21118  21119  21120  21121  21122  21123  21124  21125  21126  \\\n",
       "userIDX  ...                                                                    \n",
       "0        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "1        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "2        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "3        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "4        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "5        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "6        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "7        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "8        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "9        ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "10       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "11       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "12       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "13       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "14       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "15       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "16       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "17       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "18       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "19       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "20       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "21       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "22       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "23       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "24       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "25       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "26       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "27       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "28       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "29       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "...      ...      ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "70       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "71       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "72       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "73       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "74       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "75       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "76       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "77       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "78       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "79       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "80       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "81       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "82       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "83       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "84       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "85       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "86       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "87       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "88       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "89       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "90       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "91       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "92       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "93       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "94       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "95       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "96       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "97       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "98       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "99       ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "\n",
       "bookIDX  21127  \n",
       "userIDX         \n",
       "0          0.0  \n",
       "1          0.0  \n",
       "2          0.0  \n",
       "3          0.0  \n",
       "4          0.0  \n",
       "5          0.0  \n",
       "6          0.0  \n",
       "7          0.0  \n",
       "8          0.0  \n",
       "9          0.0  \n",
       "10         0.0  \n",
       "11         0.0  \n",
       "12         0.0  \n",
       "13         0.0  \n",
       "14         0.0  \n",
       "15         0.0  \n",
       "16         0.0  \n",
       "17         0.0  \n",
       "18         0.0  \n",
       "19         0.0  \n",
       "20         0.0  \n",
       "21         0.0  \n",
       "22         0.0  \n",
       "23         0.0  \n",
       "24         0.0  \n",
       "25         0.0  \n",
       "26         0.0  \n",
       "27         0.0  \n",
       "28         0.0  \n",
       "29         0.0  \n",
       "...        ...  \n",
       "70         0.0  \n",
       "71         0.0  \n",
       "72         0.0  \n",
       "73         0.0  \n",
       "74         0.0  \n",
       "75         0.0  \n",
       "76         0.0  \n",
       "77         0.0  \n",
       "78         0.0  \n",
       "79         0.0  \n",
       "80         0.0  \n",
       "81         0.0  \n",
       "82         0.0  \n",
       "83         0.0  \n",
       "84         0.0  \n",
       "85         0.0  \n",
       "86         0.0  \n",
       "87         0.0  \n",
       "88         0.0  \n",
       "89         0.0  \n",
       "90         0.0  \n",
       "91         0.0  \n",
       "92         0.0  \n",
       "93         0.0  \n",
       "94         0.0  \n",
       "95         0.0  \n",
       "96         0.0  \n",
       "97         0.0  \n",
       "98         0.0  \n",
       "99         0.0  \n",
       "\n",
       "[100 rows x 21128 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R_DataFrame.head(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.allclose(R_Matrix, allPredictedRatingsDataFrame.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ---------MODEL 1: Simple Matrix Multiplication Implementation--------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Users (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Books (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "UsersEmbedding (Embedding)      (None, 1, 5)         86265       Users[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "BooksEmbedding (Embedding)      (None, 1, 5)         105645      Books[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "UsersVec (Reshape)              (None, 5)            0           UsersEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "BooksVec (Reshape)              (None, 5)            0           BooksEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "DotProduct (Dot)                (None, 1)            0           UsersVec[0][0]                   \n",
      "                                                                 BooksVec[0][0]                   \n",
      "==================================================================================================\n",
      "Total params: 191,910\n",
      "Trainable params: 191,910\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "usersInput = keras.layers.Input(shape=[1],name='Users')\n",
    "booksInput = keras.layers.Input(shape=[1], name='Books')\n",
    "\n",
    "# check if we're enforcing non-negative Matrix Factorization\n",
    "# if so, include the constraint\n",
    "if NNMF:\n",
    "    usersEmbedding = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='UsersEmbedding',\n",
    "                                           embeddings_constraint=non_neg())(usersInput)\n",
    "    booksEmbedding = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='BooksEmbedding', \n",
    "                                             embeddings_constraint=non_neg())(booksInput)\n",
    "else:\n",
    "    usersEmbedding = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS, \n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='UsersEmbedding')(usersInput)\n",
    "    booksEmbedding = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='BooksEmbedding')(booksInput)\n",
    "    \n",
    "    \n",
    "usersVec = keras.layers.Reshape([MM_LATENT_FACTORS], name='UsersVec')(usersEmbedding)\n",
    "booksVec = keras.layers.Reshape([MM_LATENT_FACTORS], name='BooksVec')(booksEmbedding)\n",
    "\n",
    "\n",
    "dotProd = keras.layers.dot([usersVec, booksVec], axes=1, normalize=False, name='DotProduct')\n",
    "\n",
    "matMulModel = keras.Model([usersInput, booksInput], dotProd)\n",
    "\n",
    "\n",
    "if SGD_OPTIMIZER:\n",
    "    optimizer = SGD(SGD_lr, SGD_momentum, SGD_decay, SGD_nesterov)\n",
    "else:\n",
    "    optimizer = Adam(Adam_lr, Adam_beta_1, Adam_beta_2, Adam_epsilon, Adam_decay, Adam_amsgrad)\n",
    "\n",
    "matMulModel.compile(optimizer=optimizer, loss='mean_squared_error', metrics=['mse', 'acc'])\n",
    "\n",
    "matMulModel.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"264pt\" viewBox=\"0.00 0.00 400.00 264.00\" width=\"400pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 260)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-260 396,-260 396,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 118769550728 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>118769550728</title>\n",
       "<polygon fill=\"none\" points=\"33.5,-219.5 33.5,-255.5 151.5,-255.5 151.5,-219.5 33.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"92.5\" y=\"-233.8\">Users: InputLayer</text>\n",
       "</g>\n",
       "<!-- 118769551176 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>118769551176</title>\n",
       "<polygon fill=\"none\" points=\"0,-146.5 0,-182.5 185,-182.5 185,-146.5 0,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"92.5\" y=\"-160.8\">UsersEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 118769550728&#45;&gt;118769551176 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>118769550728-&gt;118769551176</title>\n",
       "<path d=\"M92.5,-219.4551C92.5,-211.3828 92.5,-201.6764 92.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"96.0001,-192.5903 92.5,-182.5904 89.0001,-192.5904 96.0001,-192.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118769549832 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>118769549832</title>\n",
       "<polygon fill=\"none\" points=\"236.5,-219.5 236.5,-255.5 358.5,-255.5 358.5,-219.5 236.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-233.8\">Books: InputLayer</text>\n",
       "</g>\n",
       "<!-- 118804781208 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>118804781208</title>\n",
       "<polygon fill=\"none\" points=\"203,-146.5 203,-182.5 392,-182.5 392,-146.5 203,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-160.8\">BooksEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 118769549832&#45;&gt;118804781208 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>118769549832-&gt;118804781208</title>\n",
       "<path d=\"M297.5,-219.4551C297.5,-211.3828 297.5,-201.6764 297.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"301.0001,-192.5903 297.5,-182.5904 294.0001,-192.5904 301.0001,-192.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118769551288 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>118769551288</title>\n",
       "<polygon fill=\"none\" points=\"58.5,-73.5 58.5,-109.5 184.5,-109.5 184.5,-73.5 58.5,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"121.5\" y=\"-87.8\">UsersVec: Reshape</text>\n",
       "</g>\n",
       "<!-- 118769551176&#45;&gt;118769551288 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>118769551176-&gt;118769551288</title>\n",
       "<path d=\"M99.6685,-146.4551C102.945,-138.2074 106.8993,-128.2536 110.5371,-119.0962\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"113.8741,-120.1761 114.3134,-109.5904 107.3687,-117.5917 113.8741,-120.1761\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118804779080 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>118804779080</title>\n",
       "<polygon fill=\"none\" points=\"217.5,-73.5 217.5,-109.5 347.5,-109.5 347.5,-73.5 217.5,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"282.5\" y=\"-87.8\">BooksVec: Reshape</text>\n",
       "</g>\n",
       "<!-- 118804781208&#45;&gt;118804779080 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>118804781208-&gt;118804779080</title>\n",
       "<path d=\"M293.7921,-146.4551C292.1154,-138.2951 290.0956,-128.4652 288.2306,-119.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"291.6584,-118.6812 286.2172,-109.5904 284.8016,-120.0902 291.6584,-118.6812\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118804779528 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>118804779528</title>\n",
       "<polygon fill=\"none\" points=\"140,-.5 140,-36.5 249,-36.5 249,-.5 140,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"194.5\" y=\"-14.8\">DotProduct: Dot</text>\n",
       "</g>\n",
       "<!-- 118769551288&#45;&gt;118804779528 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>118769551288-&gt;118804779528</title>\n",
       "<path d=\"M139.5449,-73.4551C148.4946,-64.5054 159.453,-53.547 169.2231,-43.7769\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"171.8134,-46.1363 176.4096,-36.5904 166.8637,-41.1866 171.8134,-46.1363\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118804779080&#45;&gt;118804779528 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>118804779080-&gt;118804779528</title>\n",
       "<path d=\"M260.7472,-73.4551C249.747,-64.3299 236.2294,-53.1165 224.2797,-43.2036\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"226.2387,-40.2812 216.3076,-36.5904 221.7695,-45.6688 226.2387,-40.2812\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_model(matMulModel, to_file='matMulModel.png', show_shapes=True)\n",
    "SVG(model_to_dot(matMulModel).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "callbacks = [keras.callbacks.EarlyStopping('val_loss', patience=3), \n",
    "             keras.callbacks.ModelCheckpoint('matMulModelWeights.h5', save_best_only=True)]\n",
    "\n",
    "trainingHistory = matMulModel.fit([trainDataFrame.userIDX, \n",
    "                                   trainDataFrame.bookIDX], \n",
    "                                  trainDataFrame.impressionRating,\n",
    "                                  validation_split=.1,  \n",
    "                                  callbacks=callbacks,\n",
    "                                  batch_size=MM_BATCH_SIZE,\n",
    "                                  epochs=MM_TRAIN_EPOCHS, \n",
    "                                  verbose=TRAIN_VERBOSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history['loss']).plot(logy=True, label=\"Train Loss\")\n",
    "pd.Series(trainingHistory.history['val_loss']).plot(logy=True, label=\"Val Loss\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Training Error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Make some preditions\n",
    "matMulRatingsPredictions = matMulModel.predict([testDataFrame.userIDX, testDataFrame.bookIDX])\n",
    "matMulRatingsPredictions = np.rint(matMulRatingsPredictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "displaySideBySide(pd.DataFrame(matMulRatingsPredictions).head(100), pd.DataFrame(actualImpressionRatings.head(100)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How did we do?\n",
    "print(\"MAE {0:2.2f}\".format(mean_absolute_error(actualImpressionRatings, matMulRatingsPredictions)))\n",
    "print(\"RMSE {0:2.2f}\".format(sqrt(mean_squared_error(actualImpressionRatings, matMulRatingsPredictions))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Retrieve the learnt embeddings\n",
    "booksLearntEmbeddings = matMulModel.get_layer(name='BooksEmbedding').get_weights()[0]\n",
    "usersLearntEmbeddings = matMulModel.get_layer(name='UsersEmbedding').get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displaySideBySide(pd.DataFrame(usersLearntEmbeddings).head(), pd.DataFrame(usersLearntEmbeddings).describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "displaySideBySide(pd.DataFrame(booksLearntEmbeddings).head(), pd.DataFrame(booksLearntEmbeddings).describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ----------Model 2: Neural Net Implementation (adding depth)-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Users (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Books (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "UsersEmbedding (Embedding)      (None, 1, 5)         86265       Users[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "BooksEmbedding (Embedding)      (None, 1, 5)         105645      Books[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "flatten_3 (Flatten)             (None, 5)            0           UsersEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "flatten_4 (Flatten)             (None, 5)            0           BooksEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "userBias (Embedding)            (None, 1, 1)         17253       Users[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "bookBias (Embedding)            (None, 1, 1)         21129       Books[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DotProduct (Dot)                (None, 1)            0           flatten_3[0][0]                  \n",
      "                                                                 flatten_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 1)            0           userBias[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "flatten_2 (Flatten)             (None, 1)            0           bookBias[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 3)            0           DotProduct[0][0]                 \n",
      "                                                                 flatten_1[0][0]                  \n",
      "                                                                 flatten_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 128)          512         concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 1)            129         dense_1[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 230,933\n",
      "Trainable params: 230,933\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "usersInput = keras.layers.Input(shape=[1],name='Users')\n",
    "booksInput = keras.layers.Input(shape=[1], name='Books')\n",
    "\n",
    "bias = 1\n",
    "\n",
    "# check if we're enforcing non-negative Matrix Factorization\n",
    "# if so, include the constraint\n",
    "if NNMF:\n",
    "    usersEmbedding = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='UsersEmbedding',\n",
    "                                           embeddings_constraint=non_neg())(usersInput)\n",
    "    booksEmbedding = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='BooksEmbedding', \n",
    "                                             embeddings_constraint=non_neg())(booksInput)\n",
    "else:\n",
    "    usersEmbedding = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='UsersEmbedding')(usersInput)\n",
    "    booksEmbedding = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS, \n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='BooksEmbedding')(booksInput)\n",
    "\n",
    "\n",
    "userBias = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                  output_dim=bias,\n",
    "                                  input_length=1,\n",
    "                                  name=\"userBias\")(usersInput)\n",
    "bookBias = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                  output_dim=bias,\n",
    "                                  input_length=1,\n",
    "                                  name=\"bookBias\")(booksInput)\n",
    "\n",
    "userBias = keras.layers.Flatten()(userBias)\n",
    "bookBias = keras.layers.Flatten()(bookBias)\n",
    "\n",
    "usersVec = keras.layers.Flatten()(usersEmbedding)\n",
    "booksVec = keras.layers.Flatten()(booksEmbedding)\n",
    "\n",
    "dotProd = keras.layers.dot([usersVec, booksVec], axes=1, normalize=True, name='DotProduct')\n",
    "\n",
    "\n",
    "\n",
    "concatVecs = keras.layers.Concatenate()([dotProd, userBias, bookBias])\n",
    "\n",
    "d1 = keras.layers.Dense(128, activation='relu')(concatVecs)\n",
    "d1_drop = keras.layers.Dropout(0.2, name='Dropout')(d1)\n",
    "\n",
    "d2 = keras.layers.Dense(1)(d1)\n",
    "d2_drop = keras.layers.Dropout(0.2, name='Dropout')(d2)\n",
    "\n",
    "neuralNetModel = keras.Model([usersInput, booksInput], d2)\n",
    "\n",
    "\n",
    "if SGD_OPTIMIZER:\n",
    "    optimizer = SGD(SGD_lr, SGD_momentum, SGD_decay, SGD_nesterov)\n",
    "else:\n",
    "    optimizer = Adam(Adam_lr, Adam_beta_1, Adam_beta_2, Adam_epsilon, Adam_decay, Adam_amsgrad)\n",
    "\n",
    "neuralNetModel.compile(optimizer=optimizer, loss='mean_squared_error', metrics=['mse', 'acc'])\n",
    "\n",
    "neuralNetModel.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"483pt\" viewBox=\"0.00 0.00 630.00 483.00\" width=\"630pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 479)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-479 626,-479 626,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 119800868312 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>119800868312</title>\n",
       "<polygon fill=\"none\" points=\"44.5,-438.5 44.5,-474.5 162.5,-474.5 162.5,-438.5 44.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"103.5\" y=\"-452.8\">Users: InputLayer</text>\n",
       "</g>\n",
       "<!-- 119970536024 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>119970536024</title>\n",
       "<polygon fill=\"none\" points=\"71,-365.5 71,-401.5 256,-401.5 256,-365.5 71,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"163.5\" y=\"-379.8\">UsersEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 119800868312&#45;&gt;119970536024 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>119800868312-&gt;119970536024</title>\n",
       "<path d=\"M118.3315,-438.4551C125.5431,-429.6809 134.3417,-418.9759 142.251,-409.353\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"144.9854,-411.5382 148.6312,-401.5904 139.5776,-407.0934 144.9854,-411.5382\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118578443320 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>118578443320</title>\n",
       "<polygon fill=\"none\" points=\"0,-292.5 0,-328.5 137,-328.5 137,-292.5 0,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"68.5\" y=\"-306.8\">userBias: Embedding</text>\n",
       "</g>\n",
       "<!-- 119800868312&#45;&gt;118578443320 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>119800868312-&gt;118578443320</title>\n",
       "<path d=\"M84.447,-438.1873C75.6392,-428.3922 66.1218,-415.569 61.5,-402 54.5111,-381.4816 56.9281,-356.9283 60.7272,-338.4243\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"64.1634,-339.0986 62.9958,-328.5683 57.3418,-337.5283 64.1634,-339.0986\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119800867864 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>119800867864</title>\n",
       "<polygon fill=\"none\" points=\"398.5,-438.5 398.5,-474.5 520.5,-474.5 520.5,-438.5 398.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"459.5\" y=\"-452.8\">Books: InputLayer</text>\n",
       "</g>\n",
       "<!-- 119970534344 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>119970534344</title>\n",
       "<polygon fill=\"none\" points=\"274,-365.5 274,-401.5 463,-401.5 463,-365.5 274,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"368.5\" y=\"-379.8\">BooksEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 119800867864&#45;&gt;119970534344 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>119800867864-&gt;119970534344</title>\n",
       "<path d=\"M437.0056,-438.4551C425.5211,-429.2422 411.383,-417.9006 398.9389,-407.918\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"401.0414,-405.1177 391.051,-401.5904 396.6612,-410.5779 401.0414,-405.1177\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119970533448 -->\n",
       "<g class=\"node\" id=\"node8\">\n",
       "<title>119970533448</title>\n",
       "<polygon fill=\"none\" points=\"481,-365.5 481,-401.5 622,-401.5 622,-365.5 481,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"551.5\" y=\"-379.8\">bookBias: Embedding</text>\n",
       "</g>\n",
       "<!-- 119800867864&#45;&gt;119970533448 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>119800867864-&gt;119970533448</title>\n",
       "<path d=\"M482.2416,-438.4551C493.8523,-429.2422 508.1458,-417.9006 520.7266,-407.918\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"523.0431,-410.5479 528.7012,-401.5904 518.6921,-405.0644 523.0431,-410.5479\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120037506520 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>120037506520</title>\n",
       "<polygon fill=\"none\" points=\"168.5,-292.5 168.5,-328.5 280.5,-328.5 280.5,-292.5 168.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"224.5\" y=\"-306.8\">flatten_3: Flatten</text>\n",
       "</g>\n",
       "<!-- 119970536024&#45;&gt;120037506520 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>119970536024-&gt;120037506520</title>\n",
       "<path d=\"M178.5787,-365.4551C185.9105,-356.6809 194.8557,-345.9759 202.8968,-336.353\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"205.6569,-338.5082 209.3834,-328.5904 200.2854,-334.0197 205.6569,-338.5082\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119987052440 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>119987052440</title>\n",
       "<polygon fill=\"none\" points=\"305.5,-292.5 305.5,-328.5 417.5,-328.5 417.5,-292.5 305.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"361.5\" y=\"-306.8\">flatten_4: Flatten</text>\n",
       "</g>\n",
       "<!-- 119970534344&#45;&gt;119987052440 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>119970534344-&gt;119987052440</title>\n",
       "<path d=\"M366.7697,-365.4551C365.9956,-357.3828 365.0649,-347.6764 364.2024,-338.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"367.6733,-338.2106 363.2347,-328.5904 360.7053,-338.8788 367.6733,-338.2106\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120020922888 -->\n",
       "<g class=\"node\" id=\"node9\">\n",
       "<title>120020922888</title>\n",
       "<polygon fill=\"none\" points=\"206,-219.5 206,-255.5 315,-255.5 315,-219.5 206,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"260.5\" y=\"-233.8\">DotProduct: Dot</text>\n",
       "</g>\n",
       "<!-- 120037506520&#45;&gt;120020922888 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "<title>120037506520-&gt;120020922888</title>\n",
       "<path d=\"M233.3989,-292.4551C237.5095,-284.1196 242.4795,-274.0416 247.0349,-264.8042\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"250.2948,-266.1071 251.5787,-255.5904 244.0167,-263.011 250.2948,-266.1071\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119987052440&#45;&gt;120020922888 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "<title>119987052440-&gt;120020922888</title>\n",
       "<path d=\"M336.5337,-292.4551C323.6658,-283.1545 307.7962,-271.6844 293.8898,-261.6332\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"295.6841,-258.6116 285.5291,-255.5904 291.5836,-264.2849 295.6841,-258.6116\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119970533896 -->\n",
       "<g class=\"node\" id=\"node10\">\n",
       "<title>119970533896</title>\n",
       "<polygon fill=\"none\" points=\"62.5,-219.5 62.5,-255.5 174.5,-255.5 174.5,-219.5 62.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"118.5\" y=\"-233.8\">flatten_1: Flatten</text>\n",
       "</g>\n",
       "<!-- 118578443320&#45;&gt;119970533896 -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "<title>118578443320-&gt;119970533896</title>\n",
       "<path d=\"M80.8596,-292.4551C86.7491,-283.8564 93.9087,-273.4034 100.3961,-263.9316\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"103.346,-265.8185 106.1093,-255.5904 97.5708,-261.8628 103.346,-265.8185\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119254544280 -->\n",
       "<g class=\"node\" id=\"node11\">\n",
       "<title>119254544280</title>\n",
       "<polygon fill=\"none\" points=\"405.5,-219.5 405.5,-255.5 517.5,-255.5 517.5,-219.5 405.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"461.5\" y=\"-233.8\">flatten_2: Flatten</text>\n",
       "</g>\n",
       "<!-- 119970533448&#45;&gt;119254544280 -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "<title>119970533448-&gt;119254544280</title>\n",
       "<path d=\"M540.2834,-365.3042C524.6539,-339.9496 496.211,-293.8089 478.1158,-264.4546\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"481.0849,-262.6011 472.8579,-255.9251 475.1261,-266.2743 481.0849,-262.6011\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120004394568 -->\n",
       "<g class=\"node\" id=\"node12\">\n",
       "<title>120004394568</title>\n",
       "<polygon fill=\"none\" points=\"174,-146.5 174,-182.5 347,-182.5 347,-146.5 174,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"260.5\" y=\"-160.8\">concatenate_1: Concatenate</text>\n",
       "</g>\n",
       "<!-- 120020922888&#45;&gt;120004394568 -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "<title>120020922888-&gt;120004394568</title>\n",
       "<path d=\"M260.5,-219.4551C260.5,-211.3828 260.5,-201.6764 260.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"264.0001,-192.5903 260.5,-182.5904 257.0001,-192.5904 264.0001,-192.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119970533896&#45;&gt;120004394568 -->\n",
       "<g class=\"edge\" id=\"edge12\">\n",
       "<title>119970533896-&gt;120004394568</title>\n",
       "<path d=\"M153.6011,-219.4551C172.5461,-209.7157 196.1186,-197.5975 216.3035,-187.2207\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"218.0172,-190.2752 225.3105,-182.5904 214.8167,-184.0497 218.0172,-190.2752\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119254544280&#45;&gt;120004394568 -->\n",
       "<g class=\"edge\" id=\"edge13\">\n",
       "<title>119254544280-&gt;120004394568</title>\n",
       "<path d=\"M411.8146,-219.4551C384.0318,-209.3648 349.2181,-196.721 319.9902,-186.1059\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"320.9046,-182.7143 310.3104,-182.5904 318.5149,-189.2939 320.9046,-182.7143\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120020923336 -->\n",
       "<g class=\"node\" id=\"node13\">\n",
       "<title>120020923336</title>\n",
       "<polygon fill=\"none\" points=\"208,-73.5 208,-109.5 313,-109.5 313,-73.5 208,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"260.5\" y=\"-87.8\">dense_1: Dense</text>\n",
       "</g>\n",
       "<!-- 120004394568&#45;&gt;120020923336 -->\n",
       "<g class=\"edge\" id=\"edge14\">\n",
       "<title>120004394568-&gt;120020923336</title>\n",
       "<path d=\"M260.5,-146.4551C260.5,-138.3828 260.5,-128.6764 260.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"264.0001,-119.5903 260.5,-109.5904 257.0001,-119.5904 264.0001,-119.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 119918071256 -->\n",
       "<g class=\"node\" id=\"node14\">\n",
       "<title>119918071256</title>\n",
       "<polygon fill=\"none\" points=\"208,-.5 208,-36.5 313,-36.5 313,-.5 208,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"260.5\" y=\"-14.8\">dense_2: Dense</text>\n",
       "</g>\n",
       "<!-- 120020923336&#45;&gt;119918071256 -->\n",
       "<g class=\"edge\" id=\"edge15\">\n",
       "<title>120020923336-&gt;119918071256</title>\n",
       "<path d=\"M260.5,-73.4551C260.5,-65.3828 260.5,-55.6764 260.5,-46.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"264.0001,-46.5903 260.5,-36.5904 257.0001,-46.5904 264.0001,-46.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_model(neuralNetModel, to_file='neuralNetModel.png', show_shapes=True)\n",
    "SVG(model_to_dot(neuralNetModel).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "callbacks = [keras.callbacks.EarlyStopping('val_loss', patience=3), \n",
    "             keras.callbacks.ModelCheckpoint('neuralNetModelWeights.h5', save_best_only=True)]\n",
    "\n",
    "trainingHistory = neuralNetModel.fit([trainDataFrame.userIDX, trainDataFrame.bookIDX], \n",
    "                                     trainDataFrame.impressionRating, \n",
    "                                     validation_split=.1, \n",
    "                                     callbacks=callbacks,\n",
    "                                     batch_size=NN_BATCH_SIZE,\n",
    "                                     epochs=NN_TRAIN_EPOCHS, \n",
    "                                     verbose=TRAIN_VERBOSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history['loss']).plot(logy=True, label=\"Train Loss\")\n",
    "pd.Series(trainingHistory.history['val_loss']).plot(logy=True, label=\"Val Loss\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Training Error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make some preditions\n",
    "neuralNetRatingsPredictions = neuralNetModel.predict([testDataFrame.userIDX, \n",
    "                                                      testDataFrame.bookIDX])\n",
    "neuralNetRatingsPredictions = np.rint(neuralNetRatingsPredictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displaySideBySide(pd.DataFrame(neuralNetRatingsPredictions).head(30), \n",
    "                  pd.DataFrame(actualImpressionRatings).head(30))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ----------Model 3: Neural Net Implementation (adding Bias and 2 features: Author & YearOfPublication)-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Users (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Books (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "UsersEmbedding (Embedding)      (None, 1, 5)         86265       Users[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "BooksEmbedding (Embedding)      (None, 1, 5)         105645      Books[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "flatten_7 (Flatten)             (None, 5)            0           UsersEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "flatten_8 (Flatten)             (None, 5)            0           BooksEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "userBias (Embedding)            (None, 1, 1)         17253       Users[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "bookBias (Embedding)            (None, 1, 1)         21129       Books[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DotProduct (Dot)                (None, 1)            0           flatten_7[0][0]                  \n",
      "                                                                 flatten_8[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "flatten_5 (Flatten)             (None, 1)            0           userBias[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "flatten_6 (Flatten)             (None, 1)            0           bookBias[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "Authors (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "YearOfPub (InputLayer)          (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 5)            0           DotProduct[0][0]                 \n",
      "                                                                 flatten_5[0][0]                  \n",
      "                                                                 flatten_6[0][0]                  \n",
      "                                                                 Authors[0][0]                    \n",
      "                                                                 YearOfPub[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 128)          768         concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 1)            129         dense_3[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 231,189\n",
      "Trainable params: 231,189\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "usersInput = keras.layers.Input(shape=[1],name='Users')\n",
    "booksInput = keras.layers.Input(shape=[1], name='Books')\n",
    "authorsInput = keras.layers.Input(shape=[1], name='Authors')\n",
    "yearOfPubInput = keras.layers.Input(shape=[1], name='YearOfPub')\n",
    "\n",
    "bias = 1\n",
    "\n",
    "# check if we're enforcing non-negative Matrix Factorization\n",
    "# if so, include the constraint\n",
    "if NNMF:\n",
    "    usersEmbedding = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS,\n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='UsersEmbedding',\n",
    "                                            embeddings_constraint=non_neg())(usersInput)\n",
    "    booksEmbedding = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS, \n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='BooksEmbedding', \n",
    "                                            embeddings_constraint=non_neg())(booksInput)\n",
    "else:\n",
    "    usersEmbedding = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS, \n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='UsersEmbedding')(usersInput)\n",
    "    booksEmbedding = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                            output_dim=MM_LATENT_FACTORS, \n",
    "                                            embeddings_regularizer=l2(1e-4),\n",
    "                                            name='BooksEmbedding')(booksInput)\n",
    "\n",
    "\n",
    "userBias = keras.layers.Embedding(input_dim=numOfUsers + 1, \n",
    "                                  output_dim=bias,\n",
    "                                  input_length=1,\n",
    "                                  name=\"userBias\")(usersInput)\n",
    "bookBias = keras.layers.Embedding(input_dim=numOfBooks + 1, \n",
    "                                  output_dim=bias,\n",
    "                                  input_length=1,\n",
    "                                  name=\"bookBias\")(booksInput)\n",
    "\n",
    "userBias = keras.layers.Flatten()(userBias)\n",
    "bookBias = keras.layers.Flatten()(bookBias)\n",
    "\n",
    "usersVec = keras.layers.Flatten()(usersEmbedding)\n",
    "booksVec = keras.layers.Flatten()(booksEmbedding)\n",
    "\n",
    "dotProd = keras.layers.dot([usersVec, booksVec], axes=1, normalize=True, name='DotProduct')\n",
    "\n",
    "\n",
    "\n",
    "concatVecs = keras.layers.Concatenate()([dotProd, userBias, bookBias, authorsInput, yearOfPubInput])\n",
    "\n",
    "d1 = keras.layers.Dense(128, activation='relu')(concatVecs)\n",
    "d1_drop = keras.layers.Dropout(0.2, name='Dropout')(d1)\n",
    "\n",
    "d2 = keras.layers.Dense(1)(d1)\n",
    "d2_drop = keras.layers.Dropout(0.2, name='Dropout')(d2)\n",
    "\n",
    "neuralNetModel = keras.Model(inputs=[usersInput, booksInput, authorsInput, yearOfPubInput], outputs=d2)\n",
    "\n",
    "\n",
    "if SGD_OPTIMIZER:\n",
    "    optimizer = SGD(SGD_lr, SGD_momentum, SGD_decay, SGD_nesterov)\n",
    "else:\n",
    "    optimizer = Adam(Adam_lr, Adam_beta_1, Adam_beta_2, Adam_epsilon, Adam_decay, Adam_amsgrad)\n",
    "\n",
    "neuralNetModel.compile(optimizer=optimizer, loss='mean_squared_error', metrics=['mse', 'acc'])\n",
    "\n",
    "neuralNetModel.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"483pt\" viewBox=\"0.00 0.00 711.50 483.00\" width=\"712pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 479)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-479 707.5,-479 707.5,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 120967574360 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>120967574360</title>\n",
       "<polygon fill=\"none\" points=\"44.5,-438.5 44.5,-474.5 162.5,-474.5 162.5,-438.5 44.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"103.5\" y=\"-452.8\">Users: InputLayer</text>\n",
       "</g>\n",
       "<!-- 120967574472 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>120967574472</title>\n",
       "<polygon fill=\"none\" points=\"71,-365.5 71,-401.5 256,-401.5 256,-365.5 71,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"163.5\" y=\"-379.8\">UsersEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 120967574360&#45;&gt;120967574472 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>120967574360-&gt;120967574472</title>\n",
       "<path d=\"M118.3315,-438.4551C125.5431,-429.6809 134.3417,-418.9759 142.251,-409.353\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"144.9854,-411.5382 148.6312,-401.5904 139.5776,-407.0934 144.9854,-411.5382\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120967573912 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>120967573912</title>\n",
       "<polygon fill=\"none\" points=\"0,-292.5 0,-328.5 137,-328.5 137,-292.5 0,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"68.5\" y=\"-306.8\">userBias: Embedding</text>\n",
       "</g>\n",
       "<!-- 120967574360&#45;&gt;120967573912 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>120967574360-&gt;120967573912</title>\n",
       "<path d=\"M84.447,-438.1873C75.6392,-428.3922 66.1218,-415.569 61.5,-402 54.5111,-381.4816 56.9281,-356.9283 60.7272,-338.4243\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"64.1634,-339.0986 62.9958,-328.5683 57.3418,-337.5283 64.1634,-339.0986\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120814587016 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>120814587016</title>\n",
       "<polygon fill=\"none\" points=\"398.5,-438.5 398.5,-474.5 520.5,-474.5 520.5,-438.5 398.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"459.5\" y=\"-452.8\">Books: InputLayer</text>\n",
       "</g>\n",
       "<!-- 120967491992 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>120967491992</title>\n",
       "<polygon fill=\"none\" points=\"274,-365.5 274,-401.5 463,-401.5 463,-365.5 274,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"368.5\" y=\"-379.8\">BooksEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 120814587016&#45;&gt;120967491992 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>120814587016-&gt;120967491992</title>\n",
       "<path d=\"M437.0056,-438.4551C425.5211,-429.2422 411.383,-417.9006 398.9389,-407.918\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"401.0414,-405.1177 391.051,-401.5904 396.6612,-410.5779 401.0414,-405.1177\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120967494344 -->\n",
       "<g class=\"node\" id=\"node8\">\n",
       "<title>120967494344</title>\n",
       "<polygon fill=\"none\" points=\"481,-365.5 481,-401.5 622,-401.5 622,-365.5 481,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"551.5\" y=\"-379.8\">bookBias: Embedding</text>\n",
       "</g>\n",
       "<!-- 120814587016&#45;&gt;120967494344 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>120814587016-&gt;120967494344</title>\n",
       "<path d=\"M482.2416,-438.4551C493.8523,-429.2422 508.1458,-417.9006 520.7266,-407.918\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"523.0431,-410.5479 528.7012,-401.5904 518.6921,-405.0644 523.0431,-410.5479\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121086269592 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>121086269592</title>\n",
       "<polygon fill=\"none\" points=\"168.5,-292.5 168.5,-328.5 280.5,-328.5 280.5,-292.5 168.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"224.5\" y=\"-306.8\">flatten_7: Flatten</text>\n",
       "</g>\n",
       "<!-- 120967574472&#45;&gt;121086269592 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>120967574472-&gt;121086269592</title>\n",
       "<path d=\"M178.5787,-365.4551C185.9105,-356.6809 194.8557,-345.9759 202.8968,-336.353\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"205.6569,-338.5082 209.3834,-328.5904 200.2854,-334.0197 205.6569,-338.5082\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121103556504 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>121103556504</title>\n",
       "<polygon fill=\"none\" points=\"305.5,-292.5 305.5,-328.5 417.5,-328.5 417.5,-292.5 305.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"361.5\" y=\"-306.8\">flatten_8: Flatten</text>\n",
       "</g>\n",
       "<!-- 120967491992&#45;&gt;121103556504 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>120967491992-&gt;121103556504</title>\n",
       "<path d=\"M366.7697,-365.4551C365.9956,-357.3828 365.0649,-347.6764 364.2024,-338.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"367.6733,-338.2106 363.2347,-328.5904 360.7053,-338.8788 367.6733,-338.2106\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121137210312 -->\n",
       "<g class=\"node\" id=\"node9\">\n",
       "<title>121137210312</title>\n",
       "<polygon fill=\"none\" points=\"238,-219.5 238,-255.5 347,-255.5 347,-219.5 238,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.5\" y=\"-233.8\">DotProduct: Dot</text>\n",
       "</g>\n",
       "<!-- 121086269592&#45;&gt;121137210312 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "<title>121086269592-&gt;121137210312</title>\n",
       "<path d=\"M241.309,-292.4551C249.5639,-283.5932 259.6535,-272.7616 268.6864,-263.0646\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"271.3937,-265.2932 275.6487,-255.5904 266.2716,-260.522 271.3937,-265.2932\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121103556504&#45;&gt;121137210312 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "<title>121103556504-&gt;121137210312</title>\n",
       "<path d=\"M344.4438,-292.4551C336.0675,-283.5932 325.8295,-272.7616 316.6638,-263.0646\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"319.0119,-260.4535 309.5991,-255.5904 313.9247,-265.262 319.0119,-260.4535\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120967412312 -->\n",
       "<g class=\"node\" id=\"node10\">\n",
       "<title>120967412312</title>\n",
       "<polygon fill=\"none\" points=\"94.5,-219.5 94.5,-255.5 206.5,-255.5 206.5,-219.5 94.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"150.5\" y=\"-233.8\">flatten_5: Flatten</text>\n",
       "</g>\n",
       "<!-- 120967573912&#45;&gt;120967412312 -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "<title>120967573912-&gt;120967412312</title>\n",
       "<path d=\"M88.7697,-292.4551C98.9212,-283.4177 111.3738,-272.3319 122.4291,-262.4899\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"125.0375,-264.8539 130.1793,-255.5904 120.383,-259.6255 125.0375,-264.8539\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120950419864 -->\n",
       "<g class=\"node\" id=\"node11\">\n",
       "<title>120950419864</title>\n",
       "<polygon fill=\"none\" points=\"465.5,-292.5 465.5,-328.5 577.5,-328.5 577.5,-292.5 465.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"521.5\" y=\"-306.8\">flatten_6: Flatten</text>\n",
       "</g>\n",
       "<!-- 120967494344&#45;&gt;120950419864 -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "<title>120967494344-&gt;120950419864</title>\n",
       "<path d=\"M544.0843,-365.4551C540.6948,-357.2074 536.6042,-347.2536 532.8409,-338.0962\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"535.9729,-336.5094 528.9344,-328.5904 529.4983,-339.1702 535.9729,-336.5094\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121052487576 -->\n",
       "<g class=\"node\" id=\"node14\">\n",
       "<title>121052487576</title>\n",
       "<polygon fill=\"none\" points=\"289,-146.5 289,-182.5 462,-182.5 462,-146.5 289,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-160.8\">concatenate_2: Concatenate</text>\n",
       "</g>\n",
       "<!-- 121137210312&#45;&gt;121052487576 -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "<title>121137210312-&gt;121052487576</title>\n",
       "<path d=\"M313.0169,-219.4551C323.392,-210.3299 336.1416,-199.1165 347.4123,-189.2036\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"349.7341,-191.8228 354.9315,-182.5904 345.1111,-186.5665 349.7341,-191.8228\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120967412312&#45;&gt;121052487576 -->\n",
       "<g class=\"edge\" id=\"edge12\">\n",
       "<title>120967412312-&gt;121052487576</title>\n",
       "<path d=\"M206.118,-219.4551C237.6238,-209.2332 277.2066,-196.3907 310.1836,-185.6916\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"311.3103,-189.0057 319.742,-182.5904 309.15,-182.3473 311.3103,-189.0057\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120950419864&#45;&gt;121052487576 -->\n",
       "<g class=\"edge\" id=\"edge13\">\n",
       "<title>120950419864-&gt;121052487576</title>\n",
       "<path d=\"M465.2207,-298.8693C440.0765,-290.8764 412.1863,-277.645 394.5,-256 380.1216,-238.4032 375.7891,-212.672 374.814,-192.9747\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"378.3076,-192.6581 374.5752,-182.7425 371.3095,-192.8215 378.3076,-192.6581\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 118578444440 -->\n",
       "<g class=\"node\" id=\"node12\">\n",
       "<title>118578444440</title>\n",
       "<polygon fill=\"none\" points=\"404,-219.5 404,-255.5 535,-255.5 535,-219.5 404,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"469.5\" y=\"-233.8\">Authors: InputLayer</text>\n",
       "</g>\n",
       "<!-- 118578444440&#45;&gt;121052487576 -->\n",
       "<g class=\"edge\" id=\"edge14\">\n",
       "<title>118578444440-&gt;121052487576</title>\n",
       "<path d=\"M446.264,-219.4551C434.4009,-210.2422 419.7967,-198.9006 406.9424,-188.918\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"408.8393,-185.9597 398.7944,-182.5904 404.5457,-191.4883 408.8393,-185.9597\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120967573576 -->\n",
       "<g class=\"node\" id=\"node13\">\n",
       "<title>120967573576</title>\n",
       "<polygon fill=\"none\" points=\"553.5,-219.5 553.5,-255.5 703.5,-255.5 703.5,-219.5 553.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"628.5\" y=\"-233.8\">YearOfPub: InputLayer</text>\n",
       "</g>\n",
       "<!-- 120967573576&#45;&gt;121052487576 -->\n",
       "<g class=\"edge\" id=\"edge15\">\n",
       "<title>120967573576-&gt;121052487576</title>\n",
       "<path d=\"M565.9607,-219.4551C530.23,-209.1454 485.2607,-196.1701 447.9915,-185.4165\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"448.7751,-181.9999 438.1967,-182.5904 446.8344,-188.7255 448.7751,-181.9999\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121103552584 -->\n",
       "<g class=\"node\" id=\"node15\">\n",
       "<title>121103552584</title>\n",
       "<polygon fill=\"none\" points=\"323,-73.5 323,-109.5 428,-109.5 428,-73.5 323,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-87.8\">dense_3: Dense</text>\n",
       "</g>\n",
       "<!-- 121052487576&#45;&gt;121103552584 -->\n",
       "<g class=\"edge\" id=\"edge16\">\n",
       "<title>121052487576-&gt;121103552584</title>\n",
       "<path d=\"M375.5,-146.4551C375.5,-138.3828 375.5,-128.6764 375.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"379.0001,-119.5903 375.5,-109.5904 372.0001,-119.5904 379.0001,-119.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 121187629768 -->\n",
       "<g class=\"node\" id=\"node16\">\n",
       "<title>121187629768</title>\n",
       "<polygon fill=\"none\" points=\"323,-.5 323,-36.5 428,-36.5 428,-.5 323,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-14.8\">dense_4: Dense</text>\n",
       "</g>\n",
       "<!-- 121103552584&#45;&gt;121187629768 -->\n",
       "<g class=\"edge\" id=\"edge17\">\n",
       "<title>121103552584-&gt;121187629768</title>\n",
       "<path d=\"M375.5,-73.4551C375.5,-65.3828 375.5,-55.6764 375.5,-46.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"379.0001,-46.5903 375.5,-36.5904 372.0001,-46.5904 379.0001,-46.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_model(neuralNetModel, to_file='neuralNetModel.png', show_shapes=True)\n",
    "SVG(model_to_dot(neuralNetModel).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [keras.callbacks.EarlyStopping('val_loss', patience=3), \n",
    "             keras.callbacks.ModelCheckpoint('neuralNetModelWeights.h5', save_best_only=True)]\n",
    "\n",
    "trainingHistory = neuralNetModel.fit([trainDataFrame.userIDX, \n",
    "                                      trainDataFrame.bookIDX, \n",
    "                                      trainDataFrame.authorIDX, \n",
    "                                      trainDataFrame.yearOfPublication], \n",
    "                                     trainDataFrame.impressionRating, \n",
    "                                     validation_split=.1, \n",
    "                                     callbacks=callbacks,\n",
    "                                     batch_size=NN_BATCH_SIZE,\n",
    "                                     epochs=NN_TRAIN_EPOCHS, \n",
    "                                     verbose=TRAIN_VERBOSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history['loss']).plot(logy=True, label=\"Train Loss\")\n",
    "pd.Series(trainingHistory.history['val_loss']).plot(logy=True, label=\"Val Loss\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Training Error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make some preditions\n",
    "neuralNetRatingsPredictions = neuralNetModel.predict([testDataFrame.userIDX, \n",
    "                                                      testDataFrame.bookIDX,\n",
    "                                                     trainDataFrame.yearOfPublication, \n",
    "                                     trainDataFrame.impressionRating])\n",
    "neuralNetRatingsPredictions = np.rint(neuralNetRatingsPredictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displaySideBySide(pd.DataFrame(neuralNetRatingsPredictions).head(30), \n",
    "                  pd.DataFrame(actualImpressionRatings).head(30))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ----------Model 4: Neural Net Implementation (adding more depth)-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Books (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Users (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "BooksEmbedding (Embedding)      (None, 1, 5)         105645      Books[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "UsesEmbedding (Embedding)       (None, 1, 5)         86265       Users[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "FlattenBooks (Flatten)          (None, 5)            0           BooksEmbedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "FlattenUsers (Flatten)          (None, 5)            0           UsesEmbedding[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 5)            0           FlattenBooks[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 5)            0           FlattenUsers[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "ConcatLayer (Concatenate)       (None, 10)           0           dropout_1[0][0]                  \n",
      "                                                                 dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-1 (Dense)        (None, 100)          1100        ConcatLayer[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-2 (Dense)        (None, 50)           5050        FullyConnected-1[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-3 (Dense)        (None, 20)           1020        FullyConnected-2[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Activation (Dense)              (None, 1)            21          FullyConnected-3[0][0]           \n",
      "==================================================================================================\n",
      "Total params: 199,101\n",
      "Trainable params: 199,101\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Neural Net Implementation\n",
    "\n",
    "booksInput = keras.layers.Input(shape=[1], name='Books')\n",
    "booksEmbedding = keras.layers.Embedding(numOfBooks + 1, \n",
    "                                         NN_MOVIE_LATENT_FACTORS,\n",
    "                                        embeddings_regularizer=l2(1e-4),\n",
    "                                         name='BooksEmbedding')(booksInput)\n",
    "booksVec = keras.layers.Flatten(name='FlattenBooks')(booksEmbedding)\n",
    "booksVec = keras.layers.Dropout(0.2)(booksVec)\n",
    "\n",
    "\n",
    "usersInput = keras.layers.Input(shape=[1], name='Users')\n",
    "usersEmbedding = keras.layers.Embedding(numOfUsers + 1, \n",
    "                                        NN_USER_LATENT_FACTORS, \n",
    "                                        embeddings_regularizer=l2(1e-4),\n",
    "                                        name='UsesEmbedding')(usersInput)\n",
    "usersVec = keras.layers.Flatten(name='FlattenUsers')(usersEmbedding)\n",
    "usersVec = keras.layers.Dropout(0.2)(usersVec)\n",
    "\n",
    "concatLayer = keras.layers.concatenate([booksVec, usersVec], axis=1, name='ConcatLayer')\n",
    "concatDropout = keras.layers.Dropout(0.2)(concatLayer)\n",
    "\n",
    "denseLayer_1 = keras.layers.Dense(200, name='FullyConnected-0')(concatLayer)\n",
    "dropoutLayer_1 = keras.layers.Dropout(0.2, name='Dropout')(denseLayer_1)\n",
    "\n",
    "denseLayer_2 = keras.layers.Dense(100, name='FullyConnected-1')(concatLayer)\n",
    "dropoutLayer_2 = keras.layers.Dropout(0.2, name='Dropout')(denseLayer_2)\n",
    "\n",
    "denseLayer_3 = keras.layers.Dense(50, name='FullyConnected-2')(denseLayer_2)\n",
    "dropoutLayer_3 = keras.layers.Dropout(0.2, name='Dropout')(denseLayer_3)\n",
    "\n",
    "denseLayer_4 = keras.layers.Dense(20, name='FullyConnected-3', activation='relu')(denseLayer_3)\n",
    "result = keras.layers.Dense(1, activation='relu', name='Activation')(denseLayer_4)\n",
    "\n",
    "if SGD_OPTIMIZER:\n",
    "    optimizer = SGD(SGD_lr, SGD_momentum, SGD_decay, SGD_nesterov)\n",
    "else:\n",
    "    optimizer = Adam(Adam_lr, Adam_beta_1, Adam_beta_2, Adam_epsilon, Adam_decay, Adam_amsgrad)\n",
    "\n",
    "neuralNetModel = keras.Model([usersInput, booksInput], result)\n",
    "neuralNetModel.compile(optimizer=optimizer, loss='mean_squared_error', metrics=['mse', 'acc'])\n",
    "\n",
    "neuralNetModel.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"629pt\" viewBox=\"0.00 0.00 395.50 629.00\" width=\"396pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 625)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-625 391.5,-625 391.5,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 120957321168 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>120957321168</title>\n",
       "<polygon fill=\"none\" points=\"33.5,-584.5 33.5,-620.5 155.5,-620.5 155.5,-584.5 33.5,-584.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"94.5\" y=\"-598.8\">Books: InputLayer</text>\n",
       "</g>\n",
       "<!-- 120957320944 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>120957320944</title>\n",
       "<polygon fill=\"none\" points=\"0,-511.5 0,-547.5 189,-547.5 189,-511.5 0,-511.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"94.5\" y=\"-525.8\">BooksEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 120957321168&#45;&gt;120957320944 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>120957321168-&gt;120957320944</title>\n",
       "<path d=\"M94.5,-584.4551C94.5,-576.3828 94.5,-566.6764 94.5,-557.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"98.0001,-557.5903 94.5,-547.5904 91.0001,-557.5904 98.0001,-557.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120957395856 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>120957395856</title>\n",
       "<polygon fill=\"none\" points=\"238.5,-584.5 238.5,-620.5 356.5,-620.5 356.5,-584.5 238.5,-584.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-598.8\">Users: InputLayer</text>\n",
       "</g>\n",
       "<!-- 120957395408 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>120957395408</title>\n",
       "<polygon fill=\"none\" points=\"207.5,-511.5 207.5,-547.5 387.5,-547.5 387.5,-511.5 207.5,-511.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-525.8\">UsesEmbedding: Embedding</text>\n",
       "</g>\n",
       "<!-- 120957395856&#45;&gt;120957395408 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>120957395856-&gt;120957395408</title>\n",
       "<path d=\"M297.5,-584.4551C297.5,-576.3828 297.5,-566.6764 297.5,-557.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"301.0001,-557.5903 297.5,-547.5904 294.0001,-557.5904 301.0001,-557.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120957396304 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>120957396304</title>\n",
       "<polygon fill=\"none\" points=\"50,-438.5 50,-474.5 187,-474.5 187,-438.5 50,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"118.5\" y=\"-452.8\">FlattenBooks: Flatten</text>\n",
       "</g>\n",
       "<!-- 120957320944&#45;&gt;120957396304 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>120957320944-&gt;120957396304</title>\n",
       "<path d=\"M100.4326,-511.4551C103.1442,-503.2074 106.4166,-493.2536 109.4273,-484.0962\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"112.7541,-485.1833 112.5525,-474.5904 106.1043,-482.997 112.7541,-485.1833\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120957395632 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>120957395632</title>\n",
       "<polygon fill=\"none\" points=\"217.5,-438.5 217.5,-474.5 351.5,-474.5 351.5,-438.5 217.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"284.5\" y=\"-452.8\">FlattenUsers: Flatten</text>\n",
       "</g>\n",
       "<!-- 120957395408&#45;&gt;120957395632 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>120957395408-&gt;120957395632</title>\n",
       "<path d=\"M294.2865,-511.4551C292.849,-503.3828 291.1205,-493.6764 289.5187,-484.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"292.9207,-483.8218 287.7216,-474.5904 286.0291,-485.0491 292.9207,-483.8218\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120260242064 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>120260242064</title>\n",
       "<polygon fill=\"none\" points=\"58.5,-365.5 58.5,-401.5 186.5,-401.5 186.5,-365.5 58.5,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"122.5\" y=\"-379.8\">dropout_1: Dropout</text>\n",
       "</g>\n",
       "<!-- 120957396304&#45;&gt;120260242064 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>120957396304-&gt;120260242064</title>\n",
       "<path d=\"M119.4888,-438.4551C119.9311,-430.3828 120.4629,-420.6764 120.9558,-411.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"124.4563,-411.7669 121.5087,-401.5904 117.4668,-411.3839 124.4563,-411.7669\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120957396080 -->\n",
       "<g class=\"node\" id=\"node8\">\n",
       "<title>120957396080</title>\n",
       "<polygon fill=\"none\" points=\"212.5,-365.5 212.5,-401.5 340.5,-401.5 340.5,-365.5 212.5,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"276.5\" y=\"-379.8\">dropout_2: Dropout</text>\n",
       "</g>\n",
       "<!-- 120957395632&#45;&gt;120957396080 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>120957395632-&gt;120957396080</title>\n",
       "<path d=\"M282.5225,-438.4551C281.6378,-430.3828 280.5741,-420.6764 279.5884,-411.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"283.0511,-411.1495 278.4825,-401.5904 276.0928,-411.9122 283.0511,-411.1495\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120906158768 -->\n",
       "<g class=\"node\" id=\"node9\">\n",
       "<title>120906158768</title>\n",
       "<polygon fill=\"none\" points=\"113,-292.5 113,-328.5 278,-328.5 278,-292.5 113,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-306.8\">ConcatLayer: Concatenate</text>\n",
       "</g>\n",
       "<!-- 120260242064&#45;&gt;120906158768 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "<title>120260242064-&gt;120906158768</title>\n",
       "<path d=\"M140.5449,-365.4551C149.4946,-356.5054 160.453,-345.547 170.2231,-335.7769\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"172.8134,-338.1363 177.4096,-328.5904 167.8637,-333.1866 172.8134,-338.1363\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120957396080&#45;&gt;120906158768 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "<title>120957396080-&gt;120906158768</title>\n",
       "<path d=\"M256.4775,-365.4551C246.4498,-356.4177 234.1491,-345.3319 223.2286,-335.4899\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"225.3444,-332.6852 215.5729,-328.5904 220.6581,-337.885 225.3444,-332.6852\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120888672496 -->\n",
       "<g class=\"node\" id=\"node10\">\n",
       "<title>120888672496</title>\n",
       "<polygon fill=\"none\" points=\"115.5,-219.5 115.5,-255.5 275.5,-255.5 275.5,-219.5 115.5,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-233.8\">FullyConnected-1: Dense</text>\n",
       "</g>\n",
       "<!-- 120906158768&#45;&gt;120888672496 -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "<title>120906158768-&gt;120888672496</title>\n",
       "<path d=\"M195.5,-292.4551C195.5,-284.3828 195.5,-274.6764 195.5,-265.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"199.0001,-265.5903 195.5,-255.5904 192.0001,-265.5904 199.0001,-265.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120854580976 -->\n",
       "<g class=\"node\" id=\"node11\">\n",
       "<title>120854580976</title>\n",
       "<polygon fill=\"none\" points=\"115.5,-146.5 115.5,-182.5 275.5,-182.5 275.5,-146.5 115.5,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-160.8\">FullyConnected-2: Dense</text>\n",
       "</g>\n",
       "<!-- 120888672496&#45;&gt;120854580976 -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "<title>120888672496-&gt;120854580976</title>\n",
       "<path d=\"M195.5,-219.4551C195.5,-211.3828 195.5,-201.6764 195.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"199.0001,-192.5903 195.5,-182.5904 192.0001,-192.5904 199.0001,-192.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120837599184 -->\n",
       "<g class=\"node\" id=\"node12\">\n",
       "<title>120837599184</title>\n",
       "<polygon fill=\"none\" points=\"115.5,-73.5 115.5,-109.5 275.5,-109.5 275.5,-73.5 115.5,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-87.8\">FullyConnected-3: Dense</text>\n",
       "</g>\n",
       "<!-- 120854580976&#45;&gt;120837599184 -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "<title>120854580976-&gt;120837599184</title>\n",
       "<path d=\"M195.5,-146.4551C195.5,-138.3828 195.5,-128.6764 195.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"199.0001,-119.5903 195.5,-109.5904 192.0001,-119.5904 199.0001,-119.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 120837595712 -->\n",
       "<g class=\"node\" id=\"node13\">\n",
       "<title>120837595712</title>\n",
       "<polygon fill=\"none\" points=\"136.5,-.5 136.5,-36.5 254.5,-36.5 254.5,-.5 136.5,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-14.8\">Activation: Dense</text>\n",
       "</g>\n",
       "<!-- 120837599184&#45;&gt;120837595712 -->\n",
       "<g class=\"edge\" id=\"edge12\">\n",
       "<title>120837599184-&gt;120837595712</title>\n",
       "<path d=\"M195.5,-73.4551C195.5,-65.3828 195.5,-55.6764 195.5,-46.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"199.0001,-46.5903 195.5,-36.5904 192.0001,-46.5904 199.0001,-46.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_model(neuralNetModel, to_file='neuralNetModel.png', show_shapes=True)\n",
    "SVG(model_to_dot(neuralNetModel).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "callbacks = [keras.callbacks.EarlyStopping('val_loss', patience=3), \n",
    "             keras.callbacks.ModelCheckpoint('neuralNetModelWeights.h5', save_best_only=True)]\n",
    "\n",
    "trainingHistory = neuralNetModel.fit([trainDataFrame.userIDX, trainDataFrame.bookIDX], \n",
    "                                     trainDataFrame.impressionRating, \n",
    "                                     validation_split=.1, \n",
    "                                     callbacks=callbacks,\n",
    "                                     batch_size=NN_BATCH_SIZE,\n",
    "                                     epochs=NN_TRAIN_EPOCHS, \n",
    "                                     verbose=TRAIN_VERBOSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(trainingHistory.history['loss']).plot(logy=True, label=\"Train Loss\")\n",
    "pd.Series(trainingHistory.history['val_loss']).plot(logy=True, label=\"Val Loss\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Training Error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make some preditions\n",
    "neuralNetRatingsPredictions = neuralNetModel.predict([testDataFrame.userIDX, \n",
    "                                                      testDataFrame.bookIDX])\n",
    "neuralNetRatingsPredictions = np.rint(neuralNetRatingsPredictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "displaySideBySide(pd.DataFrame(neuralNetRatingsPredictions).head(30), \n",
    "                  pd.DataFrame(actualImpressionRatings).head(30))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# How did we do?\n",
    "print(\"MAE {0:2.2f}\".format(mean_absolute_error(actualImpressionRatings, neuralNetRatingsPredictions)))\n",
    "print(\"RMSE {0:2.2f}\".format(sqrt(mean_squared_error(actualImpressionRatings, neuralNetRatingsPredictions))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ---------END OF NOTEBOOK--------------"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}