ch09_Recommenders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8: Recommender Systems.\n",
    "\n",
    "In this notebook we will see how to create a recommender system. To\n",
    "this end we will explore different properties of recommender systems, develope and evaluate a collaborative recommender system.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pylab as plt\n",
    "\n",
    "%matplotlib inline \n",
    "plt.style.use('seaborn-whitegrid')\n",
    "plt.rc('text', usetex=True)\n",
    "plt.rc('font', family='times')\n",
    "plt.rc('xtick', labelsize=10) \n",
    "plt.rc('ytick', labelsize=10) \n",
    "plt.rc('font', size=12) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data description\n",
    "\n",
    "We will work with the well known MovieLens dataset (http://grouplens.org/datasets/movielens/). This dataset was initially constructed to support participants in the Netflix Prize. Today, we can find several versions of this dataset with different amout of data, from 100k samples version to 20m sample version.\n",
    "Although performance on bigger dataset is expected to be better, we will work with the smallest dataset: MovieLens *100K Dataset (ml-100k-zip)*. Working with this lite version has the benefit of less computational costs, while we will also get the skills required on user-based recommender systems.\n",
    "\n",
    "With a unix machine the dataset can be downloaded with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zsh:1: command not found: wget\n",
      "unzip:  cannot find or open ml-100k.zip, ml-100k.zip.zip or ml-100k.zip.ZIP.\n"
     ]
    }
   ],
   "source": [
    "!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip \n",
    "!unzip ml-100k.zip -d \"files/ch09/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are working with a windows machine, please go to the website and download the 100k version and extract it to the subdirectory named \"files/ch8/ml-100k/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have downloaded and unzipped the file into a directory, you can create a DataFrame with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The database has 100000 observed ratings\n",
      "The database has  943  users\n",
      "The database has  1682  movies\n",
      "   user_id         title  movie_id  rating\n",
      "0      196  Kolya (1996)       242       3\n",
      "1      305  Kolya (1996)       242       5\n",
      "2        6  Kolya (1996)       242       4\n",
      "3      234  Kolya (1996)       242       4\n",
      "4       63  Kolya (1996)       242       3\n"
     ]
    }
   ],
   "source": [
    "#NETFLIX REAL 50.000.000 usuaris and 100.000 items\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "import matplotlib.pylab as plt\n",
    "from math import isnan\n",
    "from tqdm import tqdm # conda install -y tqdm\n",
    "\n",
    "\n",
    "# Load Data set\n",
    "u_cols = [\n",
    "    'user_id', 'age', 'sex', \n",
    "    'occupation', 'zip_code']\n",
    "\n",
    "users = pd.read_csv('files/ch09/ml-100k/u.user', \n",
    "                    sep='|', \n",
    "                    names=u_cols)\n",
    "\n",
    "r_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']\n",
    "ratings = pd.read_csv('files/ch09/ml-100k/u.data', sep='\\t', names=r_cols)\n",
    "\n",
    "# We will only load the first three columns of the filewith usecols\n",
    "m_cols = ['movie_id', 'title', 'release_date']\n",
    "movies = pd.read_csv('files/ch09/ml-100k/u.item', sep='|', names=m_cols, usecols=range(3), encoding='latin-1')\n",
    "\n",
    "# Construcció del DataFrame\n",
    "data = pd.merge(pd.merge(ratings, users), movies)\n",
    "data = data[['user_id','title', 'movie_id','rating']]\n",
    "print(\"The database has \"+ str(data.shape[0]) +\" observed ratings\")\n",
    "print(\"The database has \", data.user_id.nunique(),\" users\")\n",
    "print(\"The database has \", data.movie_id.nunique(), \" movies\")\n",
    "print(data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you explore the dataset in detail, you will see that it  consists of:\n",
    "<ul>\n",
    "<li>1,000,209 ratings from 6040 users of 3706 movies. Ratings are from 1 to 5.</li>\n",
    "<li>Each user has rated at least 20 movies.</li>\n",
    "<li>Simple demographic info for the users (age, gender, occupation, zip)</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Collaborative Filtering (CF) for Movilens dataset\n",
    "The idea behind any recommender system is to suggest or recommend items that are likely to be of interest to the user. If we think on the movielens dataset and a movie recommender system, the goal will consist on recommend those useen movies by a given \"user\" that are more likely to be intersting to him or her. So, the problem can be taken by predicting the ratings for the unseen movies of the given \"user\" and recommend those with the highest predicted rating.\n",
    "\n",
    "<br>The following table ilustrates the problem. This toy dataset consists on 4 users and 4 items.  Users 1, 2 and 3 have seen all the movies while user 3 has only seen Superman and Star Wars. So, the problem consist on prediction which movie, Matrix or Spiderman, will be more likely for the user 4.\n",
    "<table style=\"width:60%\">\n",
    "  <tr>\n",
    "    <td></td>\n",
    "    <td>Superman</td> \n",
    "    <td>Star Wars 1</td>\n",
    "    <td>Matrix</td>\n",
    "    <td>Spiderman</td>\n",
    "    \n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>user1</td>\n",
    "    <td>3</td> \n",
    "    <td>3.5</td>\n",
    "    <td>4.5</td>\n",
    "    <td>3</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>user2</td>\n",
    "    <td>3.5</td> \n",
    "    <td>4</td>\n",
    "    <td>5</td>\n",
    "    <td>5</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>user3</td>\n",
    "    <td>3</td> \n",
    "    <td>4</td>\n",
    "    <td>4.5</td>\n",
    "    <td>3</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>user4</td>\n",
    "    <td>3.5</td> \n",
    "    <td>5</td>\n",
    "    <td><font color=\"red\"><b>¿?</b></font></td>\n",
    "    <td><font color=\"red\"><b>¿?</b></font></td>\n",
    "  </tr>\n",
    "</table>\n",
    "<br><br><br><br>\n",
    "\n",
    "\n",
    "In order to create a collaborative recommender system we will have to define: 1) a prediction function, 2) a user-similarity function and 3) an evaluation function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction function:\n",
    "The prediction function behind the CF will be based on the movie ratings from similar users.\n",
    "So, in order to recommend a movie, $p$, from a set of movies, $P$, to a given user, $a$, we first need to see the set of users, $B$, who have already seen $p$. Then, we need to see the taste similarity between these users in $B$ and user $a$. The most simple prediction function for a user $a$ and movie $p$ can be defined as follows:\n",
    "\n",
    "$$pred(a,p) = \\frac{\\sum_{b \\in B}{sim(a,b)*(r_{b,p})}}{\\sum_{b \\in B}{sim(a,b)}}$$\n",
    "\n",
    "where $sim(a,b)$ is the similarity between user $a$ and user $b$,  $B$ is the set of users in the dataset that have already seen $p$ and $r_{b,p}$ is the rating of $p$ by $b$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Users similarity</h3>\n",
    "The computation of the similarity between items is one of the most critical steps in the CF algorithms. The basic idea behind the similarity computation between two users $a$ and $b$, is that we can first isolate the set $P$ of items rated by both users, and then apply a similarity computation technique to determine the similarity.\n",
    "\n",
    "The set of *common_movies* can be obtained with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Number of common movies 96 \n",
      "\n",
      "                                    title  rating\n",
      "14                           Kolya (1996)       5\n",
      "940                Raising Arizona (1987)       4\n",
      "1306  Truth About Cats & Dogs, The (1996)       5\n",
      "2677                          Babe (1995)       1\n",
      "3073   Four Weddings and a Funeral (1994)       3\n",
      "                                    title  rating\n",
      "2                            Kolya (1996)       4\n",
      "885                Raising Arizona (1987)       5\n",
      "1255  Truth About Cats & Dogs, The (1996)       2\n",
      "2636                          Babe (1995)       4\n",
      "3022   Four Weddings and a Funeral (1994)       3\n"
     ]
    }
   ],
   "source": [
    "# dataframe with the data from user 1\n",
    "data_user_1 = data[data.user_id==1]\n",
    "\n",
    "# dataframe with the data from user 2\n",
    "data_user_2 = data[data.user_id==6]\n",
    "\n",
    "# We first compute the set of common movies\n",
    "common_movies = set(data_user_1.movie_id).intersection(data_user_2.movie_id)\n",
    "print(\"\\nNumber of common movies\",\n",
    "      len(common_movies),'\\n')\n",
    "\n",
    "# Sub-dataframe with only the common movies \n",
    "mask = (data_user_1.movie_id.isin(common_movies))\n",
    "data_user_1 = data_user_1[mask]\n",
    "print(data_user_1[['title','rating']].head())\n",
    "\n",
    "mask = (data_user_2.movie_id.isin(common_movies))\n",
    "data_user_2 = data_user_2[mask]\n",
    "print(data_user_2[['title','rating']].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the set of ratings for all movies common to the two users has been obtained, we can compute the user similarity. These are some of the most common similarity functions used in CF methods: \n",
    "\n",
    "   <ul>\n",
    "    <li>Euclidean distance</li>\n",
    "    $$sim(a,b) = \\frac{1}{1+\\sqrt{\\sum_{p \\in P}{(r_{a,p} - r_{b,p})^2}}}$$\n",
    "    <br>\n",
    "    <li>Pearson Correlation</li>\n",
    "    $$sim(a,b) = \\frac{\\sum_{p\\in P} (r_{a,p}-\\bar{r_a})(r_{b,p}-\\bar{r_b})}{\\sqrt{\\sum_{p \\in P}(r_{a,p}-\\bar{r_a})²}\\sqrt{\\sum_{p \\in P}(r_{b,p}-\\bar{r_b})²}}$$\n",
    "    <br>\n",
    "    <li>Cosine distance</li>\n",
    "    $$ sim(a,b) = \\frac{\\vec{a}· \\vec{b}}{|\\vec{a}| * |\\vec{b}|}$$\n",
    "    <br>\n",
    "    </ul>\n",
    "  \n",
    "<br>\n",
    "Where: \n",
    "\n",
    "* $sim(a,b)$ is the similarity between user \"a\" and user \"b\"\n",
    "* $P$ is the set of common rated movies by user \"a\" and \"b\"\n",
    "* $r_{a,p}$ is the rating of movie \"p\" by user \"a\"\n",
    "* $\\bar{r_a}$ is the mean rating given by user \"a\"\n",
    "\n",
    "<br>\n",
    "\n",
    "<h4>Some issues to take into accout</h4>\n",
    "<ul>\n",
    "<li>Pearson Correlation used to work better than euclidean distance since it is based more on the ranking than on the values.</li>\n",
    "<li>Cosine distance is usually used when our data is binary/unary, i.e. like vs. not like  or buy vs. not buy.</li>\n",
    "<li>What happens if two users have very few items in common?</li>\n",
    "</ul>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us define a function to compute the user's similarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from scipy.stats import pearsonr\n",
    "from scipy.spatial.distance import euclidean\n",
    "\n",
    "# Returns a user-based similarity score for two users\n",
    "# Person correlation is set as the default\n",
    "def user_sim(df, user_1, user_2, \n",
    "                    min_common_items = 3, \n",
    "                    method = 'pearson'):\n",
    "    # GET MOVIES OF USER1\n",
    "    mov_usr1 = df[df['user_id'] == user_1 ]\n",
    "    # GET MOVIES OF USER2\n",
    "    mov_usr2 = df[df['user_id'] == user_2 ]\n",
    "    \n",
    "    # FIND SHARED FILMS\n",
    "    df_shared = pd.merge(mov_usr1, mov_usr2, on='movie_id')  \n",
    "    \n",
    "    # If there is no enough common items to comput similarity\n",
    "    if(df_shared.shape[0]<=min_common_items):\n",
    "        return 0\n",
    "    \n",
    "    if method =='pearson':\n",
    "        res = pearsonr(df_shared['rating_x'],df_shared['rating_y'])[0]\n",
    "        if(np.isnan(res)):\n",
    "            return 0\n",
    "        return res\n",
    "    \n",
    "    elif method =='euclidean':\n",
    "        return 1.0/(1.0+euclidean(df_shared['rating_x'],\n",
    "                                  df_shared['rating_y'])) \n",
    "    else:\n",
    "        print(\"method not defined\")\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how similars are user 1 with 8 and 1 with 31"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Euclidean Similarity 0.1566130288262323\n",
      "Pearson Similarity 0.6920863660773593\n"
     ]
    }
   ],
   "source": [
    "print(\"Euclidean Similarity\",\n",
    "      user_sim(data,1,8,method = 'euclidean') )\n",
    "print(\"Pearson Similarity\",user_sim(data,1,8) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Euclidean Similarity 0.1757340838011157\n",
      "Pearson Similarity -0.09221388919541466\n"
     ]
    }
   ],
   "source": [
    "print(\"Euclidean Similarity\",\n",
    "      user_sim(data,1,31, method = 'euclidean') )\n",
    "print(\"Pearson Similarity\",user_sim(data,1,31) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAECCAYAAADuGCyPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAegklEQVR4nO3deXRT94Ev8K8Wb4AlA2kSAgKyldiyyWJMYtG00yxgkkw7EQXR6TQvJja8vpnavBeYN+eUQJN25syMzevY82aaIFNnaScVaZzpMkEhcfqyIGUzJLElBbIMtszihGDpymDZWu77w2MZY1vIy9W9XH0/5+QEZKP75Vp89dPv3vu7GlEURRAR0SVPK3cAIiKaGSx0IiKVYKETEakEC52ISCVY6EREKqGXa8NtbW1ybZqI6JJWWlo67uOyFTowcah08vl8KCwslDuGInBfjOC+GMF9MUIJ+yLZYJhTLkREKsFCJyJSCRY6EZFKsNCJiFSChU5EpBIsdCIilWChExGpBAudiEglWOhERCrBQiciUglZL/0nIlKywWgcL3t78NFJAV+eHUS4L4Ci05/hvuVX4UpjrtzxxmChExFd4ESgH0+7j+FXb3chLoo4OxBLfO0/jvah7qUjuPXqedjyjWthuXY+NBqNjGlHsNCJaFwnAv04c3YQ3YFBLB2MIS9bJ3ektHjZ24OaZw8jFo9jMDb2lssD0TgA4PWPT+O9zl58c9nl+JntJmTr5Z/BZqETUUI4EsPvPziBx1/7FN29/cjSaRGLxYAXT+L+mxdi09euxnWXz5E7pmR+/8EJbP/NBwhH4il9/7nBGFo/6sGDze/g6U0rodfJW+ryv6UQkSJ8LoSxtuEN7PqdB59+cRYD0Tj6BqLoj4roj8Sw7z0/7mt8A796u1PuqJL4wB+YVJkPC0fiONwVwCO/7ZAoWepY6ESEYH8E1p+70HXmHM4Nxsb9nmhcRDgax0//4MXzbd1pTii9f3B+NOkyH9YfieH5Q8fRI4RnONXksNCJzjMQjeFQVy9OCBG5o6TVP7/6MT4XwojFx84ZX6g/EseP/r0dfQPRNCRLj+7ec2jr7J3Wc2gA/PIteT+9SFrodrsdVqsVVqsVgiBIuSmiaXvZ24PSn7yCB/a+gx/8rhvrH3dBCKu/2AeiMTz7Tte4BwAnotVo8MIh9YzSn3F3Ii6m/vcfz0A0jqfdnYjEpjbKnwmSFnogEEBLSwtaWlpgMBik3BTRtHwuhPHDZw+hbyCKvoEoBmMi3vcH8KMX2uWOJrk/fvT5pP/MucEYmg8em/kwMjng7UFkEm9oE4nG4/j0i74ZSDQ1khW63++H1+vFsmXL4HQ6pdoM0Yx4ydsz5rFITMT+9lMQpzlyU7ru3n4MRic/quwJyTtfPJNCM/RJTKvRIHBOvk91kp22aDKZ0NzcDI/HgwcffBAWi2XMKN3n80m1+ZSFw2FF5FCCTN4Xp04KEMeZPxYhwufzKebCESn09ASm9KYVj8dV83qJRmfmeEA8FkNnZyeMA5P/1DMTJD8P3Ww2Y+3atfD7/TCbzaO+JvfdswFl3MVbKTJ5X8xfGMbeQ38EzvvYnaXToKL4ShQVFcmYTHqlsZN4tj2IvoHxz26ZyIKCWap5vcyd04NA+Oz0n0irxY2F1+GGK6WbYm5ra5t485Jt9QIXljmRklxuyMU/f/cWzMnRY06OHtk6DW42zcXf3l8idzTJffOGyyFicp9AZmXrsGnVUmkCyWB10RXI1k3/U5heq8V1X5HvwivJRugOhwMejwcVFRWw2WxSbYZoxtxddAXaHrkL3hMCzpz0485bl8sdKS1y9Dp8d6UJT7uOpXymS1wU8Wc3L5I4Wfp8v3wJnnQdAzD14yU5ei0eKF8i69WikhU6S5wuRTl6HW5ePBe+s6fkjpJWNXdej5c6TuFkMIzoRc5Fz83S4u/uL8GcHPWsHLJo7iyULpkL16dfTvk5RAB/cduSmQs1BbywiIhgyM3C8z+wwDRvFmZNsAiXXqtBbpYWO+8zw3qLekbnw/53xQ3IzZpaJeZlafGd0kW4wiDvkrosdCICMHQcYX/t7fjJt4tx/eVzkKPXYk6OHnl6DfKydLCVmfAfNbfjz29dLHdUSdxoKkD9+hsnXeq5WVrcsmQeHvuW/McJ1fOZiYimLTdLh3Wli7CudBFOBoeWzz3edQxfLy1Gbpb6l8+9b/lVyNXr8MNnDyMWFzGY5KpPDYC8bB3uuGFo+Vy5V1oEWOhENIEFxjwsMOZBG8zOiDIfdlfRFXh12zfwjLsTz7zVOeYGFzn/te55+TXzsfnr16CcN7ggIlKuBcY8/HXFDdh611eHbkF3augWdAN9QRRdvRD3lizgLeiIiC4l2Xot7l2+APcuXwBg+OK7q2VONTH5J32IiGhGsNCJiFSChU5EpBIsdCIilWChExGpBAudiEglWOhERCrBQiciUgkWOhGRSrDQiYhUgoVORKQSLHQiIpVgoRMRqQQLnYhIJVjoREQqwUInIlIJFjoRkUqw0ImIVIKFTkSkEix0IiKVYKETEakEC52ISCVY6EREKsFCJyJSCRY6EZFK6OUOQETKdHYgit5zg+jtjyIeF6HVauSORBfBQieiBFEU4fr0Szz+2qd467MvoddqEY3HYXzxFB762tX47srFKJiVLXfMtAqFI3jJ04MeIYzgmSD080K4/op8uWONi4VORACAWFzEX//mA+zvOIVzgzEAQCQ29P/TfYNoaP0YT7z+GZ6tvg2FCwxyRk2LgWgMP/m9F7851A2tRoNwJAadBnj68Ju4/op81K+/EcuuVFaxcw6diAAAu37nwYvtJxNlfqFwJI7AuQhsT7hxPNCf5nTpFYnF8RdNb+M3h7oRjsRxbjCGuAhE4kA4GkfH8SCs/3oQnhNBuaOOwkLPcOFIDE1vfIZv/8tB7HzlJFyfnpY7EsnAf+YcnnvPj/5I/KLfe3YgisbWj9OQSj72Nz5D+/EgwhPsDxHA2cEYqp96D6IopjdcEpIXut/vR01NjdSboSkQRREP7H0b9QeO4AN/AO8e78dDT76HFw51yx2N0uwp9zHEUyymmAj89v3j6BuISpxKHrG4iL1v/OeEZX6+QH8EBz/5Mg2pUiN5oTudTqk3QVP07rFedJwQRr1w+yMx/PRFn6JGHSS9/e2nEIml/jPXa7V49z/PSJhIPp4TQYQj4087XejcYAwvHFbOAEjSg6JOpxMVFRVob28f9+s+n0/KzackHA4rIoccXvtIQDQ2dhTSe3YQ73d4kavP3Bm5THtdhPoHJvX9sXgMH33aiStF5YxOZ0r7iXMQxYuPzod19pxRzGtFskL3eDwwm80wGo0Tfk9hYaFUm0+Zz+dTRA45nM07g+bDvRiMjR6NzJ+dg5uKi6DRZO55x5n2usjPOwlhIPUDnTqtDjdctwSFyy6XMJU8ovlBaF77AkBqo/SlV85P62ulra1twq9JNgSrr6+H3W7Hjh074Ha74XA4pNoUTVHpkrm4yVSAvKyRl0Felg4778vsMs9E9y1fgGxd6j/zWFzEyqXzJEwkH/NVBuRl61L63lnZOlhvWShxotRJNkJvbm4GAAiCgB07dsBms0m1KZoijUaDJytXwvFuF37/wQlkxQfwP++5EStU+g+VJvb98iV40nUMQ+dvJKfXavBnNy/E7Bx1Xsai1WpQffs1+NkrR5MeGNUAmDc7G+XXzE9fuIvI3ElSAgBk67X4fvlS7PvvFuz45pUs8wy1aO4sfO/WxcjLSj4y1QCYk6tH7Z3XpyeYTB762tW4ZfFc5GaNX5EaDTA7R4+9/61MUZ9mJS90g8GAxsZGqTdDRNO0494i3H/zQuRl6TBeR+Vl6TBvdjae21KOK4256Q+YRnqdFk9tWonv3boEeVk6zM7WIUunQbZOgxy9Frcsnot//8tVirtSVJ2fmYho0rRaDf7OWgLrLQvxxOuf4Y8ffQ6dVoNYPI7L8nOx5fZrsG7FIhhys+SOmhZZOi0eua8ID6/+Kl7xfY6eYBiBL7/A+tuLsfSy2XLHGxcLnYhGWbF0HlYsnYeBaAxCfxSdn32C0uWZe6B8VrYe37rxKgCAzzeg2DIHOIdORBPI0evwlfwczM7WZmyZX2pY6EREKsFCJyJSCRY6EZFKTLrQ+/r60N2tnMVoiIhoSNJC37dvH+6++25s3boVwNDl/HfccQd27dqFvXv3piMfERGl6KKnLb788svw+/3YunUr3G43WltbkZ+fj3379qUjHxERpSjpCP38U5WcTic2b96M/PyhK6P8fr+0yYiIaFKSjtCDwSA2bdoEj8eDhoYGBINB1NbWoq+vD0VFRenKSEREKUha6FVVVVizZg0KCgoSI/Py8nJ0d3ejvLw8LQGJiCg1F51DN5lMY35/4WNERCQ/nodORKQSKRX66tWree45EZHCpVToGzZsQEFBwajH3G63FHmIiGiKUlo+98UXX8Tu3bthMBgAAKIoIhQKKeZO10RElGKhb9y4EeXl5aNG6fv375cqExERTUFKUy7l5eVoampCR0cH8vPz4XQ6cc8990idjYiIJiGlQt+1axfy8/MTB0bXr1+P2tpaSYMREdHkpFToFosF27ZtgyiKicc6OjokC0VERJOXUqEbDAYcOHAAoVAIbrcbVqsVZrNZ6mxERDQJKZ+2KIoiOjs78etf/xr33HMPfvGLX0idjYiIJiGls1wAYM2aNVizZo2UWYiIaBpSPig6fM75unXrsG7dOjz33HOSBiMioslJqdDNZjMKCwuxe/duGI1GPP/881i0aJHU2YiIaBJSKnRBENDU1ASHw4HGxkYAQze8ICIi5Uip0KuqqmA2m9Ha2oo5c+bgpZdegsVikTobERFNQtKDon19fZgzZw4AjLqhBQ+OEhEpT9JC37FjB2w225iVFgsLC6XMREREU5B0ykUQBABAIBAY9d/u3bu5PjoRkcIkHaE3NDQk7iV6vvLycuzevRsPP/ywZMGIiGhyko7QxytzIiJSpqSFnmxaxe/3z3gYIiKauqRTLrt27YLFYkncqQgYmld3uVyX/IVFA9EYPuwO4vMzA7hBFKHRaOSOJJtwZGhffNk7mPH7gkacHYii/XgQoeAgMv00iFA4As8JAWeFiKL3RdJCP3jwIHp7e0c9VlBQAIvFgqqqqos+ud1uR3t7O0pKSlBdXT29pDPozY9P4we/aoMoAtFYDP/0dhC/rLoVVxhy5Y6Wdq9+1IMfPnsYGmgQicZw7bsCnnloJebPyZE7Gsmo5VA3fvRCB/RaDQaiMdz0/jnsfXAF8nOz5I6Wdv/2dice+4MXWTotBiIxrGw/B/sDZcjL1skdbQyNeP4i5xdwu92jzj+fDI/Hk1hi12q1oqWlZdTX29raUFpaOqXnno5QOIKVf9uK/kgs8ZhOA6xYOg+OLVP7u16qzpwdhOXvWxGOxBOP6bUa3H79ZWiuXCljMvn5fL6MPT2368tzWP2z1xCOjrwusnUafPumhahbf6OMydLv454Q/vT/vjnq30i2Xos/X2nCj79VLEumZN2ZdA59qmUOIFHmHo8HNpttys8z0147+gW0F/ytYyLQ1tmLUDgiTyiZtPp6oL1geiUaF/HGx6cxeN4/Zsos+ztOInbBOG8wJuIPH56UKZF8/vDhSURio/8tDEbjeOHwCZkSJZfy8rlTIQgCnnjiCXi93nFLfXgFx3Q6cbwP8fh4ZSXi6NGjmJWV0moIqnDqVGjcfSGKIo4c+Qh6bebOpYfDYVlen0pw+osAMM4Hd1GMZ9w+OX36DDDOHIYYjylyX0ha6AaDAY2NjaipqRk1BTNMjo+0i6+JouGtVwCMTLnotRpYrp2P0uWZdRemq5ZE8PN3WnH+vsjSaXDHDZejxFwkXzAFyOQplwcX9OOZD/4fIue92efotfhO6aKM2ydVXzmL572vI3belEuuXovv3rpUtn3R1tY24dfSMhw1mUwwmUzp2NRFzc7R48nKlZg/Oxuzs3XI1mlQssiIn9lukjta2hlnZaHpgRUomJWV2Bc3m+biH9dl1jwpjbawIA8NtpswJ0eP2TlDr4tV116GH92beW/yV182G/9gXY7Z2TrMydEjS6fBnyz7Cv7X6q/KHW1cSQ+KDpvo4OiBAwdQVFQ07imMTqcTLpcrcdrjhaszynVQdFgsLuLIqRBOdR/DHSuXy5ZDCaKxOI70hPDF8U78SVlm74thmTxCHzYYjeNoTwhnTnbh6ytK5I4jq3Akhk8+70PgVBe+VirvvkjWnSlNuRw8eBBNTU0AgG3btqGwsBC7du1CRUUFvF4vCgoKEqsyDquoqEBFRcU0o0tHp9Wg6CoDNMHMOw3rQnqdFuarjPBxX9B5svVaFC80widIOjN7ScjN0l0S+yKlKRev14sNGzbgxz/+MTo6OgAA7e3tMJlMKC8vh8PhkDQkERFdXEqFXlFRgTVr1sBkMqGrqwvA0Frpw8vqtre3SxaQiIhSk/JB0bvvvjsxp9jU1IRAIACXy4U9e/aguFieE+yJiGhEShNCGzZswNq1awGMrMBYVVUFv9+PN998E1u2bJEuIRERpSTlEXp+fv6o5XTdbjdMJhMee+yxMQdEiYgo/VIaoTc1NWHPnj0IhUIAhq4k1Gg0irxSiogoU6VU6C6XC62trWNG6EREpBwpTbls3LhxzBrZF944moiI5JXSCP3xxx9HbW1t4kYXoigiFApxyoWISEFSKvQtW7bAYrEkplwEQeCUCxGRwqRU6KtWrQIwdDERMFTogiBIl4qIiCYtaaGvXr0aLS0tqKmpQUdHBwoKCiCKIoLBIEKhENavX5+unEREdBFJC/3AgQMAgOrq6jGrLb7zzjvSpSIioklL6SyX7u7uMY+98cYbMx6GiIimLukI3e/3Y9euXfB4PLDb7YnHA4EATCYTHn74YckDEhFRapIWuslkwqOPPjruDS7mzp0raTAiIpqci57lMnz7uOH59GEOhwN79+6VLBgREU1OSqct1tbWwmg04q233sJtt92GYDCIoqLMu78gEZGSpVToGzduRHl5Ofbt2weLxYJFixZh9+7dUmcjIqJJSPmeok6nE48++ig2bdoEk8kEt9vNg6JERAqSUqFv27YNXq8XANDQ0ACHwwGbzSZpMCIimpyUb3AxPGeen5+PqqoqBINByUIREdHkXbTQ3W43nnvuuVGP+Xw+bN26VapMREQ0BUkLvampCbW1tairq8NDDz0EANi3bx/uv/9+VFdXpyUgERGlJukcent7e2LNltraWmzduhUulwstLS08bZGISGGSjtBLSkoSv964cSMEQcCrr76KoqKicdd3ISIi+SQdobvdbhiNRgBAV1cXTCYT9u/fDwBwOp28UpSISEGSFvrBgwfR29s76rH29nYAwPHjx6VLRUREk5a00Jubm8csyjWMt6AjIlKWpHPoE5X5xb5GRETpl/KFRUREpGwsdCIilWChExGpBAudiEglWOhERCqR0vK5ajMQjeEp1zH89v0T0MUGUYt5uLPwCrljERFNi2QjdEEQUFNTg7KyMtTV1Um1mUkTRRGVze/i/7x8FJ4TAj7sCeOv/u0wnnIdkzsaEdG0SFboLpcLjY2NePfdd7Fv3z74/X6pNjUpH3QHcbgrgHAknnisPxJD/YEjiMbiSf4kEZGySTblUlFRkfh1cXExTCbTmO/x+XxSbX5Cr38SgiiOLe7wYBTvfeiFMVeX9kxKEQ6HZfmZKBH3xQjuixFK3xeSz6ELgjDhUruFhYVSb34M0Sjg5+8cBCCOenx2bhbKbjRDp9WkPZNS+Hw+WX4mSsR9MYL7YoQS9kVbW9uEX5P8LBeHw4Ht27dLvZmUFV1lwDe+ejnysoZG4hoAeVla7LqvKKPLnIgufZKO0F0uV+Jm0oIgwGAwSLm5lP3L927BC4eP47fvH4cm0o+aiuVYsXSe3LGIiKZFskJ3Op2or6+HyWRCMBjE5s2bR82ry0mn1eA7pYvwndJFQx+hWOZEpAKSHhRVSoETEWUCXilKRKQSLHQiIpVgoRMRqQQLnYhIJVjoREQqwUInIlIJFjoRkUqw0ImIVIKFTkSkEix0IiKVYKETEakEC52ISCVY6EREKsFCJyJSCRY6EZFKsNCJiFSChU5EpBIsdCIilWChExGpBAudiEglWOhERCrBQiciUgkWOhGRSrDQiYhUgoVORKQSLHQiIpVgoRMRqQQLnYhIJVjoREQqwUInIlIJFjoRkUqw0ImIVIKFTkSkEix0IiKVyNhCD0dicH1yGkdOhyGKotxxiIimTbJC93g8qKyslOrpp+WPRz5H6U9fxuZn2vA3L53EHbtfw4lAv9yxiIimRbJCN5vN6OjokOrpp0wIR/A/fnkIZwdi6BuIIhwV0XXmHGp/fVjuaERE0yLplIvRaJTy6afktSNfQKsZ/VgsLuJwVwChcESeUEREM0Av58Z9Pl/at3nqZB/i8fg4XxHx8dGjyMvK2MMKCIfDsvxMlIj7YgT3xQil7wtZC72wsDDt21x6bQwN7leAaDTxWJZWg1XXX4ZblpvTnkdJfD6fLD8TJeK+GMF9MUIJ+6KtrW3Cr2XccDQvW4dnHlqJKww5mJWtQ5ZOg5uXzMU/2W6SOxoR0bRINkL3eDzw+/3weDwwm5U18r158Vy4/+ZOfHa6Dye6juHrK0rkjkRENG2SFbrZbMaRI0ekevpp02o1uO7yfES+lHXWiYhoxmTclAsRkVqx0ImIVIKFTkSkEix0IiKVYKETEakEC52ISCVY6EREKsFCJyJSCRY6EZFKsNCJiFSChU5EpBIsdCIilWChExGpBAudiEglWOhERCrBQiciUgkWOhGRSmhEURTl2HCyG50SEdHESktLx31ctkInIqKZxSkXIiKVYKETEalERha6x+NBZWWl3DEUQRAE1NTUoKysDHV1dXLHkZ3dbkdNTQ3sdrvcURTB7/ejpqZG7hiys9vtsFqtsFqtEARB7jgT0ssdQA5msxkdHR1yx1AEl8uFxsZGAEBZWRk2btwIk8kkcyp5eDweVFdXAwCsVmvi15nM6XTKHUERAoEAWlpa5I5xURk5QgcAo9EodwRFqKioSPy6uLg4Y8scGHqjB4aK3WazyZxGfk6nc9TrI1P5/X54vV4sW7ZM8W9wGVvoNJogCCgqKpI7huwEQcATTzyR8VMuHo8HZrOZAx8AJpMJzc3NaGlpwSOPPKLoKRcWOgEAHA4Htm/fLncM2RkMBjQ2NqKoqAgej0fuOLKpr6+H3W7Hjh074Ha74XA45I4kO7PZjLVr18Lv98sdZUIZOYdOo7lcrsQUgyAIMBgMMieSn8lkyujpp+bmZgBDr4cdO3ZwCuo8w1NzSpSRhe7xeOD3+xMfKzOZ0+lEfX09TCYTgsEgNm/enLHzpk6nEy6XCxaLBatWreIbGwEY+vTq8XhQUVGh+Dc2XilKRKQSnEMnIlIJFjoRkUqw0ImIVIKFTkSkEix0IiKVYKGT4rhcLixbtgyVlZWTWizr/As+7rrrrhnPMvz8giCgsrISNTU1M3bVoCAIqKury/grVGl6MvI8dFI2i8UCs9k86rzfsrIymM1mWCyWcf+M3+/Hzp07ExfEzNRCSsNZLBZL4kIjg8GQWLhrps5VNxgMKCgoQCAQmJHno8zEETpdMs4fgdfV1cHlciWWdnW5XOjo6IDD4YDL5UJtbS0EQYDT6YTVaoXT6UyM+Ic5HA44HA5UVlYmnm8i4xX3+euc2O12OBwO7Ny5c9Tz2+32xOje4XDAarXCbrfP2CcIovOx0Emxurq6EqVdXFycGK17PB6EQiFYLBYUFBTA6XTCYrHAaDTCZrPBYrEklke2WCzweDywWCxoaGjAnj17AAxNcdjtdthsNthsNni93glH/xcz/EYw/FzA0FWnRqMR1dXVKCoqgt1uT2Sx2WyJTxJEM4lTLqRYoVAIdrsd+fn5owrQbDbjscceS4zGxyvi4dGzwWBI/Adg1Jz38Ih/uss/FBcXo7a2FoFAILHAmcvlgslkgsPhwOLFi2EymWA0GsfkIZpJHKGTYpnNZjQ0NIxZ7W/4oKTFYpnyqNpgMKCqqiox1bJt27YJv9doNCY9+GkwGNDa2gqv1wur1Qpg6IYIZrM5MWqfak6iyWChk6IZDAY8+eST2LlzZ2I5W4fDkVgN8fyDiMFgcNSvg8Fg0iIuKCjA9u3bYbPZko7SLRbLmPn1gwcPJv6M0+mEwWAY9SmipKQE9fX1id9z+VlKBxY6KY7L5YLf74fT6YTf74fZbMa2bdvw4IMPJqZYXC4X6urqEtMaRqMRJpMJdXV1ePvttyEIAlwuF1wuFwRBgMfjSZTy8P/37NmDu+66C1arddTBzAtVV1fDZDKhpqYGdXV1qKurw8aNG0d9z86dO0ctQ1xdXQ2j0YiysjJUVlZi7dq12L9/fyLXhYYf93q9il5vm5SNqy1SRhpePtlisSAYDMLv90MQhIxdOpjUgSN0ykh2ux3t7e0ARubIM31tfLr0cYROGUkQBDgcDrS3t6OkpAQ2m41nntAlj4VORKQSnHIhIlIJFjoRkUqw0ImIVIKFTkSkEix0IiKV+P+os4PhMZtTgwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "user_id_1, user_id_2 = 1, 8\n",
    "movies_user1=data[data['user_id'] ==user_id_1 ][['user_id','movie_id','rating']]\n",
    "movies_user2=data[data['user_id'] ==user_id_2 ][['user_id','movie_id','rating']]\n",
    "    \n",
    "# FIND SHARED FILMS\n",
    "rep=pd.merge(movies_user1 ,movies_user2,on='movie_id')\n",
    "x= rep.rating_x + np.random.normal(loc=0.0, scale=0.1,size=len(rep.rating_x))\n",
    "y= rep.rating_y +np.random.normal(loc=0.0, scale=0.1,size=len(rep.rating_y))\n",
    "    \n",
    "a=rep.groupby(['rating_x', 'rating_y']).size()\n",
    "x=[]\n",
    "y=[]\n",
    "s=[]\n",
    "for item,b in a.items():\n",
    "    x.append(item[0])\n",
    "    y.append(item[1])\n",
    "    s.append(b*30.)\n",
    "\n",
    "fig = plt.figure(figsize=(6,4))\n",
    "plt.scatter(x,y, s=s)\n",
    "plt.xlabel('Rating User 1')\n",
    "plt.ylabel('Rating User '+str(8))\n",
    "plt.axis([0.5,5.5,0.5,5.5])\n",
    "plt.savefig(\"corre18.png\",dpi= 300, bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, with user 31:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAECCAYAAADuGCyPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYRUlEQVR4nO3de3BU533G8Uc3EDHaFU6c2jHr2LUTLK1w7CG40eI2l7GNcNNc5DErz7QJciTcSVuJxtB0OgRsJtNpLegU9RZY2fJ0OqkXT5RJk1gbHNLGNitfoqaxtLu4vtRmuRhfgvZImAUhbf9QtEJIWlago3N49/uZYRBnYc9vXh2effU757ynKJPJZAQAuOQVO10AAGBuEOgAYAgCHQAMQaADgCEIdAAwRKlTO+7t7XVq1wBwSVuxYsW02x0LdGnmouZTIpFQVVWV02W4AmMxgbGYwFhMcMNY5JoM03IBAEMQ6ABgCAIdAAxBoAOAIQh0ADAEgQ4AhiDQAcAQBDoAGIJABwBDEOgAYAgCHQAMQaADgCEIdAAwBIEOAIYg0AHAEI6uhw64zatvD6n/cErH3z6ha68f0aIFJU6XBBeIH7H0v8cGlXr3ff32x0a0sNSdxwWBDkh6b+iU/vjfetV3KKWS4iJlMqN6+Nmn9JdrbtRXA9c6XR4ccmTgpJr/9Rd67Z0hlRSNHRdtz/5U3/5Sjb50y9VOlzeFrYEeCoXU3d0tSXrsscfk8Xjs3B1wQUZHM2rY/ZzeePeEhkczZ70yor/pTsi7qMyV/3lhr/TwiOr/Jap3Bk9pZNJxcUZ/2fWSlly2QJ/++BWO1TcdWwN9YGBAXV1ddu4CuGg/f+UdHRk4eU6Yjzk5PKqHf3JAX7z5IyoqKnKgOjjlxy8dlXVy+JwwH5MeHtXDkQOuC3TbToomk0nF43EtW7ZMkUjErt0AF+2n8WM6cXpkxtd/PXRaR1LpeawIbtDdf1Tv5zguDhwdVHp45tedYNsM3efzqbOzU7FYTOvWrVMgEJjSckkkEnbtPm/pdNoVdbhBoY7F8ePHc74+msno1VdelbW4ME85FepxMTg4lPP1jDI6cOCAFpa652JB249Qv9+vNWvWKJlMyu/3T3rN6adnS+54irdbFOpY3FP8Qf38jf+ecZb+oYqF+r1P1hRsy6VQj4u171eo/wf9Mx4XVVd5dPNy/7Sv2am3t3fG1+bto+XcMAfc4tMfu0JXL1mksuKpgb2orFjfXH1jwYZ5Ifv9m66SZ1GZSqY5LsrLivXNuhsdqCo32wI9HA5ry5YtikajCgaDdu0GuGjFxUV6fH2tbvnoEpWXFuuyBSX6QFmRPrCgRH91V5W+yBUuBam8rERdXw+o+qoKLSqbOC4qykv18N036fdcdkJUsrHlQojjUnL5ZQu05/5avfbO2I1Fvz52VPd+7haVl7nzBhLMj6u8i/TDP/tdJY5aevmtQaXefUv3fvYWLXBR3/xshXmWB5jB9Vcs1vVXLFYiYRHmyKq6yqOqqzxKJCzXhrnEWi4AYAwCHQAMQaADgCEIdAAwBIEOAIYg0AHAEAQ6ABiCQAcAQxDoAGAIAh0ADEGgA4AhCHQAMASBDgCGINABwBAEOgAYgkAHAEMQ6ABgCAIdAAxBoAOAIQh0ADAEgQ4AhiDQAcAQBDoAGIJABwBDEOgAYAgCHQAMQaADgCEIdAAwBIEOAIYg0AHAEAQ6ABiCQAcAQxDoAGCIUqcLAOA+Pa+9p3/+r1cVO2xpQfGo/nBVqb5Se6085WVOl4YcCPQClx4e0b8996Z++NJRLRg9pT9fcIUC13/I6bIccfL0iB6L/p+6+9/SB4qG9Y3y39Kt113udFnzbtfTr+nvn3pFJ4dHstv+Yd+revyFpP7jT2/T5ZctcLC6+Welh9XxzOv6zwPvyFt6Rt+s+IiWL/U6Xda0bG+5JJNJtbS02L0bXIBMJqOvPPK8tu99Wb9KDujFwyf1tcd+oe//9yGnS5t3o6MZBXf1aOdPX9FLh1J6Lvm+vvro84r0H3W6tHl18L339Xd7/3dSmEvSqTOjOpZK66+fTDhUmTPSwyP64j/u166fv66+wyntf/OE7tkVVc9r7zld2rRsD/RIJGL3LnCBXnzjuPqPWEoPj2a3nRwe0befTCiTyThY2fx7+pV39No7Q0qfOXssRrXtR3EHq5p///7CQY3O8L0fHs3oh786olNnRqZ93UTd/Ud1zErr1G+Oi4yk9PCovv1jdx4XtrZcIpGI6urq1NfXN+3riYTzn/bpdNoVdTjh5wcsnRkZnbL9+InT+p/+uMpLC+ec+bOxAaWnCaojA2nF43EVFRU5UNX8e+mNtzQ8MvOHeSaT0S9eiuvyRYXRrX2m/z29f3rqcfHa24OuzA3bviuxWEx+v19e78y9pqqqKrt2n7dEIuGKOpxwYtGv1fnL4zo9MvmA/eBlC3VzTXXBhJgkfbr0HX33pd4p/3mXLlmk6upqh6qaf594s0gvHHp9xlAvLirSJ2+q1sLSknmuzBmfPnVYkVf6phwXH7/S41hu9Pb2zviabVOw7du3KxQKafPmzerp6VE4HLZrV7hAKz66RDf7KrWobOIwWFRWoi2fL6wwl6RV139IN15ZofLfjEWRxsbiwT/wO1vYPLv31mtUPMP3vqykSF+4+SMFE+aSVFdzpa6uXKSFv/lptbho7Lj41ufd+SFflLG5WWpZljZv3qz29vZJ23t7e7VixQo7d52XQp6hS9LpM6MKv3hQP/zVEZWNntKf3/UJffLawruyQxo7Afbd5w+qu/+oyjOn9Y3f/4RuuWaJ02XNu879/6eHIy8rPTyi8XAoLyvWlZ5y/eBPbpP3A4V16eLQqTP61+gb+tmBt+UpGdamP7hFVVd5HKsnV3YWRiMMM1pQWqw/qr1Wf1R77diHW4GGuSSVl5Xovtuu0323XTc2FgUY5pLUuOo63bTUq3/+r9fUfzilMo1q3e/eoHtvvUaXLSy8yFi8sFRf/+wN+vpnbxg7LhwM8/Ox/bvj8XimzM4BuNuKj16uR7469uE+9lPsbztcEfJROJcxAIDhCHQAMASBDgCGINABwBAEOgAYgkAHAEPMOtCHhoZ06FDhrcYHAG6XM9D37NmjO+64Qxs2bJA0djv/5z73OW3dulWPPPLIfNQHAMjTeW8seuqpp5RMJrVhwwb19PRo3759qqio0J49e+ajPgBAnnLO0M9eoCkSiWj9+vWqqKiQNPbgCgCAe+ScoadSKd13332KxWLauXOnUqmUWltbNTQ0VFBLigLApSBnoDc1NWn16tWqrKzMzsxra2t16NAh1dbWzkuBAID8nLeH7vP5pvz53G0AAOdxHToAGCKvQL/zzju59hwAXC6vQF+7dq0qKysnbevp6bGjHgDABcrrARdPPvmkduzYIY9n7EkdmUxGg4PufOo1ABSqvAK9oaFBtbW1k2bp3d3ddtUEALgAebVcamtr1dHRof7+flVUVCgSieiuu+6yuzYAwCzkFehbt25VRUVF9sToPffco9bWVlsLAwDMTl6BHggEtHHjRmUymey2/v5+24oCAMxeXoHu8Xi0d+9eDQ4OqqenR/X19fL7/XbXBgCYhbwvW8xkMnrzzTf1+OOP66677tKjjz5qd20AgFnI6yoXSVq9erVWr15tZy0AgIuQ90nR8WvO7777bt1999164oknbC0MADA7eQW63+9XVVWVduzYIa/Xq+9973taunSp3bUBAGYhr0C3LEsdHR0Kh8Nqb2+XNPbACwCAe+QV6E1NTfL7/dq3b58WL16sn/zkJwoEAnbXBgCYhZwnRYeGhrR48WJJmvRAC06OAoD75Az0zZs3KxgMTllpsaqqys6aAAAXIGfLxbIsSdLAwMCkXzt27GB9dABwmZwz9J07d2afJXq22tpa7dixQw888IBthQEAZifnDH26MAcAuFPOQM/VVkkmk3NeDADgwuVsuWzdulWBQCD7pCJprK8ejUa5sQhAwRhMDyt2xNIJa1huviQkZ6Dv379fx48fn7StsrJSgUBATU1N533zUCikvr4+LV++XM3NzRdXKQA44LvPv6ltP4qrrKRYp4ZHdGvf+wp9ZaUWLShxurQpcgZ6Z2fnpOvPZyMWi2VDvL6+nkAHcMl55digtv0orvTwqNLDo5KkF944rr+NJPTgF2ocrm6qnD30Cw1zSdn10mOxmILB4AW/DwA45UcvHdXwyOikbafPjOr7vzziUEW55b187oWwLEu7du1SPB6fNtTHV3B0UjqddkUdbsBYTGAsJhTyWLz77q+lzNTtmdERV46JrYHu8XjU3t6ulpYWxWKxKU85csMdp4lEwhV1uAFjMYGxmFDIY9F0xQl9L/60RoYnZunlpcW693eudWxMent7Z3wtr8W5LpbP55PP55uPXQHAnLnuQ5fpb+tv0mULSrR4YanKSor0mWVX6Bt3ftzp0qaV1wy9p6dn2n763r17VV1dPe0ljJFIRNFoVIFAQKtWrZp06SMAXCq+eMvVWl1zpV59e0gDbx3UbSuWO13SjPIK9P3796ujo0OStHHjRlVVVWnr1q2qq6tTPB5XZWVldlXGcXV1daqrq5v7igFgnpWXlajmaq8Slq1d6ouWV8slHo9r7dq1evDBB9Xf3y9J6uvrk8/nU21trcLhsK1FAgDOL69Ar6ur0+rVq+Xz+XTw4EFJY2uljy+r29fXZ1uBAID85H1S9I477sie1e3o6NDAwICi0ah2796tmhr3XWAPAIUmr4bQ2rVrtWbNGkkTKzA2NTUpmUzq2Wef1f33329fhQCAvOQ9Q6+oqJi0nG5PT498Pp+2bds25YQoAGD+5TVD7+jo0O7duzU4OChJymQyKioqcuWdUgBQqPIK9Gg0qn379k2ZoQMA3COvlktDQ4OKioombTv3wdEAAGflNUP/zne+o9bW1uzdnplMRoODg7RcAMBF8gr0+++/X4FAINtysSyLlgsAuExegb5q1SpJYzcTSWOBblmWfVUBAGYtZ6Dfeeed6urqUktLi/r7+1VZWalMJqNUKqXBwUHdc88981UnAOA8cgb63r17JUnNzc1TVlt84YUX7KsKADBreV3lcujQoSnbnnnmmTkvBgBw4XLO0JPJpLZu3apYLKZQKJTdPjAwIJ/PpwceeMD2AgEA+ckZ6D6fTw899NC0D7hYsmSJrYUBAGbnvFe5jD8+bryfPi4cDuuRRx6xrTAAwOzkddlia2urvF6vnnvuOX3qU59SKpVSdXW13bUBAGYhr0BvaGhQbW2t9uzZo0AgoKVLl2rHjh121wYAmIW8nykaiUT00EMP6b777pPP51NPTw8nRQHARfIK9I0bNyoej0uSdu7cqXA4rGAwaGthAIDZyfsBF+M984qKCjU1NSmVStlWFABg9s4b6D09PXriiScmbUskEtqwYYNdNQEALkDOQO/o6FBra6va2tr0ta99TZK0Z88effnLX1Zzc/O8FAgAyE/OHnpfX192zZbW1lZt2LBB0WhUXV1dXLYIAC6Tc4a+fPny7NcNDQ2yLEs/+9nPVF1dPe36LgAA5+Scoff09Mjr9UqSDh48KJ/Pp+7ubklSJBLhTlEAcJGcgb5//34dP3580ra+vj5J0uHDh+2rCgAwazkDvbOzc8qiXON4BB0AuEvOHvpMYX6+1wAA8y/vG4sAAO5GoAOAIQh0ADAEgQ4AhiDQAcAQeS2fC6CwWOlhhZ5+XU/Fj2lxyRn9Rflv6dbrLne6LJyHbTN0y7LU0tKilStXqq2tza7dAJhjwyOj+vI/7dfup1/XgbcG9YvDJ/WVR5/XvsQxp0vDedgW6NFoVO3t7XrxxRe1Z88eJZNJu3YFYA49FT+mt1JpnTozmt2WHh7Vt3+ccLAq5MO2lktdXV3265qaGvl8vil/J5Fw/gBJp9OuqMMNGIsJhTwW0dhxnTg9MmX7wfdOFOyYjHP7cWF7D92yrBmX2q2qqrJ79+eVSCRcUYcbMBYTCnksPpM5pu/Hfzkl1G/4cEXBjsk4NxwXvb29M75m+1Uu4XBYmzZtsns3AObIZ2/8sG748GKVl43FQ3GRtKisRFu/wDMQ3M7WGXo0Gs0+TNqyLHk8Hjt3B2AOlBQXKXx/rR5/4aCeih/TIp3SA5+/RVVX8f/X7WwL9Egkou3bt8vn8ymVSmn9+vWT+uoA3Ku8rETrVl2ndauuG2szEOaXBFtPihLgADB/uFMUAAxBoAOAIQh0ADAEgQ4AhiDQAcAQBDoAGIJABwBDEOgAYAgCHQAMQaADgCEIdAAwBIEOAIYg0AHAEAQ6ABiCQAcAQxDoAGAIAh0ADEGgA4AhCHQAMASBDgCGINABwBAEOgAYgkAHAEMQ6ABgCAIdAAxBoAOAIQh0ADAEgQ4AhiDQAcAQBDoAGIJABwBDEOgAYAgCHQAMQaADgCEIdAAwhG2BHovF1NjYaNfbAwDOYVug+/1+9ff32/X2AIBz2Npy8Xq9dr49AOAspU7uPJFIOLl7SVI6nXZFHW7AWExgLCYwFhPcPhaOBnpVVZWTu5c09qHihjrcgLGYwFhMYCwmuGEsent7Z3yNq1wAwBC2XuWSTCYVi8Xs2gUA4Cy2tVz8fr9efvllu94eAHAOWi4AYAgCHQAMQaADgCEIdAAwBIEOAIYg0AHAEAQ6ABiCQAcAQxDoAGAIAh0ADEGgA4AhCHQAMASBDgCGINABwBAEOgAYgkAHAEMQ6ABgiKJMJpNxYse5HnQKAJjZihUrpt3uWKADAOYWLRcAMASBDgCGKMhAj8ViamxsdLoMV7AsSy0tLVq5cqXa2tqcLsdxoVBILS0tCoVCTpfiCslkUi0tLU6X4bhQKKT6+nrV19fLsiyny5lRqdMFOMHv96u/v9/pMlwhGo2qvb1dkrRy5Uo1NDTI5/M5XJUzYrGYmpubJUn19fXZrwtZJBJxugRXGBgYUFdXl9NlnFdBztAlyev1Ol2CK9TV1WW/rqmpKdgwl8Y+6KWxYA8Ggw5X47xIJDLp+ChUyWRS8Xhcy5Ytc/0HXMEGOiazLEvV1dVOl+E4y7K0a9eugm+5xGIx+f1+Jj6SfD6fOjs71dXVpW9961uubrkQ6JAkhcNhbdq0yekyHOfxeNTe3q7q6mrFYjGny3HM9u3bFQqFtHnzZvX09CgcDjtdkuP8fr/WrFmjZDLpdCkzKsgeOiaLRqPZFoNlWfJ4PA5X5Dyfz1fQ7afOzk5JY8fD5s2baUGdZbw150YFGeixWEzJZDL7Y2Uhi0Qi2r59u3w+n1KplNavX1+wfdNIJKJoNKpAIKBVq1bxwQZJYz+9xmIx1dXVuf6DjTtFAcAQ9NABwBAEOgAYgkAHAEMQ6ABgCAIdAAxBoMN1otGoli1bpsbGxlktlnX2DR+33377nNcy/v6WZamxsVEtLS1zdtegZVlqa2sr+DtUcXEK8jp0uFsgEJDf75903e/KlSvl9/sVCASm/TfJZFJbtmzJ3hAzVwspjdcSCASyNxp5PJ7swl1zda26x+NRZWWlBgYG5uT9UJiYoeOScfYMvK2tTdFoNLu0azQaVX9/v8LhsKLRqFpbW2VZliKRiOrr6xWJRLIz/nHhcFjhcFiNjY3Z95vJdMF99jonoVBI4XBYW7ZsmfT+oVAoO7sPh8Oqr69XKBSas58ggLMR6HCtgwcPZkO7pqYmO1uPxWIaHBxUIBBQZWWlIpGIAoGAvF6vgsGgAoFAdnnkQCCgWCymQCCgnTt3avfu3ZLGWhyhUEjBYFDBYFDxeHzG2f/5jH8QjL+XNHbXqdfrVXNzs6qrqxUKhbK1BIPB7E8SwFyi5QLXGhwcVCgUUkVFxaQA9Pv92rZtW3Y2Pl0Qj8+ePR5P9pekST3v8Rn/xS7/UFNTo9bWVg0MDGQXOItGo/L5fAqHw7rmmmvk8/nk9Xqn1APMJWbocC2/36+dO3dOWe1v/KRkIBC44Fm1x+NRU1NTttWycePGGf+u1+vNefLT4/Fo3759isfjqq+vlzT2QAS/35+dtV9oncBsEOhwNY/Ho8cee0xbtmzJLmcbDoezqyGefRIxlUpN+jqVSuUM4srKSm3atEnBYDDnLD0QCEzpr+/fvz/7byKRiDwez6SfIpYvX67t27dn/8zys5gPBDpcJxqNKplMKhKJKJlMyu/3a+PGjVq3bl22xRKNRtXW1pZta3i9Xvl8PrW1ten555+XZVmKRqOKRqOyLEuxWCwbyuO/7969W7fffrvq6+snncw8V3Nzs3w+n1paWtTW1qa2tjY1NDRM+jtbtmyZtAxxc3OzvF6vVq5cqcbGRq1Zs0bd3d3Zus41vj0ej7t6vW24G6stoiCNL58cCASUSqWUTCZlWVbBLh0MMzBDR0EKhULq6+uTNNEjL/S18XHpY4aOgmRZlsLhsPr6+rR8+XIFg0GuPMElj0AHAEPQcgEAQxDoAGAIAh0ADEGgA4AhCHQAMMT/AwwDLN6Lfez3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "user_id_1, user_id_2 = 1, 31\n",
    "movies_user1=data[data['user_id'] ==user_id_1 ][['user_id','movie_id','rating']]\n",
    "movies_user2=data[data['user_id'] ==user_id_2 ][['user_id','movie_id','rating']]\n",
    "    \n",
    "# FIND SHARED FILMS\n",
    "rep=pd.merge(movies_user1 ,movies_user2,on='movie_id')\n",
    "x= rep.rating_x + np.random.normal(loc=0.0, scale=0.1,size=len(rep.rating_x))\n",
    "y= rep.rating_y +np.random.normal(loc=0.0, scale=0.1,size=len(rep.rating_y))\n",
    "    \n",
    "a=rep.groupby(['rating_x', 'rating_y']).size()\n",
    "x=[]\n",
    "y=[]\n",
    "s=[]\n",
    "for item,b in a.items():\n",
    "    x.append(item[0])\n",
    "    y.append(item[1])\n",
    "    s.append(b*30.)\n",
    "\n",
    "fig = plt.figure(figsize=(6,4))\n",
    "plt.scatter(x,y, s=s)\n",
    "plt.xlabel('Rating User 1')\n",
    "plt.ylabel('Rating User '+str(8))\n",
    "plt.axis([0.5,5.5,0.5,5.5])\n",
    "plt.savefig(\"corre131.png\",dpi= 300, bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Evaluation: performance criterion</h3>\n",
    "In order to validate the system, we will divide the dataset into two different sets: one called \\emph{$X\\_train$} containing  $80\\%$ of the data from each user; and another called \\emph{$X\\_test$}, with the remaining $20\\%$ of the data from each user. In the following code we create a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#Training samples =  79619\n",
      "#Test samples =  20381\n",
      "#Users = 943\n",
      "#Movies = 1654\n"
     ]
    }
   ],
   "source": [
    "def assign_to_set(df):\n",
    "    sampled_ids = np.random.choice(\n",
    "        df.index,\n",
    "        size=np.int64(np.ceil(df.index.size * 0.2)),\n",
    "        replace=False)\n",
    "    df.loc[sampled_ids, 'for_testing'] = True\n",
    "    return df\n",
    "\n",
    "def create_train_test(data, key = 'user_id'):\n",
    "    data['for_testing'] = False\n",
    "    grouped = data.groupby(key, group_keys=False).apply(assign_to_set)\n",
    "    # dataframe used to train our model\n",
    "    data_train = data[grouped.for_testing == False]\n",
    "    # dataframe used to evaluate our model\n",
    "    data_test = data[grouped.for_testing == True]\n",
    "    return data_train, data_test\n",
    "\n",
    "\n",
    "X_train, X_test =  create_train_test(data)\n",
    "\n",
    "print(\"#Training samples = \",X_train.shape[0])\n",
    "print(\"#Test samples = \",X_test.shape[0])\n",
    "print('#Users =', X_train.user_id.nunique())\n",
    "print('#Movies =',X_train.movie_id.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Once the data is divided in these sets, we can build a model with the **training set** and evaluate its performance using the **test set**. To validate our methods we can use the standard Root Mean Square Error (RMSE):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_rmse(y_pred, y_true):\n",
    "    \"\"\" Compute Root Mean Squared Error. \"\"\"\n",
    "    return np.sqrt(np.mean(np.power(y_pred - y_true, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or instead, we can use ranking measures such as Precision, Reacall or Average Precision (AP)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precision(recommended_items, relevant_items):\n",
    "    is_relevant = np.in1d(recommended_items, \n",
    "                          relevant_items, \n",
    "                          assume_unique=True)\n",
    "    \n",
    "    precision = np.sum(is_relevant, dtype=np.float32) / len(is_relevant)\n",
    "    \n",
    "    return precision\n",
    "\n",
    "def recall(recommended_items, relevant_items):  \n",
    "    is_relevant = np.in1d(recommended_items, \n",
    "                          relevant_items, \n",
    "                          assume_unique=True)\n",
    "    \n",
    "    recall = np.sum(is_relevant, dtype=np.float32) / relevant_items.shape[0]\n",
    "    \n",
    "    return recall\n",
    "\n",
    "def AP(recommended_items, relevant_items):\n",
    "    is_relevant = np.in1d(recommended_items, \n",
    "                          relevant_items, \n",
    "                          assume_unique=True)\n",
    "    \n",
    "    # Cumulative sum: precision at 1, at 2, at 3 ...\n",
    "    p_at_k = is_relevant * np.cumsum(is_relevant, dtype=np.float32) / (1 + np.arange(is_relevant.shape[0]))\n",
    "    ap_score = np.sum(p_at_k) / np.min([relevant_items.shape[0], is_relevant.shape[0]])\n",
    "\n",
    "    return ap_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the function **evaluate** which computes the RMSE and the ranking metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(rec_object, train, test, \n",
    "             at = 20, thr_relevant= 4):\n",
    "    \"\"\" Perfomance evaluation \"\"\"\n",
    "    \n",
    "    # RMSE evaluation\n",
    "    ids_to_estimate = zip(test.user_id, test.movie_id)\n",
    "    y_estimated = np.array([rec_object.predict_score(u,i) \n",
    "                          if u in train.user_id else 3 \n",
    "                          for (u,i) in ids_to_estimate ])\n",
    "    \n",
    "    y_real = test.rating.values\n",
    "    rmse = compute_rmse(y_estimated, y_real)\n",
    "    \n",
    "    print(\"RMSE: {:.4f}\".format(rmse))\n",
    "          \n",
    "    cumulative_precision = 0.0\n",
    "    cumulative_recall = 0.0\n",
    "    cumulative_AP = 0.0\n",
    "\n",
    "    num_eval = 0\n",
    "\n",
    "    for user_id in tqdm(test.user_id.unique()):\n",
    "\n",
    "        relevant_items = test[(test.user_id==user_id )&( test.rating>=thr_relevant)].movie_id.values\n",
    "\n",
    "        if len(relevant_items)>0:\n",
    "\n",
    "            recommended_items = rec_object.predict_top(user_id, at=at)\n",
    "            num_eval+=1\n",
    "\n",
    "            cumulative_precision += precision(recommended_items, relevant_items)\n",
    "            cumulative_recall += recall(recommended_items, relevant_items)\n",
    "            cumulative_AP += AP(recommended_items, relevant_items)\n",
    "\n",
    "    cumulative_precision /= num_eval\n",
    "    cumulative_recall /= num_eval\n",
    "    MAP = cumulative_AP / num_eval\n",
    "\n",
    "    print(\"Precision = {:.4f}, Recall = {:.4f}, MAP = {:.4f}\".format(\n",
    "        cumulative_precision, cumulative_recall, MAP)) \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommender System \n",
    "\n",
    "### Naïve Recommender System \n",
    "\n",
    "First of all, let's build our naive recommender: A random recommender system.\n",
    "\n",
    "We can define our recommender system with a Python class. This class consists of a constructor and three methods: `fit`, `predict_score` and `predict_top`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomRecommender():\n",
    "\n",
    "    def fit(self, train):\n",
    "        self.items = train.title.unique()\n",
    "    \n",
    "    def predict_score(self, user_id, movie_id):\n",
    "        '''Given a user_id and movie_id predict its score'''\n",
    "        return np.random.uniform(1,5)\n",
    "    \n",
    "    def predict_top(self, user_id, at=5):\n",
    "        '''Given a user_id predicts its top 'at' movies'''\n",
    "        recommended_items = np.random.choice(self.items, at)\n",
    "\n",
    "        return recommended_items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to create the object and train it using training data. We can do it with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_model = RandomRecommender()\n",
    "random_model.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model is trained, we can evaluate it as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.6281\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████| 943/943 [00:00<00:00, 1603.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision = 0.0000, Recall = 0.0000, MAP = 0.0000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "evaluate(random_model,X_train,X_test, at = 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The random model has achieved an RMSE of 1.6241 and Precision, Recall and MAP of 0.0\\%. These are the numbers we need to beat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collaborative Recommender System \n",
    "\n",
    "Similarly, we can build our collaborative recommender system with a Python class. This class consists of a constructor and three methods: `fit`, `predict_score` and `predict_top`. In this case,  the `fit` will be in charge to compute and store the users similarities into a Python dictionary. This is a really simple method but quite expensive in terms of computation when dealing with a large dataset. We decided to show one of the most basic schemes in order to implement it. More complex algorithms can be used in order to improve the computations cost. Moreover, online strategies can be used when dealing with a really dynamic problems. In the \\texttt{predict} the score for a movie and a user is estimated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CollaborativeFiltering:\n",
    "    \"\"\" Collaborative filtering model \"\"\"\n",
    "    \n",
    "    def __init__(self, similarity = 'pearson'):\n",
    "        '''Constructor'''\n",
    "        self.sim_method=similarity\n",
    "        self.sim = {}\n",
    "\n",
    "    def fit(self, train):\n",
    "        '''Prepare data structures for estimation'''\n",
    "        self.train = train\n",
    "        allUsers = set(self.train['user_id'])\n",
    "        \n",
    "        # Create a dictionary with user, movie, rating\n",
    "        self.seen_movies = {user: {} for user in allUsers}\n",
    "        \n",
    "        for user in allUsers:\n",
    "            user_ratings = train[train.user_id == user][['movie_id', 'rating']]\n",
    "            self.seen_movies[user] = dict(zip(user_ratings['movie_id'], user_ratings['rating']))\n",
    "    \n",
    "        for usr_1 in tqdm(allUsers):\n",
    "            self.sim.setdefault(usr_1, {})\n",
    "            a = self.train[self.train.user_id == usr_1][['movie_id']]\n",
    "            data_reduced = pd.merge(self.train, a , on='movie_id') \n",
    "            \n",
    "            for usr_2 in allUsers:\n",
    "                # Avoid comparing a user with themselves\n",
    "                if usr_1 == usr_2: \n",
    "                    continue      \n",
    "                self.sim.setdefault(usr_2, {})\n",
    "                if(usr_1 in self.sim[usr_2]):\n",
    "                    continue # since is a simetric matrix\n",
    "                sim = user_sim(data_reduced, usr_1, usr_2, \n",
    "                               method = self.sim_method)\n",
    "                if(sim < 0):\n",
    "                    self.sim[usr_1][usr_2]=0\n",
    "                    self.sim[usr_2][usr_1]=0\n",
    "                else:\n",
    "                    self.sim[usr_1][usr_2]=sim\n",
    "                    self.sim[usr_2][usr_1]=sim\n",
    "        \n",
    "        \n",
    "    def predict_score(self, usr_id, movie_id):\n",
    "        ''' Given a user_id and movie_id it predicts its score'''\n",
    "        seen = self.train[self.train.movie_id == movie_id]\n",
    "        rating_num, rating_den = 0.0, 0.0\n",
    "        allUsers = set(seen['user_id'])\n",
    "        # Iterate through all users who have seen the movie\n",
    "        for other in allUsers:\n",
    "            if usr_id == other:\n",
    "                continue  # Skip the current user, as self-similarity is not needed\n",
    "            \n",
    "            similarity = self.sim[usr_id][other]\n",
    "            rating = float(self.seen_movies[other][movie_id])\n",
    "            \n",
    "            rating_num += similarity * rating\n",
    "            rating_den += similarity\n",
    "            \n",
    "        # If the denominator is zero (no similar users), handle the case\n",
    "        if rating_den == 0: \n",
    "            if self.train.rating[self.train['movie_id'] == movie_id].mean()>0:\n",
    "                # return the mean movie rating if there is no similar for the computation\n",
    "                return self.train.rating[self.train['movie_id'] == movie_id].mean()\n",
    "            else:# return mean user rating \n",
    "                return self.train.rating[self.train['user_id'] == usr_id].mean()\n",
    "       \n",
    "        # return the predicted score\n",
    "        return rating_num/rating_den\n",
    "\n",
    "    def predict_top(self, usr_id, at=20, remove_seen=True):\n",
    "        '''Given a usr_id predict its top 'at' movies'''\n",
    "        # Get the movies already seen by the user\n",
    "        seen_items = set(self.train[self.train.user_id == usr_id].movie_id.values)\n",
    "\n",
    "        # Get the set of unseen items\n",
    "        unseen_items = set(self.train.movie_id.values) - seen_items\n",
    "\n",
    "        # Generate predictions for unseen items\n",
    "        predictions = [(item_id, self.predict_score(usr_id, item_id)) for item_id in unseen_items]\n",
    "\n",
    "        # Sort predictions by score in descending order and select the top 'at' items\n",
    "        sorted_predictions = sorted(predictions, key=lambda x: x[1], reverse=True)[:at]\n",
    "\n",
    "        # Extract the item IDs from the sorted predictions\n",
    "        top_items = [item_id for item_id, _ in sorted_predictions]\n",
    "        \n",
    "        return top_items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                   | 0/943 [00:00<?, ?it/s]/Users/santisegui/opt/anaconda3/lib/python3.8/site-packages/scipy/stats/stats.py:4023: PearsonRConstantInputWarning: An input array is constant; the correlation coefficient is not defined.\n",
      "  warnings.warn(PearsonRConstantInputWarning())\n",
      "100%|█████████████████████████████████████████| 943/943 [14:16<00:00,  1.10it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3.904786009237921"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_model = CollaborativeFiltering()\n",
    "cf_model.fit(X_train)\n",
    "cf_model.predict_score(usr_id = 2, movie_id = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0630\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 943/943 [08:40<00:00,  1.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision = 0.0197, Recall = 0.0687, MAP = 0.0032\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# It can take a lot of time..\n",
    "evaluate(cf_model,X_train,X_test, at = 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class  = \"alert alert-success\">**EXERCISE 1**<p>\n",
    "Modify the Recomender System using as a prediction function the following equation:\n",
    "$$pred(a,p) = \\bar{r_a} + \\frac{\\sum_{b \\in N}{sim(a,b)*(r_{b,p}-\\bar{r_b})}}{\\sum_{b \\in N}{sim(a,b)}}$$\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CollaborativeFiltering_Ex1(CollaborativeFiltering):\n",
    "    def predict_score(self, usr_id, movie_id, N = 10):\n",
    "        ''' Given a user_id and movie_id it predicts its score'''\n",
    "        seen = self.train[self.train['movie_id'] ==movie_id]\n",
    "        rating_num, rating_den = 0.0, 0.0\n",
    "        allUsers = set(seen['user_id'])\n",
    "        \n",
    "        for other in allUsers:\n",
    "            if usr_id == other:\n",
    "                continue  # Skip the current user, as self-similarity is not needed\n",
    "            similarity = self.sim[usr_id][other]\n",
    "            rating = float(self.seen_movies[other][movie_id])\n",
    "            mean_user_rating = np.mean([self.seen_movies[other][key] \n",
    "                                        for key in self.seen_movies[other]])\n",
    "            rating_num += similarity * float(rating - mean_user_rating)  \n",
    "            rating_den += similarity\n",
    "        if rating_den == 0: \n",
    "            if self.train.rating[self.train['movie_id'] == movie_id].mean()>0:\n",
    "                # return the mean movie rating if there is no similar for the computation\n",
    "                return self.train.rating[self.train['movie_id'] == movie_id].mean()\n",
    "            else:\n",
    "                # else return mean user rating \n",
    "                return self.train.rating[self.train['user_id'] == usr_id].mean()\n",
    "        mean_rating_user = self.train[self.train.user_id==usr_id].rating.mean()\n",
    "        return mean_rating_user + rating_num/rating_den"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 943/943 [14:20<00:00,  1.10it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3.910297402468975"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_model_v2 = CollaborativeFiltering_Ex1()\n",
    "cf_model_v2.fit(X_train)\n",
    "cf_model_v2.predict_score(usr_id = 2, movie_id = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0135\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████| 943/943 [2:39:51<00:00, 10.17s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision = 0.0084, Recall = 0.0338, MAP = 0.0011\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "evaluate(cf_model_v2, X_train, X_test, at = 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "**EXERCISE 2:**<br>\n",
    "Modify the recomender system from the previous exercice, with one that in order to estimate the score of a movie B for the user A only uses the subset of the N most similar users to user A. Define N as a parameter of the Recoomender.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CollaborativeFiltering_Ex2(CollaborativeFiltering):\n",
    "    def predict_score(self, usr_id, movie_id, N = 10):\n",
    "        seen = self.train[self.train['movie_id'] ==movie_id]\n",
    "        rating_num, rating_den = 0.0, 0.0\n",
    "        allUsers = set(seen['user_id'])\n",
    "        \n",
    "        # SELECT TOP USERS\n",
    "        top_users = sorted([(self.sim[usr_id][other],other) \n",
    "                            for other in seen.user_id.values \n",
    "                            if other in self.sim[usr_id]])[::-1]\n",
    "        \n",
    "        for sim, other in top_users[:N]:\n",
    "            if usr_id == other: continue \n",
    "                \n",
    "            similarity = self.sim[usr_id][other]\n",
    "            rating = float(self.seen_movies[other][movie_id])\n",
    "            mean_user_rating = np.mean([self.seen_movies[other][key] \n",
    "                                        for key in self.seen_movies[other]])\n",
    "            \n",
    "            rating_num += similarity * float(rating - mean_user_rating)  \n",
    "            rating_den += similarity\n",
    "        if rating_den == 0: \n",
    "            if self.train.rating[self.train['movie_id']==movie_id].mean()>0:\n",
    "                # return the mean movie rating if there is no similar for the computation\n",
    "                return self.train.rating[self.train['movie_id']==movie_id].mean()\n",
    "            else:\n",
    "                # else return mean user rating \n",
    "                return self.train.rating[self.train['user_id'] == usr_id].mean()\n",
    "        mean_rating_user = self.train[self.train.user_id == usr_id].rating.mean()\n",
    "        return mean_rating_user + rating_num/rating_den"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 943/943 [14:42<00:00,  1.07it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3.631609239597465"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_model_v3 = CollaborativeFiltering_Ex2()\n",
    "cf_model_v3.fit(X_train)\n",
    "cf_model_v3.predict_score(usr_id = 2, movie_id = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0349\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 943/943 [28:13<00:00,  1.80s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision = 0.0126, Recall = 0.0496, MAP = 0.0022\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "evaluate(cf_model_v3, X_train, X_test, at = 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class  = \"alert alert-success\">**EXERCISE 4**<p>\n",
    "Modify the similarity function with the following:\n",
    "$$new\\_sim(a,b) = sim(a,b) * \\frac{min(50,|P_{ab}|)}{50} $$\n",
    "where $|P_{ab}|$ is the number of common items with user $a$ and user $b$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def user_sim(df, user_1, user_2, \n",
    "             min_common_items = 1, \n",
    "             method = 'pearson',\n",
    "             min_common = 20):\n",
    "    # GET MOVIES OF USER1\n",
    "    mov_usr1 = df[df['user_id'] == user_1 ]\n",
    "    # GET MOVIES OF USER2\n",
    "    mov_usr2 = df[df['user_id'] == user_2 ]\n",
    "    \n",
    "    # FIND SHARED FILMS\n",
    "    df_shared = pd.merge(mov_usr1, mov_usr2, on='movie_id')  \n",
    "    n_common = df_shared.shape[0]\n",
    "    \n",
    "    # If there is no enough common items to compute similarity\n",
    "    if(n_common<=min_common_items):\n",
    "        return 0\n",
    "    \n",
    "    penalty = min(min_common,n_common)/min_common\n",
    "    if method =='pearson':\n",
    "        res=pearsonr(df_shared['rating_x'],df_shared['rating_y'])[0]\n",
    "        if(np.isnan(res)):\n",
    "            return 0\n",
    "        return res * penalty\n",
    "    \n",
    "    elif method =='euclidean':\n",
    "        res = 1.0/(1.0+euclidean(df_shared['rating_x'],\n",
    "                                  df_shared['rating_y'])) \n",
    "        return res * penalty\n",
    "    else:\n",
    "        print(\"method not defined\")\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 943/943 [14:35<00:00,  1.08it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3.902285047933166"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_model_v4 = CollaborativeFiltering_Ex1()\n",
    "cf_model_v4.fit(X_train)\n",
    "cf_model_v4.predict_score(usr_id = 2, movie_id = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.0116\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 53%|█████████████████████▊                   | 501/943 [25:45<25:58,  3.53s/it]"
     ]
    }
   ],
   "source": [
    "evaluate(cf_model_v4,X_train,X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}