programmers_guide/programmers_guide_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implicit Ratings\n",
    "Based on users behaviour rather than explicit ratings. For example, if a person clicks on links about technology but never about sports on a news site, that tells us something about their preferences. Another example would be keeping track of things people have bought - Amazon even has a 'frequently bought together' section on the site once you put something in your cart.\n",
    "\n",
    "Problems with implicit ratings:\n",
    "- People don't bother rating items\n",
    "- People lie\n",
    "- People don't update ratings\n",
    "\n",
    "One of the tricky things about implicit ratings is that if someone buys a gift, it can confuse your system. You want to try and build up a profile of the user so that your recommendations fit their preferences.\n",
    "\n",
    "## Item Based Filtering\n",
    "So far we've been doing user-based collaborative filtering, but this is difficult to scale and results in sparse datasets. Instead, it might be better to do item-based filtering. Here we identify products that are most similar to each other.\n",
    "\n",
    "For example: let's find the cosine similarity between song 1 and song 2. To do this, you'll need to find all the instances where people have rated both songs. We can calculate the similarity using adjusted cosine similarity\n",
    "\n",
    "## Adjusted Cosine Similarity\n",
    "To adjust for inflation of ratings by users, we'll subtract the average rating of each user from each of their ratings. This equation shows the similarity between items i and j. R is the rating, u is a user. $\\bar{R}_u$ is the average rating the user gave all items that they rated \n",
    "\n",
    "$$\n",
    "s(i, j) = \\frac{\\sum_{u \\in U}(R_{u,i} - \\bar{R}_u)(R_{u,j} - \\bar{R}_u)}\n",
    "{\\sqrt{\\sum_{u \\in U}(R_{u,i} - \\bar{R}_u)^2} \\sqrt{\\sum_{u \\in U}(R_{u,j} - \\bar{R}_u)^2}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>average_rating</th>\n",
       "      <th>song1</th>\n",
       "      <th>song2</th>\n",
       "      <th>users</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tamera</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Jasmine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Arturo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Cecilia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Jesssica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jordyn</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   average_rating  song1  song2     users\n",
       "0             3.5    5.0    NaN    Tamera\n",
       "1             3.0    NaN    4.0   Jasmine\n",
       "2             3.0    1.0    2.0    Arturo\n",
       "3             3.5    5.0    5.0   Cecilia\n",
       "4             4.0    4.0    5.0  Jesssica\n",
       "5             2.5    4.0    NaN    Jordyn"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews = { 'users': ['Tamera', 'Jasmine', 'Arturo', 'Cecilia', 'Jesssica', 'Jordyn'],\n",
    "            'average_rating': [3.5, 3, 3, 3.5, 4, 2.5],\n",
    "            'song1': [5, np.nan, 1, 5, 4, 4],\n",
    "            'song2': [np.nan, 4, 2, 5, 5, np.nan]}\n",
    "df = pd.DataFrame(reviews)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only include columns where both songs were rated by user\n",
    "df = df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.824621125124\n"
     ]
    }
   ],
   "source": [
    "minus_average_squared = sum([math.pow((s1 - average) * (s2 - average), 2) for average, s1, s2 in zip(df['average_rating'], df['song1'], df['song2'])])\n",
    "cs =    (( sum([((s1 - average) * (s2 - average)) for average, s1, s2 in zip(df['average_rating'], df['song1'], df['song2'])]) ) /\n",
    "        ( math.sqrt(sum([math.pow(s1 - average, 2) for average, s1, s2 in zip(df['average_rating'], df['song1'], df['song2'])])) * \n",
    "          math.sqrt(sum([math.pow(s2 - average, 2) for average, s1, s2 in zip(df['average_rating'], df['song1'], df['song2'])])) ))\n",
    "print(cs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weighted Slope One\n",
    "Very simple algorithm to predict ratings. If Amy rated Psy 3 and Whitney Houston 4, and Ben rated Psy 4, what should we predict Ben's rating of Whitney Housten would be? Well since Amy liked Whitney Housten 1 more than Psy, we could get that Ben's rating would be 5. Slope 1 is conducted in two parts:\n",
    "\n",
    "1. Ahead of time in batch mode - you calculate the deviation between every pair of items\n",
    "2. Now you have the list of deviations, you can predict how they'd rate something they haven't seen using things they have rated.\n",
    "\n",
    "Calculating the deviations for item i with respect to item j. card(S) is how many elements are in S and X is the entire set of rankings. $card(S_{i,j}(X))$ is the ttoal number of people who rated both i and j\n",
    "\n",
    "$$\n",
    "dev_{i,j} = \\sum_{u \\in S_{i,j}} \\frac{u_i - u_j}{card(S_{i,j}(X))}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Psy</th>\n",
       "      <th>Taylor Swift</th>\n",
       "      <th>Whitney Houston</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Psy  Taylor Swift  Whitney Houston\n",
       "0  3.0           4.0              4.0\n",
       "1  2.0           5.0              NaN\n",
       "2  3.5           NaN              4.0\n",
       "3  NaN           5.0              3.0"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings = {'Taylor Swift': [4,5,np.nan,5], 'Psy': [3,2,3.5,np.nan], 'Whitney Houston': [4,np.nan,4,3]}\n",
    "df = pd.DataFrame(ratings)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Calculating Slope Deviations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0 -2.0\n",
      "The average user rated Swift 2 points higher than Psy\n"
     ]
    }
   ],
   "source": [
    "dfs = df[['Psy', 'Taylor Swift']].dropna()\n",
    "# calculate deviation for Psy with respect to Taylor Swift\n",
    "psy = sum([((a1 - a2) / len(dfs['Psy'])) for a1, a2 in zip(dfs['Taylor Swift'], dfs['Psy'])])\n",
    "# calculate the reverse: Taylor Swift with respect to Psy\n",
    "swift = sum([((a1 - a2) / len(dfs['Psy'])) for a1, a2 in zip(dfs['Psy'], dfs['Taylor Swift'])])\n",
    "print(psy, swift)\n",
    "print('The average user rated Swift 2 points higher than Psy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Making Predictions with Slope\n",
    "Let's make a prediction about what Ben would rate Whitney Houston.\n",
    "\n",
    "$$\n",
    "P^{wS1}(u)_j = \\frac{\\sum_{i \\in S(u)-\\{j\\}}(dev_{j,i} + u_i)c_{j,i}}\n",
    "{\\sum_{i \\in S(u)-\\{j\\}}c_{j,i}}\n",
    "$$\n",
    "\n",
    "- In this equation, $c_{j,i} = card((S_{j,i}(X))$\n",
    "- $P^{wS1}(u)_j$ is our prediction using user u's rating for item j\n",
    "- $\\sum_{i \\in S(u)-\\{j\\}}$ Means for every musician Ben has rated (except for Houston - {j}), we look up the deviation of Whitney Houston to the musician and will add that to Ben's rating for musician i. We multiply it by the cardinality of that pair - the number of people that rated both Whitney and musician i.\n",
    "\n",
    "\n",
    "\n",
    "| Ben's ratings  | Taylor Swift  | PSY  | Whitney Houston  |\n",
    "|:---:|:---:|:---:|:---:|\n",
    "|  Ben |  5 | 2  |  ? |\n",
    "\n",
    "\n",
    "\n",
    "| Deviations  | Taylor Swift  |  PSY |  Whitney Houston |\n",
    "|:---:|:---:|:---:|:---:|\n",
    "| Taylor Swift  |  0 | 2  | 1  |\n",
    "| PSY  |  -2 |  0 | -0.75  |\n",
    "| Whitney Houston  | -1  | 0.75  |  0 |\n",
    "\n",
    "- $u_i$ is Ben's rating of 5 for Taylor Swift and $dev_j,i$ is the deviation of Whitney Houston compared to Swift: -1. Result: 4\n",
    "- Two people rated both Swift and Houston, so $c_{j,i}$ is 2\n",
    "- Result: $(dev_{j,i} + u_i) c_{j,i} = 4 x 2 = 8$\n",
    "- Ben rated Psy and gave him 2: $(dev_{j,i} + u_i) c_{j,i} = 2.75 x 2 = 5.5$\n",
    "- Sum up result for both and get 13.5 for the numerator.\n",
    "- The denominator looks at how for every musician Ben rated, we sum the cardinalities of those musicians (how many people rated both that musician and Whitney Houston)\n",
    "- So Ben rated Swift and the cardinality of Swift and Houston (total number of people that rated them both) is 2. Ben rated Psy and his cardinality is 2. Therefore the denominator is 4\n",
    "- Predicted rating of Houston is 13.5/4 = 3.375"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-2.0"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}