programmers_guide/programmers_guide_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naïve Bayes\n",
    "Probabilistic classifiers can give you a prediction and a probability - something you can't get from nearest neighbour classifiers.\n",
    "\n",
    "## Lazy vs Eager Learners\n",
    "Lazy learners save a training set and have to remember it each time to use it while eager learners will generate a model that it can use to classify new data - this tends to be faster,\n",
    "\n",
    "## Probability\n",
    "Prior probability of a hypothesis: P(h). For a coin: P(heads) = 0.5. Rolling 1 with a die: P(1) = 1/6. Probability person is woman if even number of men and women: P(Female) = 0.5.\n",
    "\n",
    "If you research a uni (say Stanford) a person is at and realise that it is 86% female, you may change the probability of a random person there being female. You can denote this as P(Female | D) - probability of hypothesis h given data D:  P(Female | attends stanford) = 0.86. The formula is:\n",
    "\n",
    "$$\n",
    "P(A|B) = \\frac{P(A \\cap B)}\n",
    "{p(B)}\n",
    "$$\n",
    "\n",
    "Terms\n",
    "- P(h) is the prior probability before evidence\n",
    "- P(D) is the posterior probability after we observe data D. Also called conditional probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Laptop and Phone Probabilities\n",
    "\n",
    "| Name  |  Laptop |  Phone |\n",
    "|:-:|:-:|:-:|\n",
    "| Kate  | PC  | Android  |\n",
    "| Tom  | PC |  Android |\n",
    "| Harry  | PC |  Android |\n",
    "| Annika  | Mac  | iPhone  |\n",
    "| Naomi  | Mac  | Android  |\n",
    "| Joe  | Mac  | iPhone  |\n",
    "| Chakotay  | Mac  | iPhone  |\n",
    "| Neelix  | Mac  | Android  |\n",
    "| Kes  | PC | iPhone  |\n",
    "| B’Elanna  | Mac  | iPhone  |\n",
    "\n",
    "### Probability randomly selected person has an iPhone?\n",
    "p(iPhone) =  5 /10 = 0.5\n",
    "\n",
    "### Probability randomly selected person has an iPhone if they already have a Mac?\n",
    "$$\n",
    "P(iPhone|mac) = \\frac{P(mac \\cap iPhone)}\n",
    "{P(mac)}\n",
    "$$\n",
    "\n",
    "4 People have both a Mac and iPhone: $P(mac \\cap iPhone) = 4/10 = 0.4$  \n",
    "Probability random person has a mac: $P(mac) = 6/10 = 0.6$\n",
    "\n",
    "Probability that someone uses an iPhone if that person has a Mac:\n",
    "$$\n",
    "P(iPhone|mac) = \\frac{P(0.4)}\n",
    "{p(0.6)} = 0.667\n",
    "$$\n",
    "\n",
    "### Probability  someone owns a Mac if they also own an iPhone?\n",
    "$$\n",
    "P(mac|iPhone) = \\frac{P(iPhone \\cap mac)}\n",
    "{P(iPhone)} =  \\frac{P(4 / 10)}\n",
    "{P(5/10)} =  0.4 / 0.5 = 0.8\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shopping Cart Example\n",
    "Want to determine whether to show you an ad for a green tea if you're likely to purchase it.  \n",
    "P(D): probability some training data will be observed. If half the people live in the post code 88005, then the probability of someone being from that postcode P(88005) = 0.5  \n",
    "P(D|H): probability that a value holds given the hypothesis. For example, the probability that someone lives in 88005 given they've bought green tea P(88005|green tea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bought Green Tea</th>\n",
       "      <th>Bought Organic</th>\n",
       "      <th>Customer ID</th>\n",
       "      <th>Zipcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>88001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "      <td>88001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "      <td>88003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>5</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>6</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>7</td>\n",
       "      <td>88001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>8</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>9</td>\n",
       "      <td>88003</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Bought Green Tea Bought Organic  Customer ID  Zipcode\n",
       "0             True           True            0    88005\n",
       "1            False          False            1    88001\n",
       "2             True           True            2    88001\n",
       "3            False          False            3    88005\n",
       "4            False           True            4    88003\n",
       "5             True          False            5    88005\n",
       "6            False          False            6    88005\n",
       "7            False          False            7    88001\n",
       "8             True           True            8    88005\n",
       "9             True           True            9    88003"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import math\n",
    "\n",
    "customers = {   'Customer ID': range(10),\n",
    "                'Zipcode': [88005, 88001, 88001, 88005, 88003, 88005, 88005, 88001, 88005, 88003],\n",
    "                'Bought Organic': [True, False, True, False, True, False, False, False, True, True],\n",
    "                'Bought Green Tea': [True, False, True, False, False, True, False, False, True, True]}\n",
    "\n",
    "df = pd.DataFrame(customers)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probability someone who bought green tea lives in post code 88005\n",
    "P(D|H): P(88005 | green tea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bought Green Tea</th>\n",
       "      <th>Bought Organic</th>\n",
       "      <th>Customer ID</th>\n",
       "      <th>Zipcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>5</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>8</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Bought Green Tea Bought Organic  Customer ID  Zipcode\n",
       "0             True           True            0    88005\n",
       "5             True          False            5    88005\n",
       "8             True           True            8    88005"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_bought_green_tea = sum(df['Bought Green Tea'])\n",
    "bought_tea_zip = df[(df['Bought Green Tea']==True) & (df['Zipcode']==88005)]\n",
    "bought_tea_zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 / 5 = 0.6\n"
     ]
    }
   ],
   "source": [
    "num_bought_tea_and_live_88005 = len(bought_tea_zip['Zipcode'])\n",
    "\n",
    "print('{} / {} = {}'.format(num_bought_tea_and_live_88005, num_bought_green_tea, num_bought_tea_and_live_88005 / num_bought_green_tea))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opposite: Probability someone who did NOT buy green tea lives in post code 88005\n",
    "P(D|H): P(88005 | !green tea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bought Green Tea</th>\n",
       "      <th>Bought Organic</th>\n",
       "      <th>Customer ID</th>\n",
       "      <th>Zipcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>6</td>\n",
       "      <td>88005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Bought Green Tea Bought Organic  Customer ID  Zipcode\n",
       "3            False          False            3    88005\n",
       "6            False          False            6    88005"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_didnt_buy_green_tea = len(df['Bought Green Tea']) - sum(df['Bought Green Tea'])\n",
    "no_tea_zip = df[(df['Bought Green Tea']==False) & (df['Zipcode']==88005)]\n",
    "no_tea_zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 / 5 = 0.4\n"
     ]
    }
   ],
   "source": [
    "num_didnt_buy_tea_live_88005 = len(no_tea_zip['Zipcode'])\n",
    "print('{} / {} = {}'.format(num_didnt_buy_tea_live_88005, num_didnt_buy_green_tea, num_didnt_buy_tea_live_88005 / num_didnt_buy_green_tea))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probability someone being in postcode 88001?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bought Green Tea</th>\n",
       "      <th>Bought Organic</th>\n",
       "      <th>Customer ID</th>\n",
       "      <th>Zipcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>88001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "      <td>88001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>7</td>\n",
       "      <td>88001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Bought Green Tea Bought Organic  Customer ID  Zipcode\n",
       "1            False          False            1    88001\n",
       "2             True           True            2    88001\n",
       "7            False          False            7    88001"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lives_88001 = df[df['Zipcode'] == 88001]\n",
    "lives_88001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = len(lives_88001['Zipcode']) / len(df['Zipcode'])\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bayes Theorem\n",
    "Describes relationship between P(h), P(h | D), P(D), amd P(D | h)\n",
    "\n",
    "$$\n",
    "P(h | D) = \\frac{P(D | h)P(h)}{P(D)}\n",
    "$$\n",
    "\n",
    "Can use this theorem to decide between multiple hypotheses. For example, if you're given some data you can use this to determine what sport someone plays.\n",
    "\n",
    "In the tea example, we have two hypotheses:\n",
    "1. They will buy green tea: P(buy tea | 88005)\n",
    "2. They will not buy green tea: P(-buy tea | 88005)\n",
    "\n",
    "Once we calculate the probability of 0.6 that they will buy green tea, we can say its likely they will make the purchase.\n",
    "\n",
    "## Electronics Store\n",
    "3 sales fliers to send in email, they can show:\n",
    "1. Laptop\n",
    "2. Desktop\n",
    "3. Tablet\n",
    "\n",
    "Using the information we have about the customer, we want ot send the flier that will be most likely to generate a sale. They hypotheses for which flier is best:  \n",
    "$P(laptop | D) = \\frac{P(D | laptop)P(laptop)}{P(D)}$  \n",
    "$P(desktop | D) = \\frac{P(D | desktop)P(desktop)}{P(D)}$  \n",
    "$P(tablet | D) = \\frac{P(D | tablet)P(tablet)}{P(D)}$\n",
    "\n",
    "We can refer to these hypotheses as h₁, h₂, h₃ etc and they tend to be the different classes we want to predict e.g. different sports, has disease, does not have disease etc. Once we've calculated all the probabilities, we can pick the hypotheses that is most likely - called **the maximum a posteriori hypothesis** or $h_{MAP}$.\n",
    "\n",
    "$$\n",
    "h_{MAP} = arg max_{h \\in H} P(h|D)\n",
    "$$\n",
    "\n",
    "H is the set of all hypotheses, so this works out for each hypothesis out of all the hypotheses, compute the probabilities and find the one with the highest. With bayes theorem, we convert this to:\n",
    "\n",
    "$$\n",
    "h_{MAP} = arg max_{h \\in H} \\frac{P(D|h)P(h)}\n",
    "{P(D)}\n",
    "$$\n",
    "\n",
    "P(D) is independent of the hypotheses. If you find the nominator before you divide by this denominator, you can compare the hypotheses to find the largest number - most likely.\n",
    "\n",
    "## Determine if patient has cancer or not\n",
    "- 0.8% of people have some form of cancer\n",
    "- T gives binary result: positive or negative\n",
    "- Cancer present - test returns true positive 98% of the time\n",
    "- Cancer abssent - test returns correct negative result 97% of the time\n",
    "\n",
    "Therefore I can work out:\n",
    "- P(cancer) = 0.008\n",
    "- P(-cancer) = 0.992\n",
    "- P(POS|cancer) = 0.98\n",
    "- P(NEG|cancer) = 0.02\n",
    "- P(POS|-cancer) = 0.03\n",
    "- P(NEG|-cancer) = 0.97\n",
    "\n",
    "If you have a blood test done and the test result is positive, is it more likely that you have cancer than you don't?\n",
    "- P(POS|cancer): 0.98 * P(cancer): 0.008 = 0.00784\n",
    "- P(POS|-cancer): 0.03 * P(cancer) = 0.0294\n",
    "\n",
    "Most likely to not have cancer, to determine how likely the person is to have cancer:\n",
    "\n",
    "$$\n",
    "P(cancer|POS) = \\frac{0.00784}{0.00784 + 0.0294} = 0.21\n",
    "$$\n",
    "\n",
    "There is a 21% chance of having cancer.\n",
    "\n",
    "## Why use Bayes Theorem?\n",
    "\n",
    "The example of the green tea and zipcode from earlier presented us with two hypotheses:\n",
    "- P(h₁|D) = P(buygreentea|8805)\n",
    "- P(h₂|D) = P(buygreentea|8805)\n",
    "\n",
    "Can rewrite as\n",
    "\n",
    "$$\n",
    "\\frac{P(8805 | buygreentea)P(buygreentea)}\n",
    "{8805}\n",
    "$$\n",
    "\n",
    "Since we can calculate it directly from data in the table, why use the equation? Because normally its hard to calculate P(h|D) directly.\n",
    "\n",
    "## Naïve Bayes and Green Tea Problem\n",
    "Use more evidence than a single piece of data when calculating the probability with Bayes theorem. In the tea example, we have two types of evidence: zip code and whether a person has purchased organic items. To calculate the probability of a hypothesis, we multiply the individual probabilities.\n",
    "\n",
    "What is the probability that someone who lives in 8805 zipcode and bought organic items will buy green tea? \n",
    "\n",
    "P(tea|8805 & organic) = P(8805|tea)P(organic|tea)P(tea) = .6(.8)(.5) = .24\n",
    "P(-tea|8805 & organic) = P(8805|-tea)P(organic|-tea)P(-tea) = .4(.25)(.5) = .05\n",
    "\n",
    "A person who lives in 8805 and buys organic is more likely to buy green tea than not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise Wearable Example\n",
    "Company makes two models of health wearable: i100 (heart rate, GPS, etc) and the i500 (features of i100 plus blood oxygen levels, 3G connection to site). When someone buys a model, they fill out a questionaire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "questionaire = {'Main Interest': ['both', 'both', 'health', 'appearance', 'appearance', 'appearance', 'health', 'both', 'both', 'appearance', 'both', 'health', 'health', 'appearance', 'health'],\n",
    "                'Current Exercise Level': ['sedentary', 'sedentary', 'sedentary', 'active', 'moderate', 'moderate', 'moderate', 'active', 'moderate', 'active', 'active', 'active', 'sedentary', 'active', 'sedentary'],\n",
    "                'How Motivated': ['moderate', 'moderate', 'moderate', 'moderate', 'aggressive', 'aggressive', 'aggressive', 'moderate', 'aggressive', 'aggressive', 'aggressive', 'moderate', 'aggressive', 'moderate', 'moderate'],\n",
    "                'Comfotable with Tech': ['yes', 'no', 'yes', 'yes', 'yes', 'no', 'no', 'yes', 'yes', 'yes', 'no', 'no', 'yes', 'no', 'no'],\n",
    "                'Model #': ['i100', 'i100', 'i500', 'i500', 'i500', 'i100', 'i500', 'i100', 'i500', 'i500', 'i500', 'i500', 'i500', 'i100', 'i100']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Comfotable with Tech</th>\n",
       "      <th>Current Exercise Level</th>\n",
       "      <th>How Motivated</th>\n",
       "      <th>Main Interest</th>\n",
       "      <th>Model #</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>both</td>\n",
       "      <td>i100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>no</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>both</td>\n",
       "      <td>i100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>no</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>both</td>\n",
       "      <td>i100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>yes</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>both</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>both</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>no</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Comfotable with Tech Current Exercise Level How Motivated Main Interest  \\\n",
       "0                   yes              sedentary      moderate          both   \n",
       "1                    no              sedentary      moderate          both   \n",
       "2                   yes              sedentary      moderate        health   \n",
       "3                   yes                 active      moderate    appearance   \n",
       "4                   yes               moderate    aggressive    appearance   \n",
       "5                    no               moderate    aggressive    appearance   \n",
       "6                    no               moderate    aggressive        health   \n",
       "7                   yes                 active      moderate          both   \n",
       "8                   yes               moderate    aggressive          both   \n",
       "9                   yes                 active    aggressive    appearance   \n",
       "10                   no                 active    aggressive          both   \n",
       "11                   no                 active      moderate        health   \n",
       "12                  yes              sedentary    aggressive        health   \n",
       "13                   no                 active      moderate    appearance   \n",
       "14                   no              sedentary      moderate        health   \n",
       "\n",
       "   Model #  \n",
       "0     i100  \n",
       "1     i100  \n",
       "2     i500  \n",
       "3     i500  \n",
       "4     i500  \n",
       "5     i100  \n",
       "6     i500  \n",
       "7     i100  \n",
       "8     i500  \n",
       "9     i500  \n",
       "10    i500  \n",
       "11    i500  \n",
       "12    i500  \n",
       "13    i100  \n",
       "14    i100  "
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(questionaire)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using naive Bayes, which model should you recommend for someone whose:\n",
    "- main interest is health\n",
    "- current exercise level is moderate\n",
    "- moderately motivated\n",
    "- comfortable with technological devices\n",
    "\n",
    "Two hypotheses:\n",
    "- P(i100|health, modexercise, modmotivation, techcomfortable)\n",
    "- P(i500|health, modexercise, modmotivation, techcomfortable)\n",
    "\n",
    "Probabilities\n",
    "- P(i100) = d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Comfotable with Tech</th>\n",
       "      <th>Current Exercise Level</th>\n",
       "      <th>How Motivated</th>\n",
       "      <th>Main Interest</th>\n",
       "      <th>Model #</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>no</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>yes</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>both</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>both</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Comfotable with Tech Current Exercise Level How Motivated Main Interest  \\\n",
       "2                   yes              sedentary      moderate        health   \n",
       "3                   yes                 active      moderate    appearance   \n",
       "4                   yes               moderate    aggressive    appearance   \n",
       "6                    no               moderate    aggressive        health   \n",
       "8                   yes               moderate    aggressive          both   \n",
       "9                   yes                 active    aggressive    appearance   \n",
       "10                   no                 active    aggressive          both   \n",
       "11                   no                 active      moderate        health   \n",
       "12                  yes              sedentary    aggressive        health   \n",
       "\n",
       "   Model #  \n",
       "2     i500  \n",
       "3     i500  \n",
       "4     i500  \n",
       "6     i500  \n",
       "8     i500  \n",
       "9     i500  \n",
       "10    i500  \n",
       "11    i500  \n",
       "12    i500  "
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = df.shape[0]\n",
    "model_grouped = df.groupby(['Model #'])#.count()\n",
    "i100 = model_grouped.get_group('i100')\n",
    "i500 = model_grouped.get_group('i500')\n",
    "i500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'health|i100': 0.16666666666666666,\n",
       " 'health|i500': 0.4444444444444444,\n",
       " 'i100': 0.4,\n",
       " 'i500': 0.6,\n",
       " 'modexercise|i100': 0.16666666666666666,\n",
       " 'modexercise|i500': 0.3333333333333333,\n",
       " 'modmotivation|i100': 0.8333333333333334,\n",
       " 'modmotivation|i500': 0.3333333333333333,\n",
       " 'techcomfortable|i100': 0.3333333333333333,\n",
       " 'techcomfortable|i500': 0.6666666666666666}"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# conditional probability: number of people with health as priority that had also bought an i100\n",
    "P = {   'i100': i100.shape[0] / n,\n",
    "        'i500': i500.shape[0] / n,\n",
    "        'health|i100': i100[i100['Main Interest'] == 'health'].shape[0] / i100.shape[0],\n",
    "        'modexercise|i100': i100[i100['Current Exercise Level'] == 'moderate'].shape[0] / i100.shape[0],\n",
    "        'modmotivation|i100': i100[i100['How Motivated'] == 'moderate'].shape[0] / i100.shape[0],\n",
    "        'techcomfortable|i100': i100[i100['Comfotable with Tech'] == 'yes'].shape[0] / i100.shape[0],\n",
    "    \n",
    "        'health|i500': i500[i500['Main Interest'] == 'health'].shape[0] / i500.shape[0],\n",
    "        'modexercise|i500': i500[i500['Current Exercise Level'] == 'moderate'].shape[0] / i500.shape[0],\n",
    "        'modmotivation|i500': i500[i500['How Motivated'] == 'moderate'].shape[0] / i500.shape[0],\n",
    "        'techcomfortable|i500': i500[i500['Comfotable with Tech'] == 'yes'].shape[0] / i500.shape[0],\n",
    "    }\n",
    "P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of i100 0.0030864197530864196\n",
      "Probability of i500 0.019753086419753083\n",
      "The best recommendation is the i500\n"
     ]
    }
   ],
   "source": [
    "# P(i100|evidence\n",
    "p_i100 = P['health|i100'] * P['modexercise|i100'] * P['modmotivation|i100'] * P['techcomfortable|i100'] * P['i100']\n",
    "\n",
    "# P(i500|evidence\n",
    "p_i500 = P['health|i500'] * P['modexercise|i500'] * P['modmotivation|i500'] * P['techcomfortable|i500'] * P['i500']\n",
    "\n",
    "print('Probability of i100', p_i100)\n",
    "print('Probability of i500', p_i500)\n",
    "print('The best recommendation is the i500')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating Probabilities\n",
    "Probabilities in Naive Bayes are estimates of the true probabilities (probabilities from entire population). We can estimate the true probability via random representative sampling. This normally works, but when the true probability is tiny, these estimates can be poor.\n",
    "\n",
    "### 0 Probability\n",
    "When a probability is 0, it dominates the naive Bayes algorithm. Probabilities based on sample ratios can also be a biased underestimate.\n",
    "\n",
    "## Snooping Bill\n",
    "If 0 Democrats voted for a snooping bill, then if we want to predict the probability of someone else being a Democrat based on all the bills they voted on and they voted against the snooping bill, we'll have 100% confidence that they are a Democrat.\n",
    "\n",
    "P(votedNo|Democrat) = number that are both democrats and voted on snooping bill / Number that are democrats\n",
    "\n",
    "$$\n",
    "P(x|y = \\frac{x_c}{n}\n",
    "$$\n",
    "\n",
    "n is the total number of instances of class y, divided by the total number of classes that have value x. When $n_c=0$ we can have a problem. To solve this, we can rewrite the equation:\n",
    "\n",
    "$$\n",
    "P(x|y = \\frac{x_c + mp}{n + m}\n",
    "$$\n",
    "\n",
    "- m = constant (equivalent sample size). Value can vary, but you can use the number of different values an attribute takes: e.g. if the response is boolean, we can say m=2\n",
    "- p = prior estimate of probability. We usually assume uniform probability, so if we assume there's a 50% chance of voting yes or no, then p=1/2\n",
    "\n",
    "## Numbers\n",
    "\n",
    "Naive Bayes uses categorical data (data in discrete categories). If your data is along a scale, you need to be able to group things so you can count them. You have two options:\n",
    "\n",
    "1. Make categories: Discretize a continuous attribute\n",
    "2. Gaussian Distributions and use a probability density function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Comfotable with Tech</th>\n",
       "      <th>Current Exercise Level</th>\n",
       "      <th>How Motivated</th>\n",
       "      <th>Main Interest</th>\n",
       "      <th>Model #</th>\n",
       "      <th>Income ($1000s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>both</td>\n",
       "      <td>i100</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>no</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>both</td>\n",
       "      <td>i100</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "      <td>125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i100</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>no</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "      <td>150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>both</td>\n",
       "      <td>i100</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>yes</td>\n",
       "      <td>moderate</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>both</td>\n",
       "      <td>i500</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>active</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i500</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>both</td>\n",
       "      <td>i500</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>health</td>\n",
       "      <td>i500</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>no</td>\n",
       "      <td>active</td>\n",
       "      <td>moderate</td>\n",
       "      <td>appearance</td>\n",
       "      <td>i100</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>no</td>\n",
       "      <td>sedentary</td>\n",
       "      <td>moderate</td>\n",
       "      <td>health</td>\n",
       "      <td>i100</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Comfotable with Tech Current Exercise Level How Motivated Main Interest  \\\n",
       "0                   yes              sedentary      moderate          both   \n",
       "1                    no              sedentary      moderate          both   \n",
       "2                   yes              sedentary      moderate        health   \n",
       "3                   yes                 active      moderate    appearance   \n",
       "4                   yes               moderate    aggressive    appearance   \n",
       "5                    no               moderate    aggressive    appearance   \n",
       "6                    no               moderate    aggressive        health   \n",
       "7                   yes                 active      moderate          both   \n",
       "8                   yes               moderate    aggressive          both   \n",
       "9                   yes                 active    aggressive    appearance   \n",
       "10                   no                 active    aggressive          both   \n",
       "11                   no                 active      moderate        health   \n",
       "12                  yes              sedentary    aggressive        health   \n",
       "13                   no                 active      moderate    appearance   \n",
       "14                   no              sedentary      moderate        health   \n",
       "\n",
       "   Model #  Income ($1000s)  \n",
       "0     i100               60  \n",
       "1     i100               75  \n",
       "2     i500               90  \n",
       "3     i500              125  \n",
       "4     i500              100  \n",
       "5     i100               90  \n",
       "6     i500              150  \n",
       "7     i100               85  \n",
       "8     i500              100  \n",
       "9     i500              120  \n",
       "10    i500               95  \n",
       "11    i500               90  \n",
       "12    i500               85  \n",
       "13    i100               70  \n",
       "14    i100               45  "
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Income ($1000s)'] = [60, 75, 90, 125, 100, 90, 150, 85, 100, 120, 95, 90, 85, 70, 45]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to identify the typical purchaser of an i500, we could find their average income:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "106.11111111111111"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "income_i500 = df[df['Model #']=='i500']['Income ($1000s)']\n",
    "income_i500.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard Deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.10773452317095"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def standard_deviation(x):\n",
    "    return math.sqrt( (sum( [math.pow(xi - x.mean(), 2) for xi in x] )) / (len(x)) )\n",
    "\n",
    "sd = standard_deviation(income_i500)\n",
    "sd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-0.80123949779352976,\n",
       " 0.93938423879241395,\n",
       " -0.30391843019754589,\n",
       " 2.1826869077823736,\n",
       " -0.30391843019754589,\n",
       " 0.69072370499442193,\n",
       " -0.55257896399553785,\n",
       " -0.80123949779352976,\n",
       " -1.0499000315915217]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def standard_score(x, std):\n",
    "    return [((xi - x.mean()) / std) for xi in x]\n",
    "    \n",
    "z_scores = standard_score(income_i500, sd)\n",
    "z_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Population vs Sample Standard Deviation\n",
    "If you have data on the complete population, you can use the SD formula above, but if you have a sample, then you need to modify it by dividing by n-1.\n",
    "\n",
    "## Gaussian Distribution\n",
    "Otherwise called the normal distribution or bell curve. People tend to assume that the data follows a normal distribution which has the following properties:\n",
    "- 68% of instances fall within 1 std of the mean\n",
    "- 95% within 2 std of the mean\n",
    "\n",
    "Since our mean was 106.1 and the std was 21.3, 95% of the people who purchase an i500 earn between \\$42k and \\$149k. So if you asked the likelihood that someone who bought an i500 earns $100k, you'd say it was pretty likely. We can calculate the probability with the following equation:\n",
    "\n",
    "The probability equation for this is:\n",
    "\n",
    "$$\n",
    "P(x_i|y_j) = \\frac{1}\n",
    "{\\sqrt{2\\pi\\sigma}}\n",
    "e^{\\frac{-(x_i-\\mu_ij)^2}\n",
    "{2\\sigma^2_{ij}}}\n",
    "$$\n",
    "\n",
    "If we are interested in calculated P(100k|i500): the probability that a person earns \\$100k given they bought an i500. We've already calculated the mean of the income of people who bought the i500 and also their standard deviation. Mean is represented as $\\mu$ and the std as $\\sigma$.\n",
    "\n",
    "- $\\mu_{ij}$ = 106.111\n",
    "- $\\sigma_{ij}$ = 21.327\n",
    "- $x_i$ = 100\n",
    "- e = constant that is the base of the natural logarithm: value ~ 2.718\n",
    "\n",
    "$$\n",
    "P(x_i|y_j) = \\frac{1}\n",
    "{\\sqrt{2\\pi(21.327)}}\n",
    "e^{\\frac{-(100-106.111)^2}\n",
    "{2(21.327)^2_{ij}}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "P(x_i|y_j) = \\frac{1}\n",
    "{53.458}\n",
    "e^{-0.0411}\n",
    "$$\n",
    "\n",
    "$$\n",
    "P(x_i|y_j) = 0.0180\n",
    "$$\n",
    "\n",
    "The probability that the income of the person who bought an i500 is \\$100k is 0.0180."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayes vs KNN\n",
    "### Bayes\n",
    "- Simple implementation (counting things)\n",
    "- Requires less training data\n",
    "- Works well and is fast\n",
    "- Cannot learn interactions between features - e.g. cannot learn that I like foods with cheese and foods with rice but not with both\n",
    "\n",
    "### KNN\n",
    "- Simple, no assumptions about structure in data\n",
    "- Requires large amount of memory for training set\n",
    "- Useful when training set is large\n",
    "- KNN is flexible and can be used in a wide variety of fields from recommendation engines, proteomics, image classification etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}