SPAM Filter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Spam Filter with Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What the project is about:** In this project, I'll build a spam filter for SMS messages using the multinomial Naive Bayes algorithm. \n",
    "\n",
    "To train the algorithm, I'll use a dataset of 5,572 SMS messages that have already been classified by humans. The dataset was put together by Tiago A. Almeida and José María Gómez Hidalgo, and it can be downloaded from the the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection). The data collection process is described in more details on [this page](http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/#composition), where you can also find some of the papers authored by Tiago A. Almeida and José María Gómez Hidalgo.\n",
    "    \n",
    "**Goal of the project:** My goal is to write a program that classifies new messages with an accuracy greater than 80% — so I expect that more than 80% of the new messages will be classified correctly as spam or ham (non-spam)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll start by reading in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5572, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>SMS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Label                                                SMS\n",
       "0   ham  Go until jurong point, crazy.. Available only ...\n",
       "1   ham                      Ok lar... Joking wif u oni...\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3   ham  U dun say so early hor... U c already then say...\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "sms_spam = pd.read_csv('SMSSpamCollection', sep='\\t', header=None, names=['Label', 'SMS'])\n",
    "\n",
    "print(sms_spam.shape)\n",
    "sms_spam.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Below, I see that about 87% of the messages are ham, and the remaining 13% are spam. This sample looks representative, since in practice most messages that people receive are ham."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ham     0.865937\n",
       "spam    0.134063\n",
       "Name: Label, dtype: float64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sms_spam['Label'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and Test Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I am now going to split the dataset into a training and a test set, where the training set accounts for 80% of the data, and the test set for the remaining 20%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4458, 2)\n",
      "(1114, 2)\n"
     ]
    }
   ],
   "source": [
    "# Randomize the dataset\n",
    "data_randomized = sms_spam.sample(frac=1, random_state=1)\n",
    "\n",
    "# Calculate index for split\n",
    "training_test_index = round(len(data_randomized) * 0.8)\n",
    "\n",
    "# Training/Test split\n",
    "training_set = data_randomized[:training_test_index].reset_index(drop=True)\n",
    "test_set = data_randomized[training_test_index:].reset_index(drop=True)\n",
    "\n",
    "print(training_set.shape)\n",
    "print(test_set.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "I'll now analyze the percentage of spam and ham messages in the training and test sets. I expect the percentages to be close to what I have in the full dataset, where about 87% of the messages are ham, and the remaining 13% are spam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ham     0.86541\n",
       "spam    0.13459\n",
       "Name: Label, dtype: float64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_set['Label'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ham     0.868043\n",
       "spam    0.131957\n",
       "Name: Label, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set['Label'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The results look good! I'll now move on to cleaning the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate all the probabilities required by the algorithm, I'll first need to perform a bit of data cleaning to bring the data in a format that will allow me to extract easily all the information I need.\n",
    "\n",
    "Essentially, I want to bring data to this format:\n",
    "\n",
    "![img](https://camo.githubusercontent.com/cb0540b486b147de5411006a7c1717a0256871452141d70a4ab1b5d42acb3f55/68747470733a2f2f64712d636f6e74656e742e73332e616d617a6f6e6177732e636f6d2f3433332f637067705f646174617365745f332e706e67)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Letter Case and Punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll begin with removing all the punctuation and converting every letter to lower case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>SMS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Yep, by the pretty sculpture</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Yes, princess. Are you going to make me moan?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ham</td>\n",
       "      <td>Welp apparently he retired</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>Havent.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>I forgot 2 ask ü all smth.. There's a card on ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Label                                                SMS\n",
       "0   ham                       Yep, by the pretty sculpture\n",
       "1   ham      Yes, princess. Are you going to make me moan?\n",
       "2   ham                         Welp apparently he retired\n",
       "3   ham                                            Havent.\n",
       "4   ham  I forgot 2 ask ü all smth.. There's a card on ..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Before cleaning\n",
    "training_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>SMS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>yep  by the pretty sculpture</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>yes  princess  are you going to make me moan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ham</td>\n",
       "      <td>welp apparently he retired</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>havent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>i forgot 2 ask ü all smth   there s a card on ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Label                                                SMS\n",
       "0   ham                       yep  by the pretty sculpture\n",
       "1   ham      yes  princess  are you going to make me moan \n",
       "2   ham                         welp apparently he retired\n",
       "3   ham                                            havent \n",
       "4   ham  i forgot 2 ask ü all smth   there s a card on ..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# After cleaning\n",
    "training_set['SMS'] = training_set['SMS'].str.replace('\\W', ' ')\n",
    "training_set['SMS'] = training_set['SMS'].str.lower()\n",
    "training_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the Vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now move on to creating the vocabulary, which in this context means a list with all the unique words in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_set['SMS'] = training_set['SMS'].str.split()\n",
    "\n",
    "vocabulary = []\n",
    "for sms in training_set['SMS']:\n",
    "    for word in sms:\n",
    "        vocabulary.append(word)\n",
    "        \n",
    "vocabulary = list(set(vocabulary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "It looks like there are 7,783 unique words in all the messages of the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7783"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Final Training Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm now going to use the vocabulary I just created to make the desired data transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_counts_per_sms = {unique_word: [0] * len(training_set['SMS']) for unique_word in vocabulary}\n",
    "\n",
    "for index, sms in enumerate(training_set['SMS']):\n",
    "    for word in sms:\n",
    "        word_counts_per_sms[word][index] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>000pes</th>\n",
       "      <th>008704050406</th>\n",
       "      <th>0089</th>\n",
       "      <th>01223585334</th>\n",
       "      <th>02</th>\n",
       "      <th>0207</th>\n",
       "      <th>02072069400</th>\n",
       "      <th>...</th>\n",
       "      <th>zindgi</th>\n",
       "      <th>zoe</th>\n",
       "      <th>zogtorius</th>\n",
       "      <th>zouk</th>\n",
       "      <th>zyada</th>\n",
       "      <th>é</th>\n",
       "      <th>ú1</th>\n",
       "      <th>ü</th>\n",
       "      <th>〨ud</th>\n",
       "      <th>鈥</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 7783 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  00  000  000pes  008704050406  0089  01223585334  02  0207  02072069400  \\\n",
       "0  0   0    0       0             0     0            0   0     0            0   \n",
       "1  0   0    0       0             0     0            0   0     0            0   \n",
       "2  0   0    0       0             0     0            0   0     0            0   \n",
       "3  0   0    0       0             0     0            0   0     0            0   \n",
       "4  0   0    0       0             0     0            0   0     0            0   \n",
       "\n",
       "  ...  zindgi  zoe  zogtorius  zouk  zyada  é  ú1  ü  〨ud  鈥  \n",
       "0 ...       0    0          0     0      0  0   0  0    0  0  \n",
       "1 ...       0    0          0     0      0  0   0  0    0  0  \n",
       "2 ...       0    0          0     0      0  0   0  0    0  0  \n",
       "3 ...       0    0          0     0      0  0   0  0    0  0  \n",
       "4 ...       0    0          0     0      0  0   0  2    0  0  \n",
       "\n",
       "[5 rows x 7783 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_counts = pd.DataFrame(word_counts_per_sms)\n",
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>SMS</th>\n",
       "      <th>0</th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>000pes</th>\n",
       "      <th>008704050406</th>\n",
       "      <th>0089</th>\n",
       "      <th>01223585334</th>\n",
       "      <th>02</th>\n",
       "      <th>...</th>\n",
       "      <th>zindgi</th>\n",
       "      <th>zoe</th>\n",
       "      <th>zogtorius</th>\n",
       "      <th>zouk</th>\n",
       "      <th>zyada</th>\n",
       "      <th>é</th>\n",
       "      <th>ú1</th>\n",
       "      <th>ü</th>\n",
       "      <th>〨ud</th>\n",
       "      <th>鈥</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>[yep, by, the, pretty, sculpture]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>[yes, princess, are, you, going, to, make, me,...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ham</td>\n",
       "      <td>[welp, apparently, he, retired]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>[havent]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>[i, forgot, 2, ask, ü, all, smth, there, s, a,...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 7785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Label                                                SMS  0  00  000  \\\n",
       "0   ham                  [yep, by, the, pretty, sculpture]  0   0    0   \n",
       "1   ham  [yes, princess, are, you, going, to, make, me,...  0   0    0   \n",
       "2   ham                    [welp, apparently, he, retired]  0   0    0   \n",
       "3   ham                                           [havent]  0   0    0   \n",
       "4   ham  [i, forgot, 2, ask, ü, all, smth, there, s, a,...  0   0    0   \n",
       "\n",
       "   000pes  008704050406  0089  01223585334  02 ...  zindgi  zoe  zogtorius  \\\n",
       "0       0             0     0            0   0 ...       0    0          0   \n",
       "1       0             0     0            0   0 ...       0    0          0   \n",
       "2       0             0     0            0   0 ...       0    0          0   \n",
       "3       0             0     0            0   0 ...       0    0          0   \n",
       "4       0             0     0            0   0 ...       0    0          0   \n",
       "\n",
       "   zouk  zyada  é  ú1  ü  〨ud  鈥  \n",
       "0     0      0  0   0  0    0  0  \n",
       "1     0      0  0   0  0    0  0  \n",
       "2     0      0  0   0  0    0  0  \n",
       "3     0      0  0   0  0    0  0  \n",
       "4     0      0  0   0  2    0  0  \n",
       "\n",
       "[5 rows x 7785 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_set_clean = pd.concat([training_set, word_counts], axis=1)\n",
    "training_set_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating Constants First"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I am now done with cleaning the training set and can begin creating the spam filter. The Naive Bayes algorithm will need to answer two probability questions to be able to classify new messages (which I am unable to display here).\n",
    "\n",
    "Also, to calculate P(wi|Spam) and P(wi|Ham) inside the aforementioned formulas/probability questions, I'll need to use these equations:\n",
    "\n",
    "- P(wi|Spam) = [(Nwi|Spam) + α] / [NSpam + (α * NVocabulary)]\n",
    "- P(wi|Ham) = [(Nwi|Ham) + α] / [NHam + (α * NVocabulary)]\n",
    "\n",
    "Some of the terms in the equations above will have the same value for every new message. I can calculate the value of these terms once and avoid doing the computations again when a new messages comes in. Below, I'll use my training set to calculate:\n",
    "\n",
    "- P(Spam) and P(Ham)\n",
    "- NSpam, NHam, NVocabulary\n",
    "\n",
    "I'll also use Laplace smoothing and set α = 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Isolating spam and ham messages first\n",
    "spam_messages = training_set_clean[training_set_clean['Label'] == 'spam']\n",
    "ham_messages = training_set_clean[training_set_clean['Label'] == 'ham']\n",
    "\n",
    "# P(Spam) and P(Ham)\n",
    "p_spam = len(spam_messages) / len(training_set_clean)\n",
    "p_ham = len(ham_messages) / len(training_set_clean)\n",
    "\n",
    "# N_Spam\n",
    "n_words_per_spam_message = spam_messages['SMS'].apply(len)\n",
    "n_spam = n_words_per_spam_message.sum()\n",
    "\n",
    "# N_Ham\n",
    "n_words_per_ham_message = ham_messages['SMS'].apply(len)\n",
    "n_ham = n_words_per_ham_message.sum()\n",
    "\n",
    "# N_Vocabulary\n",
    "n_vocabulary = len(vocabulary)\n",
    "\n",
    "# Laplace smoothing\n",
    "alpha = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that I have the constant terms calculated above, I can move on with calculating the parameters P(wi|Spam) and P(wi|Ham). Each parameter will thus be a conditional probability value associated with each word in the vocabulary.\n",
    "\n",
    "The parameters are calculated using the formulas:\n",
    "\n",
    "- P(wi|Spam) = [(Nwi|Spam) + α] / [NSpam + (α * NVocabulary)]\n",
    "- P(wi|Ham) = [(Nwi|Ham) + α] / [NHam + (α * NVocabulary)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initiate parameters\n",
    "parameters_spam = {unique_word:0 for unique_word in vocabulary}\n",
    "parameters_ham = {unique_word:0 for unique_word in vocabulary}\n",
    "\n",
    "# Calculate parameters\n",
    "for word in vocabulary:\n",
    "    n_word_given_spam = spam_messages[word].sum()   # spam_messages already defined in a cell above\n",
    "    p_word_given_spam = (n_word_given_spam + alpha) / (n_spam + alpha*n_vocabulary)\n",
    "    parameters_spam[word] = p_word_given_spam\n",
    "    \n",
    "    n_word_given_ham = ham_messages[word].sum()   # ham_messages already defined in a cell above\n",
    "    p_word_given_ham = (n_word_given_ham + alpha) / (n_ham + alpha*n_vocabulary)\n",
    "    parameters_ham[word] = p_word_given_ham"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifying A New Message"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that I have all my parameters calculated, I can start creating the spam filter. The spam filter can be understood as a function that:\n",
    "\n",
    "- Takes in as input a new message (w1, w2, ..., wn).\n",
    "- Calculates P(Spam|w1, w2, ..., wn) and P(Ham|w1, w2, ..., wn).\n",
    "- Compares the values of P(Spam|w1, w2, ..., wn) and P(Ham|w1, w2, ..., wn), and:\n",
    "    - If P(Ham|w1, w2, ..., wn) > P(Spam|w1, w2, ..., wn), then the message is classified as ham.\n",
    "    - If P(Ham|w1, w2, ..., wn) < P(Spam|w1, w2, ..., wn), then the message is classified as spam.\n",
    "    - If P(Ham|w1, w2, ..., wn) = P(Spam|w1, w2, ..., wn), then the algorithm may request human help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def classify(message):\n",
    "    '''\n",
    "    message: a string\n",
    "    '''\n",
    "    \n",
    "    message = re.sub('\\W', ' ', message)\n",
    "    message = message.lower().split()\n",
    "    \n",
    "    p_spam_given_message = p_spam\n",
    "    p_ham_given_message = p_ham\n",
    "\n",
    "    for word in message:\n",
    "        if word in parameters_spam:\n",
    "            p_spam_given_message *= parameters_spam[word]\n",
    "            \n",
    "        if word in parameters_ham:\n",
    "            p_ham_given_message *= parameters_ham[word]\n",
    "            \n",
    "    print('P(Spam|message):', p_spam_given_message)\n",
    "    print('P(Ham|message):', p_ham_given_message)\n",
    "    \n",
    "    if p_ham_given_message > p_spam_given_message:\n",
    "        print('Label: Ham')\n",
    "    elif p_ham_given_message < p_spam_given_message:\n",
    "        print('Label: Spam')\n",
    "    else:\n",
    "        print('Equal proabilities, have a human classify this!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P(Spam|message): 1.3481290211300841e-25\n",
      "P(Ham|message): 1.9368049028589875e-27\n",
      "Label: Spam\n"
     ]
    }
   ],
   "source": [
    "classify('WINNER!! This is the secret code to unlock the money: C3421.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P(Spam|message): 2.4372375665888117e-25\n",
      "P(Ham|message): 3.687530435009238e-21\n",
      "Label: Ham\n"
     ]
    }
   ],
   "source": [
    "classify(\"Sounds good, Tom, then see u there\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measuring the Spam Filter's Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two results above look promising, but let's see how well the filter does on the test set, which has 1,114 messages.\n",
    "\n",
    "I'll start by writing a function that returns classification labels instead of printing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def classify_test_set(message):    \n",
    "    '''\n",
    "    message: a string\n",
    "    '''\n",
    "    \n",
    "    message = re.sub('\\W', ' ', message)\n",
    "    message = message.lower().split()\n",
    "    \n",
    "    p_spam_given_message = p_spam\n",
    "    p_ham_given_message = p_ham\n",
    "\n",
    "    for word in message:\n",
    "        if word in parameters_spam:\n",
    "            p_spam_given_message *= parameters_spam[word]\n",
    "            \n",
    "        if word in parameters_ham:\n",
    "            p_ham_given_message *= parameters_ham[word]\n",
    "    \n",
    "    if p_ham_given_message > p_spam_given_message:\n",
    "        return 'ham'\n",
    "    elif p_spam_given_message > p_ham_given_message:\n",
    "        return 'spam'\n",
    "    else:\n",
    "        return 'needs human classification'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now that I have a function that returns labels instead of printing them, I can use it to create a new column in my test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>SMS</th>\n",
       "      <th>predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Later i guess. I needa do mcat study too.</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>But i haf enuff space got like 4 mb...</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Had your mobile 10 mths? Update to latest Oran...</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>All sounds good. Fingers . Makes it difficult ...</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>All done, all handed in. Don't know if mega sh...</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Label                                                SMS predicted\n",
       "0   ham          Later i guess. I needa do mcat study too.       ham\n",
       "1   ham             But i haf enuff space got like 4 mb...       ham\n",
       "2  spam  Had your mobile 10 mths? Update to latest Oran...      spam\n",
       "3   ham  All sounds good. Fingers . Makes it difficult ...       ham\n",
       "4   ham  All done, all handed in. Don't know if mega sh...       ham"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set['predicted'] = test_set['SMS'].apply(classify_test_set)\n",
    "test_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Next, I'll write a function to measure the accuracy of my spam filter to find out how well the spam filter does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correct: 1100\n",
      "Incorrect: 14\n",
      "Accuracy: 0.9874326750448833\n"
     ]
    }
   ],
   "source": [
    "correct = 0\n",
    "total = test_set.shape[0]\n",
    "    \n",
    "for row in test_set.iterrows():\n",
    "    row = row[1]\n",
    "    if row['Label'] == row['predicted']:\n",
    "        correct += 1\n",
    "        \n",
    "print('Correct:', correct)\n",
    "print('Incorrect:', total - correct)\n",
    "print('Accuracy:', correct/total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The accuracy is close to 98.74%, which is really good. My spam filter looked at 1,114 messages that it hasn't seen in training, and classified 1,100 correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, I managed to build a spam filter for SMS messages using the multinomial Naive Bayes algorithm. The filter had an accuracy of 98.74% on the test set I used, which is a pretty good result. My initial goal was an accuracy of over 80%, and I managed to do way better than that.\n",
    "\n",
    "Next steps include:\n",
    "\n",
    "- Analyze the 14 messages that were classified incorrectly and try to figure out why the algorithm classified them incorrectly\n",
    "- Make the filtering process more complex by making the algorithm sensitive to letter case"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}