Email_spam_filter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 - Most Common Words Extraction\n",
    "\n",
    "The dataset selected is classified into 6 subdirectories, each with a ham and a spam directories. Each on contains ham and spam **.txt** files respectively. By using the **os** library, the dataset was filtered to a single dictionary containing the words with their respective number of occurance in all files. \n",
    "\n",
    "This was done by spliting each file's content to individual strings and adding them to a list. Next, a counter function was used to make count each word occurance in that list.\n",
    "\n",
    "It was found that there are special characters in the spam **.txt** files that the UTF-8 encoding does not classify as strings. To ignore this error, latin encoding was used to ensure that every character is treated as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 55
    },
    "colab_type": "code",
    "id": "pBIRsXF3AJWm",
    "outputId": "baa5cf51-371e-48c6-c4f1-bfa421862dca"
   },
   "outputs": [],
   "source": [
    "#Importing needed libraries\n",
    "import os\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import confusion_matrix \n",
    "\n",
    "def countFiles(rootDir): #Function returning the number of files present in the directories and its sub directories\n",
    "    i = 0\n",
    "    for directories, subdirectories, files in os.walk(rootDir):\n",
    "        for filename in files:    \n",
    "            with open(os.path.join(directories, filename),encoding=\"latin-1\") as f:\n",
    "                i += 1\n",
    "    return i\n",
    "\n",
    "def mostCommonWords(N, wordList, rootDir): #Function returning N most commonly used word\n",
    "    for directories, subdirectories, files in os.walk(rootDir):\n",
    "        for filename in files:    #For every file\n",
    "            with open(os.path.join(directories, filename),encoding=\"latin-1\") as f:\n",
    "                wordsInFile = f.read().split() #Split the file's content into word strings \n",
    "                wordList+= wordsInFile #Add every word to the global word list\n",
    "\n",
    "    dictionary = Counter(wordList) #Defining the common word dictionary\n",
    "    for word in list(dictionary): #Looping to delete non-alphabetic and single characters\n",
    "        if word.isalpha() == False: \n",
    "            del dictionary[word]\n",
    "        elif len(word) == 1:\n",
    "            del dictionary[word]\n",
    "\n",
    "    dictionary = dictionary.most_common(N) #Outputting most common words with their frequencies based on user preference\n",
    "    return (dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2  - Feature Extraction\n",
    "\n",
    "The process of feature extraction is one which a given set of data is reduced from a higher dimension to the lower dimension to reduce the processing power and save resources when it comes to doing data analysis.\n",
    "Here in this dataset, the emails were filtered to be in a two-dimension format (i.e dictionary) with the most commonly used **N** words. One can visualize such data into a huge, one-dimension vector or list of integers, each representing the number of occurances of words relevant to the dictionary, in a given email. \n",
    "\n",
    "For example, the dictionary output had the word \"**is**\" among the most occurring words in the email. Ultimately, if one email contained the message \"Hi, my name is Hossam Elghamry\", then the N-sized vector will be outputting [0,0,0...1,0,0,0]. The ***1*** integer value is the number of occurances of the \"**is**\" word in that given email, with the integer's position in the list being respective to that of the \"**is**\" in the dictionary.\n",
    "\n",
    "That said, we can represent all the feature extraction vectors in one ***MxN*** matrix. ***M*** being the number of emails processed and ***N*** being the count of the most occurring words (i.e the length of the output dictionary in **Part 1**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "def featureExtraction(commonWordsDictionary, rootDir): \n",
    "    featuresMatrix = np.zeros((fileCount,mostCommonWordsCount)) #Making a (number of emails X number \n",
    "                                                                #of most occuring words) matrix\n",
    "    fileNum = 0; #Row incrementation value\n",
    "    for directories, subdirectories, files in os.walk(rootDir):\n",
    "        for filename in files:      \n",
    "            with open(os.path.join(directories, filename),encoding=\"latin-1\") as f:\n",
    "                words = f.read().split()\n",
    "                for word in words: #for every word in every file\n",
    "                    wordNum = 0 \n",
    "                    for i,d in enumerate(commonWordsDictionary): #search for it in the dictionary of repeated words\n",
    "                        if d[0] == word:\n",
    "                        #if found, put the number of occurances of that current word by\n",
    "                        #using Count() function, in the feature matrix's word column, respective to the document's row\n",
    "                            wordNum = i\n",
    "                            featuresMatrix[fileNum,wordNum] = words.count(word) \n",
    "        fileNum = fileNum + 1 #Next document index (i.e next row in the matrix)\n",
    "    return featuresMatrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3 - Extracting Labeled Feature Vector per Training Email to One Single Two-Dimensional Matrix\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [3. 5. 5. ... 2. 1. 1.]\n",
      " [1. 5. 1. ... 1. 1. 2.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "trainingDir = \"Train\" #Insert training directory here respective to the notebook file\n",
    "wordList = [] #List containing all words used, with duplicates\n",
    "mostCommonWordsCount = 10 #Desired number of the most occurring words in the output dictionary\n",
    "\n",
    "fileCount = countFiles(trainingDir) \n",
    "commonWordsDictionary = mostCommonWords(mostCommonWordsCount, wordList, trainingDir)\n",
    "#print (commonWordsDictionary)\n",
    "\n",
    "trainingLabels = np.zeros(4080) #Initiallizing labels\n",
    "trainingLabels[2880:4080] = 1 #Labeling spam training emails by \"1\"\n",
    "trainingMatrix = featureExtraction(commonWordsDictionary, trainingDir)\n",
    "print(trainingMatrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4 - Defining and Training Naive Bayes Classifier\n",
    "Naive Bayes is a simple technique for constructing classifiers: models that assign class labels to problem instances, represented as vectors of feature values, where the class labels are drawn from some finite set. There is not a single algorithm for training such classifiers, but a family of algorithms based on a common principle: all naive Bayes classifiers assume that the value of a particular feature is independent of the value of any other feature, given the class variable.\n",
    "\n",
    "The reason for using such classifier is that it directly links the test outputs with the given labels and comparison is fairly easily at that point using **Confussion Matrixes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB_Classifier = MultinomialNB()\n",
    "NB_Classifier.fit(trainingMatrix, trainingLabels) #Training the model using the training emails"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5 - Testing the Trained Model using the Test Set Defined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1120\n",
      "[[720   1]\n",
      " [399   0]]\n"
     ]
    }
   ],
   "source": [
    "testingDir = \"Test\" #Insert test directory here respective to the notebook file\n",
    "fileCount = countFiles(testingDir)  \n",
    "testingMatrix = featureExtraction(commonWordsDictionary, testingDir)\n",
    "testLabels = np.zeros(1120) #Initiallizing labels\n",
    "testLabels[720:1119]= 1 #Labeling spam training emails by \"1\"\n",
    "\n",
    "testResult = NB_Classifier.predict(testingMatrix) #Testing the model \n",
    "print (confusion_matrix(testLabels,testResult)) #Comparing the results with the actual labels using confusion matrix"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Welcome To Colaboratory",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}