P5_IdentifyFraudFromEnronEmail.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identify Fraud From Enron Email\n",
    "*by Adrian Liaw*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background <small>(Question #1)</small>\n",
    "Enron Corporation was an American energy, commodities, and services company. Enron was one of the world's major electricity, natural gas, communications and pulp and paper companies. During 2001, it suffered one of the largest bankruptcies in the history, because of the revelations of Enron scandal, an institutionalised, systematic, and creatively planned accounting fraud. Some key people who involved in the scandal were Kenneth Lay, Jeffrey Skilling, Andrew Fastow, etc.\n",
    "\n",
    "The goal of this project is to build a person-of-interest (POI) identifier using machine learning which identifies persons who might involved in the scandal. The dataset used in this project contains financial data of Enron employees and their e-mails. Machine learning is useful here because there are too many e-mails to analyse, and people usually analyse this type of data using human intuition, which can't build a systematic model to predict things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Overview\n",
    "\n",
    "### Summary\n",
    "This table gives us the total number of data points, number of POIs/non-POIs, and number of features.\n",
    "\n",
    "| Characteristic | Quantity |\n",
    "|----------------|----------|\n",
    "| data points    | 146      |\n",
    "| POIs           | 18       |\n",
    "| non-POIs       | 128      |\n",
    "| features       | 21       |\n",
    "\n",
    "The allocation across two classes (POI / non-POI) is highly skewed, only approximately 12% of the data points are labeled as POI.  \n",
    "There are 21 features in the dataset, one of them is the label for our machine learning algorithms, `poi`. Another feature, `email_address`, is definitely not an actual feature that will be used by the algorithms, but a reference to the e-mails, I'm going to add tons of \"word features\" generated by parsing those e-mails."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing Values\n",
    "This table gives us the percentage of data points without value for each feature.\n",
    "\n",
    "| Feature                   | Percentage of data points without value |\n",
    "| ------------------------- | --------------------------------------- |\n",
    "| loan_advances             | 0.97                                    |\n",
    "| restricted_stock_deferred | 0.88                                    |\n",
    "| director_fees             | 0.88                                    |\n",
    "| deferral_payments         | 0.73                                    |\n",
    "| deferred_income           | 0.66                                    |\n",
    "| long_term_incentive       | 0.55                                    |\n",
    "| bonus                     | 0.44                                    |\n",
    "| from_messages             | 0.41                                    |\n",
    "| from_this_person_to_poi   | 0.41                                    |\n",
    "| shared_receipt_with_poi   | 0.41                                    |\n",
    "| to_messages               | 0.41                                    |\n",
    "| from_poi_to_this_person   | 0.41                                    |\n",
    "| other                     | 0.36                                    |\n",
    "| expenses                  | 0.35                                    |\n",
    "| salary                    | 0.35                                    |\n",
    "| exercised_stock_options   | 0.30                                    |\n",
    "| restricted_stock          | 0.25                                    |\n",
    "| email_address             | 0.24                                    |\n",
    "| total_stock_value         | 0.14                                    |\n",
    "| total_payments            | 0.14                                    |\n",
    "| poi                       | 0.00                                    |\n",
    "\n",
    "Some features have many missing values, the table already sorted the percentages in a reverse order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers\n",
    "There are 2 outliers in the dataset, `TOTAL` and `THE TRAVEL AGENCY IN THE PARK`, these two data points are apparently not human, they're definitely outliers, thus removing them from the dataset. Now the dataset has 144 data points. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features <small>(Question #2)</small>\n",
    "\n",
    "### Feature Engineering\n",
    "I created 2 new features based on the original dataset, they are `fraction_from_poi` and `fraction_to_poi`. `fraction_from_poi` is the fraction of all messages to this person that come from POIs, `fraction_to_poi` is the fraction of all messages from this person that are sent to POIs. The reason why I created these two features is because, there might be a possibility that, given a person who has high portion of e-mails sent to / recieved from POIs, the person himself / herself is a POI. `fraction_from_poi` depends on `from_poi_to_this_person` and `to_messages`, so if any of the two features is NaN, `fraction_from_poi` will be NaN; same case for `fraction_to_poi`, it depends on `from_this_person_to_poi` and `from_messages`.\n",
    "\n",
    "I also parsed content of all applicable e-mails, vectorised them into numerical values which became features in the dataset. I used `TfidfVectorizer` provided by scikit-learn, it generated a 144x73618 matrix, which means there are 73618 word features in total. Sum all those up, including financial, e-mails and word features, we've got 73639 features. Of course we're not going to use them all in the final model, we'll do some feature selection first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Selection\n",
    "I didn't select the features by hand, I used `SelectKBest` provided by scikit-learn instead, it automatically selected features for me. I selected 9 best features, 9 for `k` is the result of running a `GridSearchCV` on a `Pipeline` of `SelectKBest` plus `LinearSVC`, it gave the best $F_1$ score. Note that `LinearSVC` isn't my final model, it's just for helping selecting k.\n",
    "\n",
    "The final 9 features are: `[total_stock_value exercised_stock_options _audit _behavior _breakfast _embed _ene _merchant _prestohouectect]` (feature names start with \"\\_\" are the word features, others are either e-mail or financial features). There are 2 financial features and 7 word features selected, no e-mail features selected, neither `fraction_from_poi` and `fraction_to_poi`. \n",
    "\n",
    "Here's a table of selected features, ordered by score. The \"Score\" is the score of that feature calculated by `SelectKBest`, bigger is better; \"_p_-value\" is the _p_-value of that score, smaller is better.\n",
    "\n",
    "| Feature                 | Score       | _p_-value      |\n",
    "| ----------------------- | ----------- | -------------- |\n",
    "| \\_behavior              | 32.15506493 | 7.68150329e-08 |\n",
    "| \\_prestohouectect       | 29.87398197 | 2.01474034e-07 |\n",
    "| \\_ene                   | 28.11011323 | 4.29039611e-07 |\n",
    "| \\_breakfast             | 25.89880301 | 1.12148645e-06 |\n",
    "| exercised_stock_options | 25.09754153 | 1.59454385e-06 |\n",
    "| \\_merchant              | 24.84351812 | 1.78352260e-06 |\n",
    "| total_stock_value       | 24.46765405 | 2.10580665e-06 |\n",
    "| \\_audit                 | 23.97568818 | 2.61906165e-06 |\n",
    "| \\_embed                 | 23.1934391  | 3.71096211e-06 |\n",
    "\n",
    "Let's also see the scores of two features I created:\n",
    "\n",
    "| Feature           | Score         | _p_-value         |\n",
    "| ----------------- | ------------- | ----------------- |\n",
    "| fraction_to_poi   | 16.6417070705 | 7.49415402503e-05 |\n",
    "| fraction_from_poi | 3.21076191697 | 0.0752849005991   |\n",
    "\n",
    "Apparently, they aren't strong enough compared to the 9 features above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Scaling\n",
    "At first, I did feature scaling on all the features universally, because that makes it easier to test out different classifiers. Doing feature scaling may cause a huge difference on the performance of some algorithms, e.g. SVM, K-means clustering; Some other algorithms do not affected by feature scaling, e.g. Decision Tree, Linear Regression.\n",
    "\n",
    "In my final model, I applied feature scaling with `MinMaxScaler`, which scales the values into numbers between 0 and 1, the smallest value converts to 0, the largest one converts to 1. I did this because the final model involves SVM and Naïve Bayes, they're both affected by feature scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algorithm Selection <small>(Question #3)</small>\n",
    "\n",
    "### Investigation\n",
    "I tried a variety of classifiers provided by scikit-learn.  \n",
    "`LinearSVC`'s performance first caught my eye, it hits a precision score of 1.0 using our testing script, and about 0.5 for the recall score, which means, if the classifier labelled a person as a POI, we can almost ensure the person is a POI.  \n",
    "`GaussianNB` isn't outstanding for the precision score, but pretty good at recall, it hits 0.83 for the recall score.   \n",
    "`DecisionTreeClassifier` isn't too outstanding on both precision and recall score, they're both around 0.5.  \n",
    "`LogisticRegression` turned out to have similar behaviour as `LinearSVC`, but since we're not expecting probabilities as outputs, I didn't use it in my final model.  \n",
    "I also tried some ensemble methods such as `RandomForestClassifier` and `AdaBoostClassifier`, using `AdaBoostClassifier` with `base_estimator` of `DecisionTreeClassifier` made better result compared to pure `DecisionTreeClassifier`.  \n",
    "Here's a table of model performances (using default parameters without tuning them):\n",
    "\n",
    "| Algorithm                    | Accuracy | Precision | Recall | $F_1$ | $F_2$ | $F_{0.5}$ |\n",
    "| ---------------------------- | -------- | --------- | ------ | ----- | ----- | --------- |\n",
    "| `LinearSVC`                  | 0.935    | 1.000     | 0.514  | 0.679 | 0.570 | 0.841     |\n",
    "| `GaussianNB`                 | 0.919    | 0.655     | 0.832  | 0.733 | 0.789 | 0.684     |\n",
    "| `DecisionTreeClassifier`     | 0.876    | 0.540     | 0.466  | 0.500 | 0.479 | 0.524     |\n",
    "| `LogisticRegression`         | 0.885    | 1.000     | 0.135  | 0.238 | 0.163 | 0.438     |\n",
    "| `RandomForestClassifier`     | 0.902    | 0.741     | 0.409  | 0.527 | 0.449 | 0.637     |\n",
    "| `AdaBoostClassifier`         | 0.886    | 0.573     | 0.564  | 0.569 | 0.566 | 0.571     |\n",
    "| `GradientBoostingClassifier` | 0.886    | 0.609     | 0.414  | 0.493 | 0.443 | 0.557     |\n",
    "| `ExtraTreesClassifier`       | 0.899    | 0.679     | 0.464  | 0.551 | 0.495 | 0.621     |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final Algorithm\n",
    "I ended up using an ensemble method, `VotingClassifier`, with 3 classifier members: `LinearSVC`, `GaussianNB` and `AdaBoostClassifier`. Of course it involves parameter tuning on each classifier. `VotingClassifier` is exactly how it sounds like, it takes a list of classifiers, then determine the result based on majority rule."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter Tuning <small>(Question #4)</small>\n",
    "\n",
    "Parameter tuning is to tweak the algorithm's settings, it's an important step when building machine learning models. Depending on the situation and your goal, you might want to tune the paramters into different values, and that makes the classifier behave differently. A common mistake you can make when you don't do it well is overfitting, overfitting makes the classifier overly sensitive. An overfitted model will try to correctly label all the training data points, but sometimes the data point might be just an exception case, we don't have to be such sensitive. So an overfitted model will have high accuracy score on training dataset, low score on the testing dataset. If you tune the sensitivity in a wrong way, it happens to be overfitting. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning My Model\n",
    "\n",
    "I tuned `LinearSVC` by hand, by trying out different values and observing how the performance varied according to validation and evaluation. I found out that with `C=1.3`, the classifier gives the best performance, with precision and recall of 0.99 and 0.52.  \n",
    "I tuned `AdaBoostClassifier` using `GridSearchCV`, which exhaustively searched over specified parameters and automatically figure out the best set of parameters.  \n",
    "For `GaussianNB`, I don't have to, because it accepts no parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation <small>(Question #5)</small>\n",
    "\n",
    "Validation is a step of validating, testing and evaluating machine learning models, we usually do validation when we're trying out different algorithms or parameters to see if the model can generalise to unseen data. A classic mistake you would make if you do it wrong is overfitting, like I just described in the parameter tuning phase. Overfitting usually happens when you use a set of data that the classifier already seen to validate a classifier's performance, instead of using an independent dataset. So the most basic validation is to split the dataset into \"training set\" and \"testing set\", where training set is for training the model, testing set is to test and validate the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate My Model\n",
    "\n",
    "In my project, I used a very common validation technique called \"cross-validation\". The core idea of cross-validation is to generate a set of training  and testing set pair, then, for each pair, train the model using training set, validate it using testing set. The most basic approach is to use _k_-fold, but because of our small dataset, I used `StratifiedShuffleSplit`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Metrics <small>(Question #6)</small>\n",
    "\n",
    "There are many metrics for evaluating performance. Accuracy score is the most basic one, but since our dataset is highly skewed, number of non-POIs is way more than POIs, it isn't ideal to use accuracy as our evaluation metric. Precision and recall scores are more appropriate ones, or $F_{\\beta}$ score if considering both precision and recall. Depend on your situation or goal, you might want to use different metrics, e.g. if the POI identifier is for identify real criminals, we might consider precision score more, and it should be very high; if it's for detecting who might potentially be a POI, then do deeper investigation later, we'll consider recall score more. $F_{\\beta}$ score is a kind of weighted average of precision and recall, $\\beta$ stands for the weight of recall score, some common values for $\\beta$ are 1, 2 and 0.5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Performance\n",
    "\n",
    "Here are the scores of my final model using different metrics:\n",
    "\n",
    "| metric    | score   |\n",
    "|-----------|---------|\n",
    "| Accuracy  | 0.93067 |\n",
    "| Precision | 0.77273 |\n",
    "| Recall    | 0.68000 |\n",
    "| $F_1$     | 0.72340 |\n",
    "| $F_2$     | 0.69672 |\n",
    "| $F_{0.5}$ | 0.75221 |\n",
    "\n",
    "We've got an accuracy score of 0.93, which means, 93% of the time the identifier will correctly classify wether a person is a POI  \n",
    "Precision score of 0.77, which means that when it identifies a person as a POI, 77% of the time the person is actually a POI  \n",
    "Recall score of 0.68, which means, if a person is actually a POI, 68% of the time the identifier identifies the person as a POI  \n",
    "$F_1$ score is the harmonic mean of precision and recall, which is basically a way to average them. Here we've got 0.72  \n",
    "We can see that $F_2$ is more effected by recall since it weights recall more, and $F_{0.5}$ is more effected by precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#############\n",
    "# emails.py #\n",
    "#############\n",
    "\n",
    "import string\n",
    "import pickle\n",
    "from pathlib import Path\n",
    "from nltk.stem import SnowballStemmer\n",
    "from progress.bar import Bar\n",
    "\n",
    "\n",
    "# Copied from text learning mini-project\n",
    "def parse_out_text(path):\n",
    "    \"\"\"Get the content of an email\"\"\"\n",
    "\n",
    "    words = \"\"\n",
    "    stemmer = SnowballStemmer(\"english\")\n",
    "\n",
    "    with path.open() as f:\n",
    "        all_text = f.read()\n",
    "        content = all_text.split(\"X-FileName:\")\n",
    "\n",
    "        if len(content) > 1:\n",
    "            # Remove punctuations and numbers\n",
    "            text_string = content[1].translate({\n",
    "                ord(char): None for char in (string.punctuation + string.digits)\n",
    "            })\n",
    "\n",
    "            # Stem each word, then combine into single string\n",
    "            words = \" \".join([stemmer.stem(x) for x in text_string.split()])\n",
    "\n",
    "    return words\n",
    "\n",
    "\n",
    "def get_all_words_by_author(email):\n",
    "    \"\"\"Get all the email content sent from a specific email address\"\"\"\n",
    "\n",
    "    list_path = Path(\"emails_by_address/from_%s.txt\" % email)\n",
    "    contents = []\n",
    "\n",
    "    # The corpus doesn't contain some of the email data\n",
    "    if list_path.exists():\n",
    "\n",
    "        with list_path.open() as f:\n",
    "            # Every line of the reference file refers to an actual email\n",
    "            for line in f:\n",
    "\n",
    "                # The reference path is actually wrong in our case,\n",
    "                # they all have a prefix of \"enron_mail_20110402/maildir/\",\n",
    "                # but the maildir/ is actually located in our parent folder\n",
    "                incorrect_path = Path(line.strip())\n",
    "                real_path = Path(\"..\", *incorrect_path.parts[1:]).resolve()\n",
    "\n",
    "                contents.append(parse_out_text(real_path))\n",
    "\n",
    "    # Return a single string which contains all of the content\n",
    "    return \" \".join(contents)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    with open(\"final_project_dataset.pkl\", \"rb\") as f:\n",
    "        dataset = pickle.load(f)\n",
    "\n",
    "    word_data = {}\n",
    "\n",
    "    # Output format: {\"NAME\": \"CONTENT\", ...}\n",
    "\n",
    "    for name, row in Bar(\"Processing\").iter(sorted(dataset.items())):\n",
    "        # These two are outliers\n",
    "        if name not in (\"TOTAL\", \"THE TRAVEL AGENCY IN THE PARK\"):\n",
    "            if row[\"email_address\"] != \"NaN\":\n",
    "                word_data[name] = get_all_words_by_author(row[\"email_address\"])\n",
    "            else:\n",
    "                word_data[name] = \"\"\n",
    "\n",
    "    with open(\"word_data.pkl\", \"wb\") as f:\n",
    "        pickle.dump(word_data, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading dataset...   Done\n",
      "Removing outliers...   Done\n",
      "Creating new features...   Done\n",
      "Loading word data...   Done\n",
      "Vectorizing email data...   Done\n",
      "Preprocessing data...   Done\n",
      "Selecting features...   Done\n",
      "Generating final dataset...   Done\n",
      "Building model...   Done\n",
      "Tuning model...   Done\n",
      "Dumping classifier and data...   Done\n"
     ]
    }
   ],
   "source": [
    "#############\n",
    "# poi_id.py #\n",
    "#############\n",
    "\n",
    "import sys\n",
    "import pickle\n",
    "import warnings\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.ensemble import AdaBoostClassifier, VotingClassifier\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.metrics import fbeta_score, make_scorer\n",
    "from sklearn.cross_validation import StratifiedKFold\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "sys.path.append(\"../tools/\")\n",
    "\n",
    "from feature_format import featureFormat, targetFeatureSplit\n",
    "from tester import dump_classifier_and_data\n",
    "\n",
    "# Warning messages from GridSearchCV sometimes annoying\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=RuntimeWarning)\n",
    "\n",
    "### Task 1: Select what features you'll use.\n",
    "### features_list is a list of strings, each of which is a feature name.\n",
    "### The first feature must be \"poi\".\n",
    "\n",
    "fin_features = [\"salary\", \"deferral_payments\", \"total_payments\",\n",
    "                \"loan_advances\", \"bonus\", \"restricted_stock_deferred\",\n",
    "                \"deferred_income\", \"total_stock_value\", \"expenses\",\n",
    "                \"exercised_stock_options\", \"other\", \"long_term_incentive\",\n",
    "                \"restricted_stock\", \"director_fees\"]\n",
    "\n",
    "eml_features = [\"to_messages\", \"from_poi_to_this_person\", \"from_messages\",\n",
    "                \"from_this_person_to_poi\", \"shared_receipt_with_poi\",\n",
    "                \"fraction_to_poi\", \"fraction_from_poi\"]\n",
    "\n",
    "# This is not the final features_list, we'll build it with SelectKBest\n",
    "all_features = [\"poi\"] + fin_features + eml_features\n",
    "\n",
    "sys.stdout.write(\"Loading dataset...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "### Load the dictionary containing the dataset\n",
    "with open(\"final_project_dataset.pkl\", \"rb\") as data_file:\n",
    "    data_dict = pickle.load(data_file)\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Removing outliers...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "### Task 2: Remove outliers\n",
    "data_dict.pop(\"TOTAL\")\n",
    "data_dict.pop(\"THE TRAVEL AGENCY IN THE PARK\")\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Creating new features...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "### Task 3: Create new feature(s)\n",
    "### Store to my_dataset for easy export below.\n",
    "\n",
    "# New features:\n",
    "#     1. fraction_from_poi: from_poi_to_this_person / to_messages\n",
    "#     2. fraction_to_poi: from_this_person_to_poi / from_messages\n",
    "\n",
    "for point in data_dict.values():\n",
    "    # Default values for those having missing values\n",
    "    point.update({\"fraction_from_poi\": 0, \"fraction_to_poi\": 0})\n",
    "\n",
    "    if \"NaN\" not in (point[\"from_poi_to_this_person\"], point[\"to_messages\"]):\n",
    "        point[\"fraction_from_poi\"] = \\\n",
    "            point[\"from_poi_to_this_person\"] / point[\"to_messages\"]\n",
    "\n",
    "    if \"NaN\" not in (point[\"from_this_person_to_poi\"], point[\"from_messages\"]):\n",
    "        point[\"fraction_to_poi\"] = \\\n",
    "            point[\"from_this_person_to_poi\"] / point[\"from_messages\"]\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Loading word data...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "# Pre-built e-mail content dict generated by emails.py\n",
    "with open(\"word_data.pkl\", \"rb\") as f:\n",
    "    word_data = pickle.load(f)\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Vectorizing email data...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "vectorizer = TfidfVectorizer(stop_words=\"english\")\n",
    "# Sort by names, make it compatible with featureFormat\n",
    "tf = vectorizer.fit_transform([x[1] for x in sorted(word_data.items())])\n",
    "\n",
    "word_features = vectorizer.get_feature_names()\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Preprocessing data...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "# Here I'm not adding all the word features into data_dict or my_dataset,\n",
    "# that will waste too much time and they'll be extemely large. Instead,\n",
    "# I preprocess the data_dict with featureFormat, then concatenate with\n",
    "# the matrix generated by TfidfVectorizer.\n",
    "# After that, do feature scaling and selection, then transform the\n",
    "# final numpy array into original dict format\n",
    "data = featureFormat(data_dict, all_features,\n",
    "                     remove_all_zeroes=False, sort_keys=True)\n",
    "\n",
    "# Concatenate two arrays vertically with np.hstack\n",
    "data = np.hstack((data, tf.toarray()))\n",
    "\n",
    "labels, features = targetFeatureSplit(data)\n",
    "\n",
    "# Feature scaling\n",
    "# Note that this applies to the final dataset used by tester.py,\n",
    "# see L110~L115 and L178~L185\n",
    "features = MinMaxScaler().fit_transform(features)\n",
    "\n",
    "# Add an underscore before every word feature name to avoid ambiguity with\n",
    "# original features\n",
    "for feature in word_features:\n",
    "    all_features.append(\"_\" + feature)\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Selecting features...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "# Feature selection\n",
    "selector = SelectKBest()\n",
    "\n",
    "# Switch to True to perform grid search\n",
    "search = False\n",
    "\n",
    "if search:\n",
    "    sys.stdout.write(\"\\n\")\n",
    "\n",
    "    estimator = Pipeline([\n",
    "        (\"select\", SelectKBest()),\n",
    "        # Note that this SVM isn't the final model\n",
    "        (\"svm\", LinearSVC()),\n",
    "    ])\n",
    "    # More than 20 is too much\n",
    "    params = {\"select__k\": list(range(2, 20))}\n",
    "\n",
    "    # Run 2 jobs at the same time, also print the progress into console\n",
    "    # Here I use StratifiedKFold with 10 folds as CV for searching\n",
    "    searcher = GridSearchCV(estimator, params, scoring=\"f1\", n_jobs=2,\n",
    "                            cv=StratifiedKFold(labels, 10), verbose=1)\n",
    "    searcher.fit(features, labels)\n",
    "\n",
    "    selector.k = searcher.best_params_[\"select__k\"]\n",
    "\n",
    "else:\n",
    "    # The result I got is 9\n",
    "    selector.k = 9\n",
    "\n",
    "features = selector.fit_transform(features, labels)\n",
    "# Get selected features using numpy array indexing\n",
    "# all_features contains \"poi\" which isn't a feature\n",
    "selected_features = np.array(all_features[1:])[selector.get_support()]\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "sys.stdout.write(\"Generating final dataset...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "# Generate my_dataset for testing script\n",
    "my_dataset = {}\n",
    "# Use sorting to make sure the features are mapping to the correct person\n",
    "for name, point, label in zip(sorted(data_dict), features, labels):\n",
    "    my_dataset[name] = {\"poi\": label}\n",
    "    # point is an 1D array, each row of the 2D array features\n",
    "    for feature, value in zip(selected_features, point):\n",
    "        my_dataset[name][feature] = value\n",
    "\n",
    "# Generate features_list for testing script\n",
    "features_list = [\"poi\"] + list(selected_features)\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "### Task 4: Try a varity of classifiers\n",
    "### Please name your classifier clf for easy export below.\n",
    "### Note that if you want to do PCA or other multi-stage operations,\n",
    "### you'll need to use Pipelines. For more info:\n",
    "### http://scikit-learn.org/stable/modules/pipeline.html\n",
    "\n",
    "sys.stdout.write(\"Building model...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "# C Parameter for this LinearSVC is tuned by hand\n",
    "svm = LinearSVC(C=1.3)\n",
    "\n",
    "nb = GaussianNB()\n",
    "\n",
    "# Tune parameters in the next task\n",
    "adabst = AdaBoostClassifier(DecisionTreeClassifier())\n",
    "\n",
    "clf = VotingClassifier([\n",
    "    (\"svm\", svm),\n",
    "    (\"nb\", nb),\n",
    "    (\"ada\", adabst),\n",
    "])\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "### Task 5: Tune your classifier to achieve better than .3 precision and recall\n",
    "### using our testing script. Check the tester.py script in the final project\n",
    "### folder for details on the evaluation method, especially the test_classifier\n",
    "### function. Because of the small size of the dataset, the script uses\n",
    "### stratified shuffle split cross validation. For more info:\n",
    "### http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedShuffleSplit.html\n",
    "\n",
    "sys.stdout.write(\"Tuning model...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "# Switch to True to perform grid search\n",
    "search = False\n",
    "\n",
    "if search:\n",
    "\n",
    "    sys.stdout.write(\"\\n\")\n",
    "\n",
    "    # Here I use F2 score, which weights recall score more\n",
    "    f2_score = make_scorer(fbeta_score, beta=2)\n",
    "\n",
    "    param_grid = {\n",
    "        \"base_estimator__max_features\": [None, \"sqrt\", \"log2\"],\n",
    "        \"base_estimator__max_depth\": [None, 3, 5, 8, 10],\n",
    "        \"base_estimator__min_samples_leaf\": [1, 2, 3, 5, 8],\n",
    "        \"learning_rate\": [0.01, 0.1, 0.5, 0.8, 1.0],\n",
    "    }\n",
    "\n",
    "    searcher = GridSearchCV(adabst, param_grid, f2_score, n_jobs=2, verbose=1,\n",
    "                            cv=StratifiedKFold(labels, 10))\n",
    "\n",
    "    searcher.fit(features, labels)\n",
    "\n",
    "    # Apply tuned parameters to the model\n",
    "    adabst.set_params(**searcher.best_params_)\n",
    "\n",
    "else:\n",
    "\n",
    "    # Result I got when I ran the above searching\n",
    "    adabst.set_params(base_estimator__max_features=\"sqrt\",\n",
    "                      base_estimator__min_samples_leaf=1,\n",
    "                      base_estimator__max_depth=3,\n",
    "                      learning_rate=0.01)\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")\n",
    "\n",
    "### Task 6: Dump your classifier, dataset, and features_list so anyone can\n",
    "### check your results. You do not need to change anything below, but make sure\n",
    "### that the version of poi_id.py that you submit can be run on its own and\n",
    "### generates the necessary .pkl files for validating your results.\n",
    "\n",
    "sys.stdout.write(\"Dumping classifier and data...   \")\n",
    "sys.stdout.flush()\n",
    "\n",
    "dump_classifier_and_data(clf, my_dataset, features_list)\n",
    "\n",
    "sys.stdout.write(\"Done\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VotingClassifier(estimators=[('svm', LinearSVC(C=1.3, class_weight=None, dual=True, fit_intercept=True,\r\n",
      "     intercept_scaling=1, loss='squared_hinge', max_iter=1000,\r\n",
      "     multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,\r\n",
      "     verbose=0)), ('nb', GaussianNB()), ('ada', AdaBoostClassifier(algorithm='SAM...om_state=None, splitter='best'),\r\n",
      "          learning_rate=0.01, n_estimators=50, random_state=None))],\r\n",
      "         voting='hard', weights=None)\r\n",
      "\tAccuracy: 0.93053\tPrecision: 0.77216\tRecall: 0.67950\tF1: 0.72287\tF2: 0.69621\r\n",
      "\tTotal predictions: 15000\tTrue positives: 1359\tFalse positives:  401\tFalse negatives:  641\tTrue negatives: 12599\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!python tester.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}