NLTKCh6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Learning to Classify Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Important: From this chapter onwards, our program samples will assume you begin your interactive session or your program with the \n",
    "# following import statements:\n",
    "\n",
    "from __future__ import division  # Python 2 users only\n",
    "import nltk, re, pprint # nltk, regular expression, and pretty print?\n",
    "from nltk import word_tokenize\n",
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Detecting patterns is a central part of Natural Language Processing. Words ending in -ed tend to be past tense verbs (5). Frequent use of will is indicative of news text (3). These observable patterns — word structure and word frequency — happen to correlate with particular aspects of meaning, such as tense and topic. But how did we know where to start looking, which aspects of form to associate with which aspects of meaning?\n",
    "\n",
    "The goal of this chapter is to answer the following questions:\n",
    "\n",
    "1. How can we identify particular features of language data that are salient for classifying it?\n",
    "2. How can we construct models of language that can be used to perform language processing tasks automatically?\n",
    "3. What can we learn about language from these models?\n",
    "\n",
    "Along the way we will study some important machine learning techniques, including decision trees, naive Bayes' classifiers, and maximum entropy classifiers. We will gloss over the mathematical and statistical underpinnings of these techniques, focusing instead on how and when to use them (see the Further Readings section for more technical background). Before looking at these methods, we first need to appreciate the broad scope of this topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Supervised Classification"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Classification is the task of choosing the correct class label for a given input. In basic classification tasks, each input is considered in isolation from all other inputs, and the set of labels is defined in advance. Some examples of classification tasks are:\n",
    "\n",
    "- Deciding whether an email is spam or not.\n",
    "- Deciding what the topic of a news article is, from a fixed list of topic areas such as \"sports,\" \"technology,\" and \"politics.\"\n",
    "- Deciding whether a given occurrence of the word bank is used to refer to a river bank, a financial institution, the act of tilting to the side, or the act of depositing something in a financial institution.\n",
    "\n",
    "The basic classification task has a number of interesting variants. For example, in multi-class classification, each instance may be assigned multiple labels; in open-class classification, the set of labels is not defined in advance; and in sequence classification, a list of inputs are jointly classified.\n",
    "\n",
    "A classifier is called supervised if it is built based on training corpora containing the correct label for each input. The framework used by supervised classification is shown in 6.1.\n",
    "\n",
    "Figure 6.1: Supervised Classification. (a) During training, a feature extractor is used to convert each input value to a feature set. These feature sets, which capture the basic information about each input that should be used to classify it, are discussed in the next section. Pairs of feature sets and labels are fed into the machine learning algorithm to generate a model. (b) During prediction, the same feature extractor is used to convert unseen inputs to feature sets. These feature sets are then fed into the model, which generates predicted labels.\n",
    "\n",
    "In the rest of this section, we will look at how classifiers can be employed to solve a wide variety of tasks. Our discussion is not intended to be comprehensive, but to give a representative sample of tasks that can be performed with the help of text classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gender Identification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In 2.4 we saw that male and female names have some distinctive characteristics. Names ending in a, e and i are likely to be female, while names ending in k, o, r, s and t are likely to be male. Let's build a classifier to model these differences more precisely.\n",
    "\n",
    "The first step in creating a classifier is deciding what features of the input are relevant, and how to encode those features. For this example, we'll start by just looking at the final letter of a given name. The following feature extractor function builds a dictionary containing relevant information about a given name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gender_features(word):\n",
    "    return {'last_letter': word[-1]}\n",
    "# Our only feature is the last letter of the name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'last_letter': 'k'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_features('Shrek')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'last_letter': 'b'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_features('Bob')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned dictionary, known as a feature set, maps from features' names to their values. Feature names are case-sensitive strings that typically provide a short human-readable description of the feature. Feature values are values with simple types, such as booleans, numbers, and strings.\n",
    "\n",
    "Note\n",
    "----\n",
    "Most classification methods require that features be encoded using simple value types, such as booleans, numbers, and strings. But note that just because a feature has a simple type, does not necessarily mean that the feature's value is simple to express or compute; indeed, it is even possible to use very complex and informative values, such as the output of a second supervised classifier, as features.\n",
    "\n",
    "Now that we've defined a feature extractor, we need to prepare a list of examples and corresponding class labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7944"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import names # we import names to access list of (fe)male names\n",
    "import random\n",
    "names = ([(name, 'male') for name in names.words('male.txt')] +\n",
    "         [(name, 'female') for name in names.words('female.txt')])\n",
    "import random # unclear why we imported random twice\n",
    "random.shuffle(names)\n",
    "len(names) # 7944 names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use the feature extractor to process the names data, and divide the resulting list of feature sets into a training set and a test set. The training set is used to train a new \"naive Bayes\" classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "featuresets = [(gender_features(n), g) for (n,g) in names]\n",
    "train_set, test_set = featuresets[500:], featuresets[:500] # test set is first 500 records\n",
    "# train set is everything after the first 500 records, actually 7544 names. We train on far\n",
    "# more data than we test\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We will learn more about the naive Bayes classifier later in the chapter. For now, let's just test it out on some names that did not appear in its training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'male'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Neo'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'female'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Trinity'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'male'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Bob'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'female'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Deandrea'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe that these character names from The Matrix are correctly classified. Although this science fiction movie is set in 2199, it still conforms with our expectations about names and genders. We can systematically evaluate the classifier on a much larger quantity of unseen data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.708"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.classify.accuracy(classifier, test_set)\n",
    "# I assume we get the accuracy by calculating what percent of the test set we predicted accurately\n",
    "# accuracy = (total # predicted right)/(total in test set) ? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can examine the classifier to determine which features it found most effective for distinguishing the names' genders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "             last_letter = u'a'           female : male   =     37.3 : 1.0\n",
      "             last_letter = u'k'             male : female =     32.6 : 1.0\n",
      "             last_letter = u'f'             male : female =     17.2 : 1.0\n",
      "             last_letter = u'p'             male : female =     12.5 : 1.0\n",
      "             last_letter = u'v'             male : female =     10.5 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier.show_most_informative_features(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This listing shows that the names in the training set that end in \"a\" are female 39 times more often than they are male, but names that end in \"k\" are male 32 times more often than they are female. These ratios are known as likelihood ratios, and can be useful for comparing different feature-outcome relationships.\n",
    "\n",
    "Note\n",
    "\n",
    "Your Turn: Modify the gender_features() function to provide the classifier with features encoding the length of the name, its first letter, and any other features that seem like they might be informative. Retrain the classifier with these new features, and test its accuracy.\n",
    "\n",
    "When working with large corpora, constructing a single list that contains the features of every instance can use up a large amount of memory. In these cases, use the function nltk.classify.apply_features, which returns an object that acts like a list but does not store all the feature sets in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def gender_features(word):\n",
    "    return {'last_letter': word[-1], 'first_letter': word[0], 'name_length': len(word)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'first_letter': 'B', 'last_letter': 'b', 'name_length': 3}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_features('Bob')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'first_letter': 'D', 'last_letter': 'a', 'name_length': 8}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_features('Deandrea')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "random.shuffle(names)\n",
    "featuresets = [(gender_features(n), g) for (n,g) in names]\n",
    "train_set, test_set = featuresets[500:], featuresets[:500]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'male'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Neo'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'male'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(gender_features('Trinity'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.774"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.classify.accuracy(classifier, test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, slightly worse than previous classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "             last_letter = u'a'           female : male   =     35.8 : 1.0\n",
      "             last_letter = u'k'             male : female =     31.6 : 1.0\n",
      "             last_letter = u'f'             male : female =     28.7 : 1.0\n",
      "             last_letter = u'p'             male : female =     18.6 : 1.0\n",
      "             last_letter = u'd'             male : female =     10.8 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier.show_most_informative_features(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.classify import apply_features\n",
    "train_set = apply_features(gender_features, names[500:])\n",
    "test_set = apply_features(gender_features, names[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing the Right Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Selecting relevant features and deciding how to encode them for a learning method can have an enormous impact on the learning method's ability to extract a good model. Much of the interesting work in building a classifier is deciding what features might be relevant, and how we can represent them. Although it's often possible to get decent performance by using a fairly simple and obvious set of features, there are usually significant gains to be had by using carefully constructed features based on a thorough understanding of the task at hand.\n",
    "\n",
    "Typically, feature extractors are built through a process of trial-and-error, guided by intuitions about what information is relevant to the problem. It's common to start with a \"kitchen sink\" approach, including all the features that you can think of, and then checking to see which features actually are helpful. We take this approach for name gender features in 6.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gender_features2(name):\n",
    "    features = {}\n",
    "    features[\"firstletter\"] = name[0].lower()\n",
    "    features[\"lastletter\"] = name[-1].lower()\n",
    "    for letter in 'abcdefghijklmnopqrstuvwxyz':\n",
    "        features[\"count(%s)\" % letter] = name.lower().count(letter)\n",
    "        features[\"has(%s)\" % letter] = (letter in name.lower())\n",
    "    return features\n",
    "# We have 54 features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'count(a)': 0,\n",
       " 'count(b)': 0,\n",
       " 'count(c)': 0,\n",
       " 'count(d)': 0,\n",
       " 'count(e)': 0,\n",
       " 'count(f)': 0,\n",
       " 'count(g)': 0,\n",
       " 'count(h)': 1,\n",
       " 'count(i)': 0,\n",
       " 'count(j)': 1,\n",
       " 'count(k)': 0,\n",
       " 'count(l)': 0,\n",
       " 'count(m)': 0,\n",
       " 'count(n)': 1,\n",
       " 'count(o)': 1,\n",
       " 'count(p)': 0,\n",
       " 'count(q)': 0,\n",
       " 'count(r)': 0,\n",
       " 'count(s)': 0,\n",
       " 'count(t)': 0,\n",
       " 'count(u)': 0,\n",
       " 'count(v)': 0,\n",
       " 'count(w)': 0,\n",
       " 'count(x)': 0,\n",
       " 'count(y)': 0,\n",
       " 'count(z)': 0,\n",
       " 'firstletter': 'j',\n",
       " 'has(a)': False,\n",
       " 'has(b)': False,\n",
       " 'has(c)': False,\n",
       " 'has(d)': False,\n",
       " 'has(e)': False,\n",
       " 'has(f)': False,\n",
       " 'has(g)': False,\n",
       " 'has(h)': True,\n",
       " 'has(i)': False,\n",
       " 'has(j)': True,\n",
       " 'has(k)': False,\n",
       " 'has(l)': False,\n",
       " 'has(m)': False,\n",
       " 'has(n)': True,\n",
       " 'has(o)': True,\n",
       " 'has(p)': False,\n",
       " 'has(q)': False,\n",
       " 'has(r)': False,\n",
       " 'has(s)': False,\n",
       " 'has(t)': False,\n",
       " 'has(u)': False,\n",
       " 'has(v)': False,\n",
       " 'has(w)': False,\n",
       " 'has(x)': False,\n",
       " 'has(y)': False,\n",
       " 'has(z)': False,\n",
       " 'lastletter': 'n'}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_features2('John')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 6.2: A Feature Extractor that Overfits Gender Features. The featuresets returned by this feature extractor contain a large number of specific features, leading to overfitting for the relatively small Names Corpus.\n",
    "\n",
    "However, there are usually limits to the number of features that you should use with a given learning algorithm — if you provide too many features, then the algorithm will have a higher chance of relying on idiosyncrasies of your training data that don't generalize well to new examples. This problem is known as overfitting, and can be especially problematic when working with small training sets. For example, if we train a naive Bayes classifier using the feature extractor shown in 6.2, it will overfit relatively small training set, resulting in a system whose accuracy is about 1% lower than the accuracy of a classifier that only pays attention to the final letter of each name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.776"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "featuresets = [(gender_features2(n), g) for (n,g) in names]\n",
    "train_set, test_set = featuresets[500:], featuresets[:500]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "nltk.classify.accuracy(classifier, test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Actually, this classifier is more accurate, despite what NLTK book shows]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once an initial set of features has been chosen, a very productive method for refining the feature set is error analysis. First, we select a development set, containing the corpus data for creating the model. This development set is then subdivided into the training set and the dev-test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_names = names[1500:]\n",
    "devtest_names = names[500:1500]\n",
    "test_names = names[:500]\n",
    "\n",
    "# test set (0-499), devtest set (500-1499), and train set (1500 and beyond)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training set is used to train the model, and the dev-test set is used to perform error analysis. The test set serves in our final evaluation of the system. For reasons discussed below, it is important that we employ a separate dev-test set for error analysis, rather than just using the test set. The division of the corpus data into different subsets is shown in 6.3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 6.3: Organization of corpus data for training supervised classifiers. The corpus data is divided into two sets: the development set, and the test set. The development set is often further subdivided into a training set and a dev-test set.\n",
    "\n",
    "Having divided the corpus into appropriate datasets, we train a model using the training set [1], and then run it on the devtest set [2]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.771"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set = [(gender_features(n), g) for (n,g) in train_names]\n",
    "devtest_set = [(gender_features(n), g) for (n,g) in devtest_names]\n",
    "test_set = [(gender_features(n), g) for (n,g) in test_names]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "nltk.classify.accuracy(classifier, devtest_set)\n",
    "# We assess the accuracy on the devtest set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the dev-test set, we can generate a list of the errors that the classifier makes when predicting name genders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "errors = []\n",
    "for (name, tag) in devtest_names:\n",
    "    guess = classifier.classify(gender_features(name))\n",
    "    if guess != tag:\n",
    "        errors.append( (tag, guess, name) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then examine individual error cases where the model predicted the wrong label, and try to determine what additional pieces of information would allow it to make the right decision (or which existing pieces of information are tricking it into making the wrong decision). The feature set can then be adjusted accordingly. The names classifier that we have built generates about 100 errors on the dev-test corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "female male Alexis\n",
      "female male Amargo\n",
      "female male Anet\n",
      "female male Beilul\n",
      "female male Bev\n",
      "female male Blanch\n",
      "female male Brigid\n",
      "female male Brit\n",
      "female male Brook\n",
      "female male Calypso\n",
      "female male Cass\n",
      "female male Cher\n",
      "female male Chris\n",
      "female male Dagmar\n",
      "female male Debor\n",
      "female male Deeann\n",
      "female male Dell\n",
      "female male Deloris\n",
      "female male Doris\n",
      "female male Dorit\n",
      "female male Eilis\n",
      "female male Ellynn\n",
      "female male Em\n",
      "female male Evelyn\n",
      "female male Fawn\n",
      "female male Fey\n",
      "female male Frances\n",
      "female male Gayleen\n",
      "female male Gen\n",
      "female male Gennifer\n",
      "female male Gertrudis\n",
      "female male Gillan\n",
      "female male Gillian\n",
      "female male Gredel\n",
      "female male Greer\n",
      "female male Grethel\n",
      "female male Grier\n",
      "female male Harmony\n",
      "female male Harriet\n",
      "female male Hatty\n",
      "female male Hedy\n",
      "female male Hildegaard\n",
      "female male Honor\n",
      "female male Jaclin\n",
      "female male Jasmin\n",
      "female male Jess\n",
      "female male Justin\n",
      "female male Kris\n",
      "female male Lilias\n",
      "female male Linnet\n",
      "female male Lust\n",
      "female male Lynnett\n",
      "female male Margalo\n",
      "female male Margot\n",
      "female male Marlo\n",
      "female male Marylou\n",
      "female male Mellicent\n",
      "female male Nadeen\n",
      "female male Orel\n",
      "female male Perl\n",
      "female male Pru\n",
      "female male Quentin\n",
      "female male Remy\n",
      "female male Renel\n",
      "female male Robinet\n",
      "female male Robyn\n",
      "female male Rosamund\n",
      "female male Roselyn\n",
      "female male Roxy\n",
      "female male Scarlett\n",
      "female male Shannen\n",
      "female male Shaylynn\n",
      "female male Sheila-Kathryn\n",
      "female male Sheilakathryn\n",
      "female male Sherill\n",
      "female male Shir\n",
      "female male Sibeal\n",
      "female male Susy\n",
      "female male Tammy\n",
      "female male Teddy\n",
      "female male Tess\n",
      "female male Ursuline\n",
      "female male Yoshiko\n",
      "male female Abbie\n",
      "male female Adnan\n",
      "male female Aguste\n",
      "male female Ajay\n",
      "male female Alain\n",
      "male female Alfonse\n",
      "male female Algernon\n",
      "male female Anatole\n",
      "male female Andrej\n",
      "male female Andrzej\n",
      "male female Antin\n",
      "male female Anton\n",
      "male female Antoni\n",
      "male female Ashby\n",
      "male female Baillie\n",
      "male female Baily\n",
      "male female Barnabe\n",
      "male female Barney\n",
      "male female Bjorne\n",
      "male female Brandy\n",
      "male female Broddy\n",
      "male female Brody\n",
      "male female Bucky\n",
      "male female Burnaby\n",
      "male female Carlie\n",
      "male female Cary\n",
      "male female Chrisy\n",
      "male female Clemmie\n",
      "male female Clive\n",
      "male female Cobbie\n",
      "male female Coleman\n",
      "male female Collin\n",
      "male female Cortese\n",
      "male female Corwin\n",
      "male female Daniel\n",
      "male female Dannie\n",
      "male female Darin\n",
      "male female Darrel\n",
      "male female Dietrich\n",
      "male female Duffy\n",
      "male female Dwain\n",
      "male female Earle\n",
      "male female Erich\n",
      "male female Erwin\n",
      "male female Eugene\n",
      "male female Felice\n",
      "male female Felipe\n",
      "male female Felix\n",
      "male female Ferdie\n",
      "male female Gerri\n",
      "male female Giuseppe\n",
      "male female Herbie\n",
      "male female Herve\n",
      "male female Howie\n",
      "male female Hymie\n",
      "male female Israel\n",
      "male female Jae\n",
      "male female Jamey\n",
      "male female Jason\n",
      "male female Jefferey\n",
      "male female Jeffie\n",
      "male female Jeffrey\n",
      "male female Jesse\n",
      "male female Jodie\n",
      "male female Jody\n",
      "male female Johnny\n",
      "male female Jonathon\n",
      "male female Kalle\n",
      "male female Kelley\n",
      "male female Kelvin\n",
      "male female Kermie\n",
      "male female Kerry\n",
      "male female Kevin\n",
      "male female Kimball\n",
      "male female Lance\n",
      "male female Lane\n",
      "male female Lennie\n",
      "male female Lenny\n",
      "male female Leroy\n",
      "male female Lesley\n",
      "male female Lindsey\n",
      "male female Lion\n",
      "male female Locke\n",
      "male female Lucian\n",
      "male female Maddie\n",
      "male female Marchall\n",
      "male female Marshall\n",
      "male female Melvyn\n",
      "male female Meredeth\n",
      "male female Merell\n",
      "male female Merrill\n",
      "male female Mervin\n",
      "male female Micah\n",
      "male female Miguel\n",
      "male female Mikael\n",
      "male female Mikey\n",
      "male female Milton\n",
      "male female Mitchel\n",
      "male female Mitchell\n",
      "male female Montague\n",
      "male female Morrie\n",
      "male female Mose\n",
      "male female Mugsy\n",
      "male female Myke\n",
      "male female Nichole\n",
      "male female Nickie\n",
      "male female Nigel\n",
      "male female Ollie\n",
      "male female Osbourne\n",
      "male female Patty\n",
      "male female Percy\n",
      "male female Perry\n",
      "male female Petey\n",
      "male female Rabi\n",
      "male female Rawley\n",
      "male female Reube\n",
      "male female Richy\n",
      "male female Rickey\n",
      "male female Ricki\n",
      "male female Ripley\n",
      "male female Rourke\n",
      "male female Rudie\n",
      "male female Sammie\n",
      "male female Sandy\n",
      "male female Scotti\n",
      "male female Skelly\n",
      "male female Skippie\n",
      "male female Smitty\n",
      "male female Spence\n",
      "male female Sully\n",
      "male female Tammie\n",
      "male female Tarrance\n",
      "male female Tate\n",
      "male female Terence\n",
      "male female Tymothy\n",
      "male female Vasily\n",
      "male female Verge\n",
      "male female Vijay\n",
      "male female Vinnie\n",
      "male female Vinny\n",
      "male female Vite\n",
      "male female Yance\n",
      "male female Zacharia\n",
      "male female Zacharie\n",
      "male female Zollie\n",
      "male female Zolly\n"
     ]
    }
   ],
   "source": [
    "for (tag, guess, name) in sorted(errors):\n",
    "    print (tag, guess, name) # modified so that I can print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking through this list of errors makes it clear that some suffixes that are more than one letter can be indicative of name genders. For example, names ending in yn appear to be predominantly female, despite the fact that names ending in n tend to be male; and names ending in ch are usually male, even though names that end in h tend to be female. We therefore adjust our feature extractor to include features for two-letter suffixes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gender_features(word):\n",
    "    return {'suffix1': word[-1:],\n",
    "            'suffix2': word[-2:]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rebuilding the classifier with the new feature extractor, we see that the performance on the dev-test dataset improves by almost 3 percentage points (from 76.5% to 78.2%):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.771"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set = [(gender_features(n), g) for (n,g) in train_names]\n",
    "devtest_set = [(gender_features(n), g) for (n,g) in devtest_names]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "nltk.classify.accuracy(classifier, devtest_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This error analysis procedure can then be repeated, checking for patterns in the errors that are made by the newly improved classifier. Each time the error analysis procedure is repeated, we should select a different dev-test/training split, to ensure that the classifier does not start to reflect idiosyncrasies in the dev-test set.\n",
    "\n",
    "But once we've used the dev-test set to help us develop the model, we can no longer trust that it will give us an accurate idea of how well the model would perform on new data. It is therefore important to keep the test set separate, and unused, until our model development is complete. At that point, we can use the test set to evaluate well our model will perform on new input values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Document Classification\n",
    "\n",
    "In 2.1, we saw several examples of corpora where documents have been labeled with categories. Using these corpora, we can build classifiers that will automatically tag new documents with appropriate category labels. First, we construct a list of documents, labeled with the appropriate categories. For this example, we've chosen the Movie Reviews Corpus, which categorizes each review as positive or negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import movie_reviews\n",
    "documents = [(list(movie_reviews.words(fileid)), category)\n",
    "             for category in movie_reviews.categories()\n",
    "             for fileid in movie_reviews.fileids(category)]\n",
    "random.shuffle(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define a feature extractor for documents, so the classifier will know which aspects of the data it should pay attention to (6.4). For document topic identification, we can define a feature for each word, indicating whether the document contains that word. To limit the number of features that the classifier needs to process, we begin by constructing a list of the 2000 most frequent words in the overall corpus [1]. We can then define a feature extractor [2] that simply checks whether each of these words is present in a given document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words())\n",
    "# first generate the frequency distribution for all words\n",
    "word_features = all_words.keys()[:2000] # [1]\n",
    "# select the top 2000 most frequent words\n",
    "\n",
    "def document_features(document): # [2]\n",
    "    document_words = set(document) # [3]\n",
    "    features = {}\n",
    "    for word in word_features:\n",
    "        features['contains(%s)' % word] = (word in document_words)\n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'contains(corporate)': False,\n",
       " u'contains(barred)': False,\n",
       " u'contains(batmans)': False,\n",
       " u'contains(menacing)': False,\n",
       " u'contains(rags)': False,\n",
       " u'contains(compton)': False,\n",
       " u'contains(inquires)': False,\n",
       " u'contains(nosebleeding)': False,\n",
       " u'contains(playhouse)': False,\n",
       " u'contains(peculiarities)': False,\n",
       " u'contains(guarded)': False,\n",
       " u'contains(kilgore)': False,\n",
       " u'contains(tarnish)': False,\n",
       " u'contains(sand)': False,\n",
       " u'contains(busting)': False,\n",
       " u'contains(wedge)': False,\n",
       " u'contains(smelling)': False,\n",
       " u'contains(tulip)': False,\n",
       " u'contains(singled)': False,\n",
       " u'contains(wahlberg)': False,\n",
       " u'contains(needed)': False,\n",
       " u'contains(lydia)': False,\n",
       " u'contains(rick)': False,\n",
       " u'contains(cambodia)': False,\n",
       " u'contains(grandma)': False,\n",
       " u'contains(jovivich)': False,\n",
       " u'contains(pinon)': False,\n",
       " u'contains(fix)': False,\n",
       " u'contains(marla)': False,\n",
       " u'contains(resources)': False,\n",
       " u'contains(nomi)': False,\n",
       " u'contains(irs)': False,\n",
       " u'contains(mason)': False,\n",
       " u'contains(vicariously)': False,\n",
       " u'contains(ingrained)': False,\n",
       " u'contains(skepticism)': False,\n",
       " u'contains(tested)': False,\n",
       " u'contains(spat)': False,\n",
       " u'contains(nfeatured)': False,\n",
       " u'contains(accomplishes)': False,\n",
       " u'contains(sierra)': False,\n",
       " u'contains(lunges)': False,\n",
       " u'contains(telefixated)': False,\n",
       " u'contains(wednesday)': False,\n",
       " u'contains(spiders)': False,\n",
       " u'contains(waft)': False,\n",
       " u'contains(jim)': False,\n",
       " u'contains(cinematic)': False,\n",
       " u'contains(pseudonymous)': False,\n",
       " u'contains(troubles)': False,\n",
       " u'contains(dimension)': False,\n",
       " u'contains(cassette)': False,\n",
       " u'contains(corsucant)': False,\n",
       " u'contains(absolute)': False,\n",
       " u'contains(harmonious)': False,\n",
       " u'contains(saying)': False,\n",
       " u'contains(misdemeanors)': False,\n",
       " u'contains(prodigious)': False,\n",
       " u'contains(reclining)': False,\n",
       " u'contains(loudness)': False,\n",
       " u'contains(lombard)': False,\n",
       " u'contains(sickening)': False,\n",
       " u'contains(cooking)': False,\n",
       " u'contains(oprah)': False,\n",
       " u'contains(platt)': False,\n",
       " u'contains(rolf)': False,\n",
       " u'contains(episodes)': False,\n",
       " u'contains(derivative)': False,\n",
       " u'contains(receipt)': False,\n",
       " u'contains(impactful)': False,\n",
       " u'contains(fonzie)': False,\n",
       " u'contains(spilling)': False,\n",
       " u'contains(chyron)': False,\n",
       " u'contains(dragonflies)': False,\n",
       " u'contains(ugliness)': False,\n",
       " u'contains(sauce)': False,\n",
       " u'contains(crossroads)': False,\n",
       " u'contains(paired)': False,\n",
       " u'contains(tomei)': False,\n",
       " u'contains(hounds)': False,\n",
       " u'contains(marshmallows)': False,\n",
       " u'contains(lick)': False,\n",
       " u'contains(hass)': False,\n",
       " u'contains(surges)': False,\n",
       " u'contains(watched)': False,\n",
       " u'contains(ivey)': False,\n",
       " u'contains(nuez)': False,\n",
       " u'contains(mentors)': False,\n",
       " u'contains(tricky)': False,\n",
       " u'contains(surveying)': False,\n",
       " u'contains(ditching)': False,\n",
       " u'contains(haystack)': False,\n",
       " u'contains(warriors)': False,\n",
       " u'contains(purged)': False,\n",
       " u'contains(rehabilitation)': False,\n",
       " u'contains(petitions)': False,\n",
       " u'contains(bogs)': False,\n",
       " u'contains(absense)': False,\n",
       " u'contains(courtesans)': False,\n",
       " u'contains(followed)': False,\n",
       " u'contains(har)': False,\n",
       " u'contains(anthesis)': False,\n",
       " u'contains(grotesqe)': False,\n",
       " u'contains(_i_know_what_you_did_last_summer_)': False,\n",
       " u'contains(survey)': False,\n",
       " u'contains(bartok)': False,\n",
       " u'contains(thivisol)': False,\n",
       " u'contains(exponential)': False,\n",
       " u'contains(birdie)': False,\n",
       " u'contains(cincinnati)': False,\n",
       " u'contains(unthrilling)': False,\n",
       " u'contains(balsan)': False,\n",
       " u'contains(bumming)': False,\n",
       " u'contains(celebrated)': False,\n",
       " u'contains(renovated)': False,\n",
       " u'contains(crowd)': False,\n",
       " u'contains(shows)': False,\n",
       " u'contains(lana)': False,\n",
       " u'contains(haskin)': False,\n",
       " u'contains(despite)': False,\n",
       " u'contains(scharzenegger)': False,\n",
       " u'contains(undergone)': False,\n",
       " u'contains(aissa)': False,\n",
       " u'contains(leit)': False,\n",
       " u'contains(mouth)': False,\n",
       " u'contains(demeaning)': False,\n",
       " u'contains(drilling)': False,\n",
       " u'contains(unsworth)': False,\n",
       " u'contains(bannister)': False,\n",
       " u'contains(wiseguy)': False,\n",
       " u'contains(lackies)': False,\n",
       " u'contains(norville)': False,\n",
       " u'contains(dampens)': False,\n",
       " u'contains(satisfactory)': False,\n",
       " u'contains(abuser)': False,\n",
       " u'contains(had)': False,\n",
       " u'contains(nature)': False,\n",
       " u'contains(awake)': False,\n",
       " u'contains(wiseguys)': False,\n",
       " u'contains(cerebrally)': False,\n",
       " u'contains(ballisitic)': False,\n",
       " u'contains(trauma)': False,\n",
       " u'contains(geography)': False,\n",
       " u'contains(genieveve)': False,\n",
       " u'contains(leeanne)': False,\n",
       " u'contains(cannibal)': False,\n",
       " u'contains(intoxicating)': False,\n",
       " u'contains(olds)': False,\n",
       " u'contains(tastefully)': False,\n",
       " u'contains(defy)': False,\n",
       " u'contains(drafted)': False,\n",
       " u'contains(strums)': False,\n",
       " u'contains(dilemnas)': False,\n",
       " u'contains(amoeba)': False,\n",
       " u'contains(assimilation)': False,\n",
       " u'contains(reprisal)': False,\n",
       " u'contains(janusz)': False,\n",
       " u'contains(leonardi)': False,\n",
       " u'contains(procures)': False,\n",
       " u'contains(canoeing)': False,\n",
       " u'contains(emergence)': False,\n",
       " u'contains(diving)': False,\n",
       " u'contains(aatish)': False,\n",
       " u'contains(masturbatory)': False,\n",
       " u'contains(peaceably)': False,\n",
       " u'contains(persistently)': False,\n",
       " u'contains(survival)': False,\n",
       " u'contains(portrays)': False,\n",
       " u'contains(ceasing)': False,\n",
       " u'contains(bubble)': False,\n",
       " u'contains(sonja)': False,\n",
       " u'contains(unarguably)': False,\n",
       " u'contains(unbuttoning)': False,\n",
       " u'contains(jessalyn)': False,\n",
       " u'contains(windmill)': False,\n",
       " u'contains(longs)': False,\n",
       " u'contains(baby)': False,\n",
       " u'contains(mute)': False,\n",
       " u'contains(sikh)': False,\n",
       " u'contains(nicely)': False,\n",
       " u'contains(burial)': False,\n",
       " u'contains(formed)': False,\n",
       " u'contains(locket)': False,\n",
       " u'contains(frizzy)': False,\n",
       " u'contains(darnell)': False,\n",
       " u'contains(outraging)': False,\n",
       " u'contains(politicos)': False,\n",
       " u'contains(2293)': False,\n",
       " u'contains(devoured)': False,\n",
       " u'contains(universality)': False,\n",
       " u'contains(didn)': False,\n",
       " u'contains(perpetrator)': False,\n",
       " u'contains(thats)': False,\n",
       " u'contains(dethroned)': False,\n",
       " u'contains(malintentioned)': False,\n",
       " u'contains(insubordinate)': False,\n",
       " u'contains(humpback)': False,\n",
       " u'contains(meditative)': False,\n",
       " u'contains(pimply)': False,\n",
       " u'contains(tyranny)': False,\n",
       " u'contains(loyalties)': False,\n",
       " u'contains(humbler)': False,\n",
       " u'contains(kindergartner)': False,\n",
       " u'contains(suitcases)': False,\n",
       " u'contains(grope)': False,\n",
       " u'contains(touts)': False,\n",
       " u'contains(advertisement)': False,\n",
       " u'contains(shirts)': False,\n",
       " u'contains(videodrome)': False,\n",
       " u'contains(eggar)': False,\n",
       " u'contains(portillo)': False,\n",
       " u'contains(macht)': False,\n",
       " u'contains(perishes)': False,\n",
       " u'contains(preference)': False,\n",
       " u'contains(pell)': False,\n",
       " u'contains(moniker)': False,\n",
       " u'contains(fakery)': False,\n",
       " u'contains(edouard)': False,\n",
       " u'contains(aughra)': False,\n",
       " u'contains(foregrounds)': False,\n",
       " u'contains(snobbish)': False,\n",
       " u'contains(printed)': False,\n",
       " u'contains(dispite)': False,\n",
       " u'contains(pulled)': False,\n",
       " u'contains(russ)': False,\n",
       " u'contains(admires)': False,\n",
       " u'contains(hurricaine)': False,\n",
       " u'contains(milked)': False,\n",
       " u'contains(master)': False,\n",
       " u'contains(spurting)': False,\n",
       " u'contains(oversimplified)': False,\n",
       " u'contains(overlord)': False,\n",
       " u'contains(goody)': False,\n",
       " u'contains(has)': True,\n",
       " u'contains(pringles)': False,\n",
       " u'contains(celebrates)': False,\n",
       " u'contains(domed)': False,\n",
       " u'contains(wrinkles)': False,\n",
       " u'contains(abandons)': False,\n",
       " u'contains(solaris)': False,\n",
       " u'contains(clumsiness)': False,\n",
       " u'contains(deductions)': False,\n",
       " u'contains(sugarplums)': False,\n",
       " u'contains(lombardo)': False,\n",
       " u'contains(humbled)': False,\n",
       " u'contains(beware)': False,\n",
       " u'contains(antichrist)': False,\n",
       " u'contains(cavity)': False,\n",
       " u'contains(anthropologists)': False,\n",
       " u'contains(typicalness)': False,\n",
       " u'contains(crowe)': False,\n",
       " u'contains(scramble)': False,\n",
       " u'contains(stoically)': False,\n",
       " u'contains(tanker)': False,\n",
       " u'contains(thanking)': False,\n",
       " u'contains(brunette)': False,\n",
       " u'contains(retells)': False,\n",
       " u'contains(rudnick)': False,\n",
       " u'contains(sidebars)': False,\n",
       " u'contains(interviewed)': False,\n",
       " u'contains(greenwald)': False,\n",
       " u'contains(brutally)': False,\n",
       " u'contains(misconstrued)': False,\n",
       " u'contains(piscapo)': False,\n",
       " u'contains(vocalized)': False,\n",
       " u'contains(negg)': False,\n",
       " u'contains(benigness)': False,\n",
       " u'contains(cigarettes)': False,\n",
       " u'contains(want)': False,\n",
       " u'contains(huge)': False,\n",
       " u'contains(mediocrity)': False,\n",
       " u'contains(opar)': False,\n",
       " u'contains(abortion)': False,\n",
       " u'contains(chap)': False,\n",
       " u'contains(paled)': False,\n",
       " u'contains(distraction)': False,\n",
       " u'contains(hypsy)': False,\n",
       " u'contains(elders)': False,\n",
       " u'contains(compass)': False,\n",
       " u'contains(circuitry)': False,\n",
       " u'contains(rebuilding)': False,\n",
       " u'contains(hat)': False,\n",
       " u'contains(marshall)': False,\n",
       " u'contains(professory)': False,\n",
       " u'contains(dismisses)': False,\n",
       " u'contains(oderkerk)': False,\n",
       " u'contains(bunker)': False,\n",
       " u'contains(spit)': False,\n",
       " u'contains(abject)': False,\n",
       " u'contains(s)': True,\n",
       " u'contains(underdone)': False,\n",
       " u'contains(starring)': False,\n",
       " u'contains(dared)': False,\n",
       " u'contains(antidote)': False,\n",
       " u'contains(igniting)': False,\n",
       " u'contains(touchy)': False,\n",
       " u'contains(doubts)': False,\n",
       " u'contains(erich)': False,\n",
       " u'contains(trails)': False,\n",
       " u'contains(socking)': False,\n",
       " u'contains(surfing)': False,\n",
       " u'contains(promotional)': False,\n",
       " u'contains(sponsorship)': False,\n",
       " u'contains(circumstances)': False,\n",
       " u'contains(zoe)': False,\n",
       " u'contains(process)': False,\n",
       " u'contains(sponsor)': False,\n",
       " u'contains(mortified)': False,\n",
       " u'contains(snickered)': False,\n",
       " u'contains(minah)': False,\n",
       " u'contains(soaked)': False,\n",
       " u'contains(entering)': False,\n",
       " u'contains(cheddar)': False,\n",
       " u'contains(soundbite)': False,\n",
       " u'contains(recombination)': False,\n",
       " u'contains(zigged)': False,\n",
       " u'contains(sublimated)': False,\n",
       " u'contains(niall)': False,\n",
       " u'contains(sixteen)': False,\n",
       " u'contains(harassing)': False,\n",
       " u'contains(loitered)': False,\n",
       " u'contains(dna)': False,\n",
       " u'contains(noblewoman)': False,\n",
       " u'contains(interogation)': False,\n",
       " u'contains(locusts)': False,\n",
       " u'contains(skirmish)': False,\n",
       " u'contains(unintentionally)': False,\n",
       " u'contains(bounce)': False,\n",
       " u'contains(rebounding)': False,\n",
       " u'contains(flipped)': False,\n",
       " u'contains(mori)': False,\n",
       " u'contains(cages)': False,\n",
       " u'contains(atlantis)': False,\n",
       " u'contains(sustaining)': False,\n",
       " u'contains(straighten)': False,\n",
       " u'contains(otto)': False,\n",
       " u'contains(flotsam)': False,\n",
       " u'contains(centuries)': False,\n",
       " u'contains(promiss)': False,\n",
       " u'contains(maude)': False,\n",
       " u'contains(hawkes)': False,\n",
       " u'contains(scott)': False,\n",
       " u'contains(edged)': False,\n",
       " u'contains(wrong)': True,\n",
       " u'contains(frenchmen)': False,\n",
       " u'contains(dilbert)': False,\n",
       " u'contains(indicative)': False,\n",
       " u'contains(bruckheimer)': False,\n",
       " u'contains(porpoise)': False,\n",
       " u'contains(uncharacteristically)': False,\n",
       " u'contains(ellen)': False,\n",
       " u'contains(stifle)': False,\n",
       " u'contains(stormare)': False,\n",
       " u'contains(wracked)': False,\n",
       " u'contains(encourage)': False,\n",
       " u'contains(wits)': False,\n",
       " u'contains(bios)': False,\n",
       " u'contains(fry)': False,\n",
       " u'contains(resonated)': False,\n",
       " u'contains(lyndon)': False,\n",
       " u'contains(stylings)': False,\n",
       " u'contains(rangers)': False,\n",
       " u'contains(marlo)': False,\n",
       " u'contains(stillness)': False,\n",
       " u'contains(arlington)': False,\n",
       " u'contains(scold)': False,\n",
       " u'contains(belt)': False,\n",
       " u'contains(mcferran)': False,\n",
       " u'contains(ching)': False,\n",
       " u'contains(stern)': False,\n",
       " u'contains(crackles)': False,\n",
       " u'contains(antwerp)': False,\n",
       " u'contains(remembrance)': False,\n",
       " u'contains(expressively)': False,\n",
       " u'contains(byline)': False,\n",
       " u'contains(admirer)': False,\n",
       " u'contains(squirt)': False,\n",
       " u'contains(saucy)': False,\n",
       " u'contains(clients)': False,\n",
       " u'contains(mirrors)': False,\n",
       " u'contains(gloating)': False,\n",
       " u'contains(herzfeld)': False,\n",
       " u'contains(inconsistent)': False,\n",
       " u'contains(untouched)': False,\n",
       " u'contains(whizzing)': False,\n",
       " u'contains(traumatizes)': False,\n",
       " u'contains(bouyant)': False,\n",
       " u'contains(initiated)': False,\n",
       " u'contains(yogi)': False,\n",
       " u'contains(wales)': False,\n",
       " u'contains(disintegrated)': False,\n",
       " u'contains(spotty)': False,\n",
       " u'contains(fisherman)': False,\n",
       " u'contains(greatness)': False,\n",
       " u'contains(neary)': False,\n",
       " u'contains(boiler)': False,\n",
       " u'contains(westlake)': False,\n",
       " u'contains(allan)': False,\n",
       " u'contains(organisations)': False,\n",
       " u'contains(premier)': False,\n",
       " u'contains(apace)': False,\n",
       " u'contains(matilda)': False,\n",
       " u'contains(belloq)': False,\n",
       " u'contains(glenne)': False,\n",
       " u'contains(nothin)': False,\n",
       " u'contains(implicit)': False,\n",
       " u'contains(macgowan)': False,\n",
       " u'contains(faded)': False,\n",
       " u'contains(ryan_)': False,\n",
       " u'contains(terrifyingly)': False,\n",
       " u'contains(awaiting)': False,\n",
       " u'contains(onofrio)': False,\n",
       " u'contains(goodman)': False,\n",
       " u'contains(caitlyn)': False,\n",
       " u'contains(virus)': False,\n",
       " u'contains(adroit)': False,\n",
       " u'contains(prostration)': False,\n",
       " u'contains(resorts)': False,\n",
       " u'contains(yugoslavians)': False,\n",
       " u'contains(signs)': False,\n",
       " u'contains(formulates)': False,\n",
       " u'contains(squabble)': False,\n",
       " u'contains(much)': False,\n",
       " u'contains(screaming)': False,\n",
       " u'contains(methods)': False,\n",
       " u'contains(despises)': False,\n",
       " u'contains(bogg)': False,\n",
       " u'contains(ony)': False,\n",
       " u'contains(excising)': False,\n",
       " u'contains(dahlings)': False,\n",
       " u'contains(makes)': False,\n",
       " u'contains(sympathize)': False,\n",
       " u'contains(stoicism)': False,\n",
       " u'contains(isaac)': False,\n",
       " u'contains(juergen)': False,\n",
       " u'contains(bolstered)': False,\n",
       " u'contains(blonde)': False,\n",
       " u'contains(aggressivelly)': False,\n",
       " u'contains(fruity)': False,\n",
       " u'contains(mutilated)': False,\n",
       " u'contains(mensch)': False,\n",
       " u'contains(behave)': False,\n",
       " u'contains(red)': False,\n",
       " u'contains(palladino)': False,\n",
       " u'contains(welcomes)': False,\n",
       " u'contains(seaman)': False,\n",
       " u'contains(couple)': False,\n",
       " u'contains(excitable)': False,\n",
       " u'contains(charges)': False,\n",
       " u'contains(rich)': False,\n",
       " u'contains(broader)': False,\n",
       " u'contains(shapely)': False,\n",
       " u'contains(betraying)': False,\n",
       " u'contains(gladys)': False,\n",
       " u'contains(wives)': False,\n",
       " u'contains(detectives)': False,\n",
       " u'contains(cobblers)': False,\n",
       " u'contains(unrecognizable)': False,\n",
       " u'contains(colonists)': False,\n",
       " u'contains(mintues)': False,\n",
       " u'contains(mystic)': False,\n",
       " u'contains(sportsmanship)': False,\n",
       " u'contains(benson)': False,\n",
       " u'contains(parachute)': False,\n",
       " u'contains(millieu)': False,\n",
       " u'contains(defensively)': False,\n",
       " u'contains(being)': False,\n",
       " u'contains(seminal)': False,\n",
       " u'contains(music)': False,\n",
       " u'contains(pull)': False,\n",
       " u'contains(90210)': False,\n",
       " u'contains(cloris)': False,\n",
       " u'contains(traumatised)': False,\n",
       " u'contains(diapers)': False,\n",
       " u'contains(replied)': False,\n",
       " u'contains(wachowskis)': False,\n",
       " u'contains(speedboat)': False,\n",
       " u'contains(krippendorf)': False,\n",
       " u'contains(hercules)': False,\n",
       " u'contains(meteorologist)': False,\n",
       " u'contains(bogglingly)': False,\n",
       " u'contains(smiling)': False,\n",
       " u'contains(pinto)': False,\n",
       " u'contains(bursting)': False,\n",
       " u'contains(modestly)': False,\n",
       " u'contains(climber)': False,\n",
       " u'contains(dodie)': False,\n",
       " u'contains(retell)': False,\n",
       " u'contains(wage)': False,\n",
       " u'contains(sela)': False,\n",
       " u'contains(henpecked)': False,\n",
       " u'contains(complications)': False,\n",
       " u'contains(pretzel)': False,\n",
       " u'contains(stifled)': False,\n",
       " u'contains(trys)': False,\n",
       " u'contains(reopens)': False,\n",
       " u'contains(babe)': False,\n",
       " u'contains(discerning)': False,\n",
       " u'contains(headset)': False,\n",
       " u'contains(stoker)': False,\n",
       " u'contains(doom)': False,\n",
       " u'contains(snuggles)': False,\n",
       " u'contains(cessation)': False,\n",
       " u'contains(abbreviated)': False,\n",
       " u'contains(enjoys)': False,\n",
       " u'contains(ideally)': False,\n",
       " u'contains(raped)': False,\n",
       " u'contains(_seven_nights_)': False,\n",
       " u'contains(haviland)': False,\n",
       " u'contains(confrontatory)': False,\n",
       " u'contains(charnel)': False,\n",
       " u'contains(pasadena)': False,\n",
       " u'contains(portentuous)': False,\n",
       " u'contains(titanium)': False,\n",
       " u'contains(abnormally)': False,\n",
       " u'contains(unwelcomed)': False,\n",
       " u'contains(procreate)': False,\n",
       " u'contains(keeble)': False,\n",
       " u'contains(sumptuous)': False,\n",
       " u'contains(conscious)': False,\n",
       " u'contains(regressive)': False,\n",
       " u'contains(embark)': False,\n",
       " u'contains(expound)': False,\n",
       " u'contains(wacked)': False,\n",
       " u'contains(engages)': False,\n",
       " u'contains(touchstone)': False,\n",
       " u'contains(amarcord)': False,\n",
       " u'contains(tsui)': False,\n",
       " u'contains(petrice)': False,\n",
       " u'contains(numeric)': False,\n",
       " u'contains(hotcakes)': False,\n",
       " u'contains(chlorine)': False,\n",
       " u'contains(locked)': False,\n",
       " u'contains(tremble)': False,\n",
       " u'contains(featherbrained)': False,\n",
       " u'contains(conceptually)': False,\n",
       " u'contains(genesis)': False,\n",
       " u'contains(meddled)': False,\n",
       " u'contains(awkwardness)': False,\n",
       " u'contains(friggin)': False,\n",
       " u'contains(mutinies)': False,\n",
       " u'contains(stoked)': False,\n",
       " u'contains(copious)': False,\n",
       " u'contains(revelatory)': False,\n",
       " u'contains(cronenberg)': False,\n",
       " u'contains(absorbs)': False,\n",
       " u'contains(rhythmless)': False,\n",
       " u'contains(burley)': False,\n",
       " u'contains(whoever)': False,\n",
       " u'contains(mosques)': False,\n",
       " u'contains(hag)': False,\n",
       " u'contains(bianca)': False,\n",
       " u'contains(anonymously)': False,\n",
       " u'contains(prior)': False,\n",
       " u'contains(procreating)': False,\n",
       " u'contains(carelessly)': False,\n",
       " u'contains(chad)': False,\n",
       " u'contains(forwarned)': False,\n",
       " u'contains(maker)': False,\n",
       " u'contains(fin)': False,\n",
       " u'contains(bulow)': False,\n",
       " u'contains(informants)': False,\n",
       " u'contains(nora)': False,\n",
       " u'contains(ballet)': False,\n",
       " u'contains(chomped)': False,\n",
       " u'contains(sommers)': False,\n",
       " u'contains(spacemusic)': False,\n",
       " u'contains(bastad)': False,\n",
       " u'contains(rabal)': False,\n",
       " u'contains(golden)': False,\n",
       " u'contains(whoaaaaaa)': False,\n",
       " u'contains(toning)': False,\n",
       " u'contains(overturned)': False,\n",
       " u'contains(handcuffs)': False,\n",
       " u'contains(nonsensical)': False,\n",
       " u'contains(preservation)': False,\n",
       " u'contains(eyelids)': False,\n",
       " u'contains(considerably)': False,\n",
       " u'contains(rioters)': False,\n",
       " u'contains(pressed)': False,\n",
       " u'contains(tightened)': False,\n",
       " u'contains(extent)': False,\n",
       " u'contains(wandering)': False,\n",
       " u'contains(dozen)': False,\n",
       " u'contains(remastered)': False,\n",
       " u'contains(deserves)': False,\n",
       " u'contains(beowolf)': False,\n",
       " u'contains(warfield)': False,\n",
       " u'contains(audaciously)': False,\n",
       " u'contains(majorino)': False,\n",
       " u'contains(featherweight)': False,\n",
       " u'contains(superiority)': False,\n",
       " u'contains(portents)': False,\n",
       " u'contains(harvests)': False,\n",
       " u'contains(colleague)': False,\n",
       " u'contains(fewer)': False,\n",
       " u'contains(turned)': False,\n",
       " u'contains(wunderkind)': False,\n",
       " u'contains(opportunists)': False,\n",
       " u'contains(goggins)': False,\n",
       " u'contains(zellwegger)': False,\n",
       " u'contains(middleton)': False,\n",
       " u'contains(atlantic)': False,\n",
       " u'contains(pimp)': False,\n",
       " u'contains(grounded)': False,\n",
       " u'contains(doreen)': False,\n",
       " u'contains(caan)': False,\n",
       " u'contains(unknowns)': False,\n",
       " u'contains(seldes)': False,\n",
       " u'contains(spine)': False,\n",
       " u'contains(measurements)': False,\n",
       " u'contains(stifler)': False,\n",
       " u'contains(heebie)': False,\n",
       " u'contains(turner)': False,\n",
       " u'contains(complicates)': False,\n",
       " u'contains(absurdities)': False,\n",
       " u'contains(unredeemable)': False,\n",
       " u'contains(navigators)': False,\n",
       " u'contains(rice)': False,\n",
       " u'contains(graffiti)': False,\n",
       " u'contains(oxymoron)': False,\n",
       " u'contains(sweaty)': False,\n",
       " u'contains(smashed)': False,\n",
       " u'contains(sorted)': False,\n",
       " u'contains(joely)': False,\n",
       " u'contains(phallus)': False,\n",
       " u'contains(roll)': False,\n",
       " u'contains(damnit)': False,\n",
       " u'contains(fugitives)': False,\n",
       " u'contains(shootout)': False,\n",
       " u'contains(saddened)': False,\n",
       " u'contains(span)': False,\n",
       " u'contains(condolences)': False,\n",
       " u'contains(koppelman)': False,\n",
       " u'contains(gardenia)': False,\n",
       " u'contains(locklear)': False,\n",
       " u'contains(cheekbones)': False,\n",
       " u'contains(horndog)': False,\n",
       " u'contains(topping)': False,\n",
       " u'contains(simplistic)': False,\n",
       " u'contains(foregin)': False,\n",
       " u'contains(muth)': False,\n",
       " u'contains(bamboo)': False,\n",
       " u'contains(leary)': False,\n",
       " u'contains(18th)': False,\n",
       " u'contains(gisbourne)': False,\n",
       " u'contains(oingo)': False,\n",
       " u'contains(natured)': False,\n",
       " u'contains(mistic)': False,\n",
       " u'contains(disaster_)': False,\n",
       " u'contains(ugc)': False,\n",
       " u'contains(spins)': False,\n",
       " u'contains(euphegenia)': False,\n",
       " u'contains(overpopulated)': False,\n",
       " u'contains(mba)': False,\n",
       " u'contains(more)': False,\n",
       " u'contains(lanky)': False,\n",
       " u'contains(oldies)': False,\n",
       " u'contains(_titus_andronicus_)': False,\n",
       " u'contains(thora)': False,\n",
       " u'contains(rooker)': False,\n",
       " u'contains(copulate)': False,\n",
       " u'contains(islamic)': False,\n",
       " u'contains(sympathetically)': False,\n",
       " u'contains(dolph)': False,\n",
       " u'contains(dynamics)': False,\n",
       " u'contains(convincingly)': False,\n",
       " u'contains(bedridden)': False,\n",
       " u'contains(woody)': False,\n",
       " u'contains(sunny)': False,\n",
       " u'contains(combatants)': False,\n",
       " u'contains(scenic)': False,\n",
       " u'contains(cannibals)': False,\n",
       " u'contains(allah)': False,\n",
       " u'contains(capoeira)': False,\n",
       " u'contains(petey)': False,\n",
       " u'contains(codename)': False,\n",
       " u'contains(goregeous)': False,\n",
       " u'contains(beloved)': False,\n",
       " u'contains(hav)': False,\n",
       " u'contains(macho)': False,\n",
       " u'contains(chins)': False,\n",
       " u'contains(goddamn)': False,\n",
       " u'contains(desiring)': False,\n",
       " u'contains(watcher)': False,\n",
       " u'contains(congratulations)': False,\n",
       " u'contains(velda)': False,\n",
       " u'contains(stylistics)': False,\n",
       " u'contains(crowbar)': False,\n",
       " u'contains(ahern)': False,\n",
       " u'contains(pyroclastic)': False,\n",
       " u'contains(incongruent)': False,\n",
       " u'contains(isacsson)': False,\n",
       " u'contains(shocks)': False,\n",
       " u'contains(hideaway)': False,\n",
       " u'contains(alexandre)': False,\n",
       " u'contains(chirping)': False,\n",
       " u'contains(gunshot)': False,\n",
       " u'contains(nervousness)': False,\n",
       " u'contains(retroactive)': False,\n",
       " u'contains(nighthawks)': False,\n",
       " u'contains(cranked)': False,\n",
       " u'contains(disasters)': False,\n",
       " u'contains(quaint)': False,\n",
       " u'contains(wrinkled)': False,\n",
       " u'contains(frontgate)': False,\n",
       " u'contains(shadow)': False,\n",
       " u'contains(onw)': False,\n",
       " u'contains(binges)': False,\n",
       " u'contains(unjust)': False,\n",
       " u'contains(meantime)': False,\n",
       " u'contains(scorces)': False,\n",
       " u'contains(mirabella)': False,\n",
       " u'contains(charmed)': False,\n",
       " u'contains(stays)': False,\n",
       " u'contains(disgrace)': False,\n",
       " u'contains(yakov)': False,\n",
       " u'contains(therefore)': False,\n",
       " u'contains(grapples)': False,\n",
       " u'contains(painkiller)': False,\n",
       " u'contains(waning)': False,\n",
       " u'contains(recycles)': False,\n",
       " u'contains(symmetry)': False,\n",
       " u'contains(wants)': False,\n",
       " u'contains(f)': False,\n",
       " u'contains(melinda)': False,\n",
       " u'contains(magma)': False,\n",
       " u'contains(falstaff)': False,\n",
       " u'contains(battleships)': False,\n",
       " u'contains(shenanigans)': False,\n",
       " u'contains(locations)': False,\n",
       " u'contains(dumbest)': False,\n",
       " u'contains(diggler)': False,\n",
       " u'contains(dismantling)': False,\n",
       " u'contains(canran)': False,\n",
       " u'contains(ahmet)': False,\n",
       " u'contains(prizes)': False,\n",
       " u'contains(longo)': False,\n",
       " u'contains(overcooked)': False,\n",
       " u'contains(stereotypical)': False,\n",
       " u'contains(vivien)': False,\n",
       " u'contains(negativity)': False,\n",
       " u'contains(impacted)': False,\n",
       " u'contains(snugly)': False,\n",
       " u'contains(collides)': False,\n",
       " u'contains(stipulate)': False,\n",
       " u'contains(stifles)': False,\n",
       " u'contains(originally)': False,\n",
       " u'contains(upstate)': False,\n",
       " u'contains(gigolo)': False,\n",
       " u'contains(delectably)': False,\n",
       " u'contains(unjustifyably)': False,\n",
       " u'contains(pivot)': False,\n",
       " u'contains(spewing)': False,\n",
       " u'contains(former)': False,\n",
       " u'contains(fedoras)': False,\n",
       " u'contains(devito)': False,\n",
       " u'contains(telescope)': False,\n",
       " u'contains(collectively)': False,\n",
       " u'contains(savini)': False,\n",
       " u'contains(periodic)': False,\n",
       " u'contains(skewer)': False,\n",
       " u'contains(disasterous)': False,\n",
       " u'contains(enthrall)': False,\n",
       " u'contains(displays)': False,\n",
       " u'contains(carrion)': False,\n",
       " u'contains(weirdoes)': False,\n",
       " u'contains(buliwyf)': False,\n",
       " u'contains(palms)': False,\n",
       " u'contains(omaha)': False,\n",
       " u'contains(beesley)': False,\n",
       " u'contains(spyglass)': False,\n",
       " u'contains(spam)': False,\n",
       " u'contains(dyer)': False,\n",
       " u'contains(feeling)': False,\n",
       " u'contains(======)': False,\n",
       " u'contains(fueled)': False,\n",
       " u'contains(aussies)': False,\n",
       " u'contains(tenacious)': False,\n",
       " u'contains(restraints)': False,\n",
       " u'contains(doos)': False,\n",
       " u'contains(modest)': False,\n",
       " u'contains(chu)': False,\n",
       " u'contains(guitry)': False,\n",
       " u'contains(boarded)': False,\n",
       " u'contains(fistfights)': False,\n",
       " u'contains(trekkie)': False,\n",
       " u'contains(gripe)': False,\n",
       " u'contains(universally)': False,\n",
       " u'contains(lingering)': False,\n",
       " u'contains(breakfast)': False,\n",
       " u'contains(societal)': False,\n",
       " u'contains(courses)': False,\n",
       " u'contains(outburst)': False,\n",
       " u'contains(hatchette)': False,\n",
       " u'contains(obscured)': False,\n",
       " u'contains(floated)': False,\n",
       " u'contains(limburgher)': False,\n",
       " u'contains(gorman)': False,\n",
       " u'contains(pissant)': False,\n",
       " u'contains(mussenden)': False,\n",
       " u'contains(underlining)': False,\n",
       " u'contains(duet)': False,\n",
       " u'contains(estimate)': False,\n",
       " u'contains(testaments)': False,\n",
       " u'contains(francesca)': False,\n",
       " u'contains(lameness)': False,\n",
       " u'contains(bliss)': False,\n",
       " u'contains(slothful)': False,\n",
       " u'contains(bazooms)': False,\n",
       " u'contains(smirk)': False,\n",
       " u'contains(haw)': False,\n",
       " u'contains(glowers)': False,\n",
       " u'contains(learn)': False,\n",
       " u'contains(playfulness)': False,\n",
       " u'contains(hackwork)': False,\n",
       " u'contains(distractedness)': False,\n",
       " u'contains(perturbed)': False,\n",
       " u'contains(aboard)': False,\n",
       " u'contains(enliven)': False,\n",
       " u'contains(briesewitz)': False,\n",
       " u'contains(exasperating)': False,\n",
       " u'contains(ambiguities)': False,\n",
       " u'contains(waitering)': False,\n",
       " u'contains(israel)': False,\n",
       " u'contains(insane)': False,\n",
       " u'contains(kfc)': False,\n",
       " u'contains(spin)': False,\n",
       " u'contains(tracking)': False,\n",
       " u'contains(massironi)': False,\n",
       " u'contains(directs)': False,\n",
       " u'contains(limitless)': False,\n",
       " u'contains(slaver)': False,\n",
       " u'contains(chrissy)': False,\n",
       " u'contains(lang)': False,\n",
       " u'contains(droning)': False,\n",
       " u'contains(deserved)': False,\n",
       " u'contains(bozo)': False,\n",
       " u'contains(apropos)': False,\n",
       " u'contains(subplots)': False,\n",
       " u'contains(parlay)': False,\n",
       " u'contains(hai)': False,\n",
       " u'contains(healed)': False,\n",
       " u'contains(collided)': False,\n",
       " u'contains(expanded)': False,\n",
       " u'contains(academic)': False,\n",
       " u'contains(chaz)': False,\n",
       " u'contains(pridefully)': False,\n",
       " u'contains(twisting)': False,\n",
       " u'contains(whacking)': False,\n",
       " u'contains(francesco)': False,\n",
       " u'contains(turkish)': False,\n",
       " u'contains(extend)': False,\n",
       " u'contains(sock)': False,\n",
       " u'contains(kriss)': False,\n",
       " u'contains(tolan)': False,\n",
       " u'contains(concentrates)': False,\n",
       " u'contains(friend)': False,\n",
       " u'contains(small)': False,\n",
       " u'contains(landslide)': False,\n",
       " u'contains(himalayas)': False,\n",
       " u'contains(clarkson)': False,\n",
       " u'contains(delicately)': False,\n",
       " u'contains(punishable)': False,\n",
       " u'contains(chow)': False,\n",
       " u'contains(simplify)': False,\n",
       " u'contains(romper)': False,\n",
       " u'contains(rickman)': False,\n",
       " u'contains(shipments)': False,\n",
       " u'contains(inanimate)': False,\n",
       " u'contains(gershon)': False,\n",
       " u'contains(unsubstantial)': False,\n",
       " u'contains(neophytes)': False,\n",
       " u'contains(unsinkable)': False,\n",
       " u'contains(bohemians)': False,\n",
       " u'contains(surname)': False,\n",
       " u'contains(noteables)': False,\n",
       " u'contains(pooper)': False,\n",
       " u'contains(boozed)': False,\n",
       " u'contains(villiany)': False,\n",
       " u'contains(cuisine)': False,\n",
       " u'contains(arkin)': False,\n",
       " u'contains(wires)': False,\n",
       " u'contains(barrel)': False,\n",
       " u'contains(coaxed)': False,\n",
       " u'contains(duper)': False,\n",
       " u'contains(nordoff)': False,\n",
       " u'contains(aboslutely)': False,\n",
       " u'contains(jonnie)': False,\n",
       " u'contains(flipper)': False,\n",
       " u'contains(issac)': False,\n",
       " u'contains(uplifting)': False,\n",
       " u'contains(satire)': False,\n",
       " u'contains(mesmerize)': False,\n",
       " u'contains(norm)': False,\n",
       " u'contains(sects)': False,\n",
       " u'contains(decapitation)': False,\n",
       " u'contains(weendigo)': False,\n",
       " u'contains(union)': False,\n",
       " u'contains(admired)': False,\n",
       " u'contains(municipality)': False,\n",
       " u'contains(initiate)': False,\n",
       " u'contains(sked)': False,\n",
       " u'contains(freehold)': False,\n",
       " u'contains(butchers)': False,\n",
       " u'contains(conscription)': False,\n",
       " u'contains(tempted)': False,\n",
       " u'contains(engaged)': False,\n",
       " u'contains(soviets)': False,\n",
       " u'contains(milquetoast)': False,\n",
       " u'contains(scrubs)': False,\n",
       " u'contains(binding)': False,\n",
       " u'contains(.)': True,\n",
       " u'contains(ire)': False,\n",
       " u'contains(neurologist)': False,\n",
       " u'contains(address)': False,\n",
       " u'contains(tantalizingly)': False,\n",
       " u'contains(shockwave)': False,\n",
       " u'contains(dickinson)': False,\n",
       " u'contains(tightrope)': False,\n",
       " u'contains(transported)': False,\n",
       " u'contains(symbolise)': False,\n",
       " u'contains(plucked)': False,\n",
       " u'contains(assuring)': False,\n",
       " u'contains(investment)': False,\n",
       " u'contains(tricks)': False,\n",
       " u'contains(r2)': False,\n",
       " u'contains(phelps)': False,\n",
       " u'contains(door)': False,\n",
       " u'contains(ghostbusters)': False,\n",
       " u'contains(emmylou)': False,\n",
       " u'contains(houseman)': False,\n",
       " u'contains(proprietor)': False,\n",
       " u'contains(shandling)': False,\n",
       " u'contains(masturbates)': False,\n",
       " u'contains(sonorra)': False,\n",
       " u'contains(obliges)': False,\n",
       " u'contains(bicentennial)': False,\n",
       " u'contains(faulkner)': False,\n",
       " u'contains(minnie)': False,\n",
       " u'contains(ambling)': False,\n",
       " u'contains(mythos)': False,\n",
       " u'contains(courageously)': False,\n",
       " u'contains(marquee)': False,\n",
       " u'contains(storyboarded)': False,\n",
       " u'contains(hairdresser)': False,\n",
       " u'contains(following)': False,\n",
       " u'contains(guamo)': False,\n",
       " u'contains(horsing)': False,\n",
       " u'contains(sedition)': False,\n",
       " u'contains(redirection)': False,\n",
       " u'contains(roots)': False,\n",
       " u'contains(painstakingly)': False,\n",
       " u'contains(contrastingly)': False,\n",
       " u'contains(dues)': False,\n",
       " u'contains(huddled)': False,\n",
       " u'contains(disengaging)': False,\n",
       " u'contains(disgraced)': False,\n",
       " u'contains(crater)': False,\n",
       " u'contains(fabric)': False,\n",
       " u'contains(zippers)': False,\n",
       " u'contains(attenuated)': False,\n",
       " u'contains(cultivating)': False,\n",
       " u'contains(gown)': False,\n",
       " u'contains(positively)': False,\n",
       " u'contains(singed)': False,\n",
       " u'contains(twotg)': False,\n",
       " u'contains(disturbed)': False,\n",
       " u'contains(liebes)': False,\n",
       " u'contains(peppy)': False,\n",
       " u'contains(vexing)': False,\n",
       " u'contains(holdover)': False,\n",
       " u'contains(singer)': False,\n",
       " u'contains(shawn)': False,\n",
       " u'contains(sprinkle)': False,\n",
       " u'contains(consoles)': False,\n",
       " u'contains(sensitively)': False,\n",
       " u'contains(midget)': False,\n",
       " u'contains(land)': False,\n",
       " u'contains(katie)': False,\n",
       " u'contains(jitters)': False,\n",
       " u'contains(corrects)': False,\n",
       " u'contains(magyuver)': False,\n",
       " u'contains(indiscretion)': False,\n",
       " u'contains(locks)': False,\n",
       " u'contains(integrating)': False,\n",
       " u'contains(loco)': False,\n",
       " u'contains(ratttz)': False,\n",
       " u'contains(flyboy)': False,\n",
       " u'contains(hessian)': False,\n",
       " u'contains(budget)': False,\n",
       " u'contains(turvy)': False,\n",
       " u'contains(occurs)': False,\n",
       " u'contains(kroon)': False,\n",
       " u'contains(customer)': False,\n",
       " u'contains(hugh)': False,\n",
       " u'contains(ayla)': False,\n",
       " u'contains(crowdpleasing)': False,\n",
       " u'contains(embrassment)': False,\n",
       " u'contains(hockley)': False,\n",
       " ...}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document_features(movie_reviews.words('pos/cv957_8737.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Note\n",
    "\n",
    "The reason that we compute the set of all words in a document in [3], rather than just checking if word in document, is that checking whether a word occurs in a set is much faster than checking whether it occurs in a list (4.7).\n",
    "\n",
    "Now that we've defined our feature extractor, we can use it to train a classifier to label new movie reviews (6.5). To check how reliable the resulting classifier is, we compute its accuracy on the test set [1]. And once again, we can use show_most_informative_features() to find out which features the classifier found to be most informative [2]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "featuresets = [(document_features(d), c) for (d,c) in documents]\n",
    "train_set, test_set = featuresets[100:], featuresets[:100]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.65"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.classify.accuracy(classifier, test_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "    contains(mediocrity) = True              neg : pos    =      7.7 : 1.0\n",
      "          contains(sans) = True              neg : pos    =      7.7 : 1.0\n",
      "     contains(dismissed) = True              pos : neg    =      7.0 : 1.0\n",
      "   contains(bruckheimer) = True              neg : pos    =      6.3 : 1.0\n",
      "   contains(overwhelmed) = True              pos : neg    =      6.3 : 1.0\n"
     ]
    }
   ],
   "source": [
    "classifier.show_most_informative_features(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part-of-Speech Tagging\n",
    "\n",
    "In 5 we built a regular expression tagger that chooses a part-of-speech tag for a word by looking at the internal make-up of the word. However, this regular expression tagger had to be hand-crafted. Instead, we can train a classifier to work out which suffixes are most informative. Let's begin by finding out what the most common suffixes are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import brown\n",
    "suffix_fdist = nltk.FreqDist()\n",
    "for word in brown.words():\n",
    "    word = word.lower()\n",
    "    suffix_fdist.update(word[-1:])\n",
    "    suffix_fdist.update(word[-2:])\n",
    "    suffix_fdist.update(word[-3:])\n",
    "    \n",
    "# Note that I used *.update instead of *.inc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'!',\n",
       " u'%',\n",
       " u'$',\n",
       " u\"'\",\n",
       " u'&',\n",
       " u')',\n",
       " u'(',\n",
       " u'+',\n",
       " u'*',\n",
       " u'-',\n",
       " u',',\n",
       " u'/',\n",
       " u'.',\n",
       " u'1',\n",
       " u'0',\n",
       " u'3',\n",
       " u'2',\n",
       " u'5',\n",
       " u'4',\n",
       " u'7',\n",
       " u'6',\n",
       " u'9',\n",
       " u'8',\n",
       " u';',\n",
       " u':',\n",
       " u'?',\n",
       " u'[',\n",
       " u']',\n",
       " u'a',\n",
       " u'`',\n",
       " u'c',\n",
       " u'b',\n",
       " u'e',\n",
       " u'd',\n",
       " u'g',\n",
       " u'f',\n",
       " u'i',\n",
       " u'h',\n",
       " u'k',\n",
       " u'j',\n",
       " u'm',\n",
       " u'l',\n",
       " u'o',\n",
       " u'n',\n",
       " u'q',\n",
       " u'p',\n",
       " u's',\n",
       " u'r',\n",
       " u'u',\n",
       " u't',\n",
       " u'w',\n",
       " u'v',\n",
       " u'y',\n",
       " u'x',\n",
       " u'{',\n",
       " u'z',\n",
       " u'}']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_suffixes = suffix_fdist.keys()[:100]\n",
    "common_suffixes "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define a feature extractor function which checks a given word for these suffixes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pos_features(word):\n",
    "    features = {}\n",
    "    for suffix in common_suffixes:\n",
    "        features['endswith(%s)' % suffix] = word.lower().endswith(suffix)\n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature extraction functions behave like tinted glasses, highlighting some of the properties (colors) in our data and making it impossible to see other properties. The classifier will rely exclusively on these highlighted properties when determining how to label inputs. In this case, the classifier will make its decisions based only on information about which of the common suffixes (if any) a given word has.\n",
    "\n",
    "Now that we've defined our feature extractor, we can use it to train a new \"decision tree\" classifier (to be discussed in 6.4):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tagged_words = brown.tagged_words(categories='news')\n",
    "featuresets = [(pos_features(n), g) for (n,g) in tagged_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "size = int(len(featuresets) * 0.1)\n",
    "train_set, test_set = featuresets[size:], featuresets[:size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.44266534062655394"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = nltk.DecisionTreeClassifier.train(train_set)\n",
    "nltk.classify.accuracy(classifier, test_set)\n",
    "# not as good as the one in the book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'NNS'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.classify(pos_features('cats'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One nice feature of decision tree models is that they are often fairly easy to interpret — we can even instruct NLTK to print them out as pseudocode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if endswith(,) == False: \n",
      "  if endswith(s) == False: \n",
      "    if endswith(e) == False: \n",
      "      if endswith(.) == False: return u'.'\n",
      "      if endswith(.) == True: return u'.'\n",
      "    if endswith(e) == True: return u'AT'\n",
      "  if endswith(s) == True: return u'NNS'\n",
      "if endswith(,) == True: return u','\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print (classifier.pseudocode(depth=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Note: below is based on book results]\n",
    "Here, we can see that the classifier begins by checking whether a word ends with a comma — if so, then it will receive the special tag \",\". Next, the classifier checks if the word ends in \"the\", in which case it's almost certainly a determiner. This \"suffix\" gets used early by the decision tree because the word \"the\" is so common. Continuing on, the classifier checks if the word ends in \"s\". If so, then it's most likely to receive the verb tag VBZ (unless it's the word \"is\", which has a special tag BEZ), and if not, then it's most likely a noun (unless it's the punctuation mark \".\"). The actual classifier contains further nested if-then statements below the ones shown here, but the depth=4 argument just displays the top portion of the decision tree.\n",
    "\n",
    "### Exploiting Context\n",
    "\n",
    "By augmenting the feature extraction function, we could modify this part-of-speech tagger to leverage a variety of other word-internal features, such as the length of the word, the number of syllables it contains, or its prefix. However, as long as the feature extractor just looks at the target word, we have no way to add features that depend on the context that the word appears in. But contextual features often provide powerful clues about the correct tag — for example, when tagging the word \"fly,\" knowing that the previous word is \"a\" will allow us to determine that it is functioning as a noun, not a verb.\n",
    "\n",
    "In order to accommodate features that depend on a word's context, we must revise the pattern that we used to define our feature extractor. Instead of just passing in the word to be tagged, we will pass in a complete (untagged) sentence, along with the index of the target word. This approach is demonstrated in 6.6, which employs a context-dependent feature extractor to define a part of speech tag classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pos_features(sentence, i): # [1]\n",
    "    features = {\"suffix(1)\": sentence[i][-1:],\n",
    "                \"suffix(2)\": sentence[i][-2:],\n",
    "                \"suffix(3)\": sentence[i][-3:]}\n",
    "    if i == 0:\n",
    "        features[\"prev-word\"] = \"<START>\"\n",
    "    else:\n",
    "        features[\"prev-word\"] = sentence[i-1]\n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'prev-word': u'an',\n",
       " 'suffix(1)': u'n',\n",
       " 'suffix(2)': u'on',\n",
       " 'suffix(3)': u'ion'}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_features(brown.sents()[0], 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7891596220785678"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagged_sents = brown.tagged_sents(categories='news')\n",
    "featuresets = []\n",
    "for tagged_sent in tagged_sents:\n",
    "    untagged_sent = nltk.tag.untag(tagged_sent)\n",
    "    for i, (word, tag) in enumerate(tagged_sent):\n",
    "        featuresets.append( (pos_features(untagged_sent, i), tag) )\n",
    "\n",
    "size = int(len(featuresets) * 0.1)\n",
    "train_set, test_set = featuresets[size:], featuresets[:size]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "\n",
    "nltk.classify.accuracy(classifier, test_set)\n",
    "\n",
    "# Example 6.6 (code_suffix_pos_tag.py): Figure 6.6: A part-of-speech classifier \n",
    "# whose feature detector examines the context in which a word appears in order \n",
    "# to determine which part of speech tag should be assigned. In particular, the \n",
    "# identity of the previous word is included as a feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its clear that exploiting contextual features improves the performance of our part-of-speech tagger. For example, the classifier learns that a word is likely to be a noun if it comes immediately after the word \"large\" or the word \"gubernatorial\". However, it is unable to learn the generalization that a word is probably a noun if it follows an adjective, because it doesn't have access to the previous word's part-of-speech tag. In general, simple classifiers always treat each input as independent from all other inputs. In many contexts, this makes perfect sense. For example, decisions about whether names tend to be male or female can be made on a case-by-case basis. However, there are often cases, such as part-of-speech tagging, where we are interested in solving classification problems that are closely related to one another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequence Classification\n",
    "\n",
    "In order to capture the dependencies between related classification tasks, we can use joint classifier models, which choose an appropriate labeling for a collection of related inputs. In the case of part-of-speech tagging, a variety of different sequence classifier models can be used to jointly choose part-of-speech tags for all the words in a given sentence.\n",
    "\n",
    "One sequence classification strategy, known as consecutive classification or greedy sequence classification, is to find the most likely class label for the first input, then to use that answer to help find the best label for the next input. The process can then be repeated until all of the inputs have been labeled. This is the approach that was taken by the bigram tagger from 5.5, which began by choosing a part-of-speech tag for the first word in the sentence, and then chose the tag for each subsequent word based on the word itself and the predicted tag for the previous word.\n",
    "\n",
    "This strategy is demonstrated in 6.7. First, we must augment our feature extractor function to take a history argument, which provides a list of the tags that we've predicted for the sentence so far [1]. Each tag in history corresponds with a word in sentence. But note that history will only contain tags for words we've already classified, that is, words to the left of the target word. Thus, while it is possible to look at some features of words to the right of the target word, it is not possible to look at the tags for those words (since we haven't generated them yet).\n",
    "\n",
    "Having defined a feature extractor, we can proceed to build our sequence classifier [2]. During training, we use the annotated tags to provide the appropriate history to the feature extractor, but when tagging new sentences, we generate the history list based on the output of the tagger itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pos_features(sentence, i, history): # [1]\n",
    "    features = {\"suffix(1)\": sentence[i][-1:],\n",
    "                \"suffix(2)\": sentence[i][-2:],\n",
    "                \"suffix(3)\": sentence[i][-3:]}\n",
    "    if i == 0:\n",
    "        features[\"prev-word\"] = \"<START>\"\n",
    "        features[\"prev-tag\"] = \"<START>\"\n",
    "    else:\n",
    "        features[\"prev-word\"] = sentence[i-1]\n",
    "        features[\"prev-tag\"] = history[i-1]\n",
    "    return features\n",
    "\n",
    "class ConsecutivePosTagger(nltk.TaggerI): # [2]\n",
    "\n",
    "    def __init__(self, train_sents):\n",
    "        train_set = []\n",
    "        for tagged_sent in train_sents:\n",
    "            untagged_sent = nltk.tag.untag(tagged_sent)\n",
    "            history = []\n",
    "            for i, (word, tag) in enumerate(tagged_sent):\n",
    "                featureset = pos_features(untagged_sent, i, history)\n",
    "                train_set.append( (featureset, tag) )\n",
    "                history.append(tag)\n",
    "        self.classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "\n",
    "    def tag(self, sentence):\n",
    "        history = []\n",
    "        for i, word in enumerate(sentence):\n",
    "            featureset = pos_features(sentence, i, history)\n",
    "            tag = self.classifier.classify(featureset)\n",
    "            history.append(tag)\n",
    "        return zip(sentence, history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.798052851182\n"
     ]
    }
   ],
   "source": [
    "tagged_sents = brown.tagged_sents(categories='news')\n",
    "size = int(len(tagged_sents) * 0.1)\n",
    "train_sents, test_sents = tagged_sents[size:], tagged_sents[:size]\n",
    "tagger = ConsecutivePosTagger(train_sents)\n",
    "print (tagger.evaluate(test_sents))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Methods for Sequence Classification\n",
    "\n",
    "One shortcoming of this approach is that we commit to every decision that we make. For example, if we decide to label a word as a noun, but later find evidence that it should have been a verb, there's no way to go back and fix our mistake. One solution to this problem is to adopt a transformational strategy instead. Transformational joint classifiers work by creating an initial assignment of labels for the inputs, and then iteratively refining that assignment in an attempt to repair inconsistencies between related inputs. The Brill tagger, described in (1), is a good example of this strategy.\n",
    "\n",
    "Another solution is to assign scores to all of the possible sequences of part-of-speech tags, and to choose the sequence whose overall score is highest. This is the approach taken by Hidden Markov Models. Hidden Markov Models are similar to consecutive classifiers in that they look at both the inputs and the history of predicted tags. However, rather than simply finding the single best tag for a given word, they generate a probability distribution over tags. These probabilities are then combined to calculate probability scores for tag sequences, and the tag sequence with the highest probability is chosen. Unfortunately, the number of possible tag sequences is quite large. Given a tag set with 30 tags, there are about 600 trillion (3010) ways to label a 10-word sentence. In order to avoid considering all these possible sequences separately, Hidden Markov Models require that the feature extractor only look at the most recent tag (or the most recent n tags, where n is fairly small). Given that restriction, it is possible to use dynamic programming (4.7) to efficiently find the most likely tag sequence. In particular, for each consecutive word index i, a score is computed for each possible current and previous tag. This same basic approach is taken by two more advanced models, called Maximum Entropy Markov Models and Linear-Chain Conditional Random Field Models; but different algorithms are used to find scores for tag sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2   Further Examples of Supervised Classification\n",
    "\n",
    "### Sentence Segmentation\n",
    "\n",
    "Sentence segmentation can be viewed as a classification task for punctuation: whenever we encounter a symbol that could possibly end a sentence, such as a period or a question mark, we have to decide whether it terminates the preceding sentence.\n",
    "\n",
    "The first step is to obtain some data that has already been segmented into sentences and convert it into a form that is suitable for extracting features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sents = nltk.corpus.treebank_raw.sents()\n",
    "tokens = []\n",
    "boundaries = set()\n",
    "offset = 0\n",
    "for sent in nltk.corpus.treebank_raw.sents():\n",
    "    tokens.extend(sent)\n",
    "    offset += len(sent)\n",
    "    boundaries.add(offset-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, tokens is a merged list of tokens from the individual sentences, and boundaries is a set containing the indexes of all sentence-boundary tokens. Next, we need to specify the features of the data that will be used in order to decide whether punctuation indicates a sentence-boundary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def punct_features(tokens, i):\n",
    "    return {'next-word-capitalized': tokens[i+1][0].isupper(),\n",
    "            'prevword': tokens[i-1].lower(),\n",
    "            'punct': tokens[i],\n",
    "            'prev-word-is-one-char': len(tokens[i-1]) == 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this feature extractor, we can create a list of labeled featuresets by selecting all the punctuation tokens, and tagging whether they are boundary tokens or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "featuresets = [(punct_features(tokens, i), (i in boundaries))\n",
    "               for i in range(1, len(tokens)-1)\n",
    "               if tokens[i] in '.?!']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using these featuresets, we can train and evaluate a punctuation classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.936026936026936"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "size = int(len(featuresets) * 0.1)\n",
    "train_set, test_set = featuresets[size:], featuresets[:size]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "nltk.classify.accuracy(classifier, test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use this classifier to perform sentence segmentation, we simply check each punctuation mark to see whether it's labeled as a boundary; and divide the list of words at the boundary marks. The listing in 6.8 shows how this can be done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def segment_sentences(words):\n",
    "    start = 0\n",
    "    sents = []\n",
    "    for i, word in enumerate(words):\n",
    "        if word in '.?!' and classifier.classify(punct_features(words, i)) == True:\n",
    "            sents.append(words[start:i+1])\n",
    "            start = i+1\n",
    "    if start < len(words):\n",
    "        sents.append(words[start:])\n",
    "    return sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identifying Dialogue Act Types\n",
    "\n",
    "When processing dialogue, it can be useful to think of utterances as a type of action performed by the speaker. This interpretation is most straightforward for performative statements such as \"I forgive you\" or \"I bet you can't climb that hill.\" But greetings, questions, answers, assertions, and clarifications can all be thought of as types of speech-based actions. Recognizing the dialogue acts underlying the utterances in a dialogue can be an important first step in understanding the conversation.\n",
    "\n",
    "The NPS Chat Corpus, which was demonstrated in 2.1, consists of over 10,000 posts from instant messaging sessions. These posts have all been labeled with one of 15 dialogue act types, such as \"Statement,\" \"Emotion,\" \"ynQuestion\", and \"Continuer.\" We can therefore use this data to build a classifier that can identify the dialogue act types for new instant messaging posts. The first step is to extract the basic messaging data. We will call xml_posts() to get a data structure representing the XML annotation for each post:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "posts = nltk.corpus.nps_chat.xml_posts()[:10000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define a simple feature extractor that checks what words the post contains:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dialogue_act_features(post):\n",
    "    features = {}\n",
    "    for word in nltk.word_tokenize(post):\n",
    "        features['contains(%s)' % word.lower()] = True\n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we construct the training and testing data by applying the feature extractor to each post (using post.get('class') to get a post's dialogue act type), and create a new classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.668\n"
     ]
    }
   ],
   "source": [
    "featuresets = [(dialogue_act_features(post.text), post.get('class'))\n",
    "               for post in posts]\n",
    "size = int(len(featuresets) * 0.1)\n",
    "train_set, test_set = featuresets[size:], featuresets[:size]\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)\n",
    "print (nltk.classify.accuracy(classifier, test_set))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recognizing Textual Entailment - I read this section. Return to if important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}