7-Information-Extraction/7-Information-Extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 7 - Information Extraction\n",
    "\n",
    "\n",
    "This week, we move from arbitrary textual classification to the use of computation and linguistic models to parse precise claims from documents. Rather than focusing on simply the *ideas* in a corpus, here we focus on understanding and extracting its precise *claims*. This process involves a sequential pipeline of classifying and structuring tokens from text, each of which generates potentially useful data for the content analyst. Steps in this process, which we examine in this notebook, include: 1) tagging words by their part of speech (POS) to reveal the linguistic role they play in the sentence (e.g., Verb, Noun, Adjective, etc.); 2) tagging words as named entities (NER) such as places or organizations; 3) structuring or \"parsing\" sentences into nested phrases that are local to, describe or depend on one another; and 4) extracting informational claims from those phrases, like the Subject-Verb-Object (SVO) triples we extract here. While much of this can be done directly in the python package NLTK that we introduced in week 2, here we use NLTK bindings to the Stanford NLP group's open software, written in Java. Try typing a sentence into the online version [here]('http://nlp.stanford.edu:8080/corenlp/') to get a sense of its potential. It is superior in performance to NLTK's implementations, but takes time to run, and so for these exercises we will parse and extract information for a very small text corpus. Of course, for final projects that draw on these tools, we encourage you to install the software on your own machines or shared servers at the university (RCC, SSRC) in order to perform these operations on much more text. \n",
    "\n",
    "For this notebook we will be using the following packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#All these packages need to be installed from pip\n",
    "#For NLP\n",
    "import nltk\n",
    "from nltk.tag import StanfordNERTagger\n",
    "from nltk.tag import StanfordPOSTagger\n",
    "from nltk.parse import stanford\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tree import Tree\n",
    "from nltk.draw.tree import TreeView\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "import numpy as np #For arrays\n",
    "import pandas #Gives us DataFrames\n",
    "import matplotlib.pyplot as plt #For graphics\n",
    "import seaborn #Makes the graphics look nicer\n",
    "\n",
    "#Displays the graphs\n",
    "import graphviz #You also need to install the command line graphviz\n",
    "\n",
    "#These are from the standard library\n",
    "import os.path\n",
    "import zipfile\n",
    "import subprocess\n",
    "import io\n",
    "import tempfile\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to use the [Stanford NLP group](http://nlp.stanford.edu/) programs with nltk on your own machine (you do *not* need to do this for this assignment), it will require a little bit of setup. We are basing these instructions on those provided by nltk, [here](https://github.com/nltk/nltk/wiki/Installing-Third-Party-Software#stanford-tagger-ner-tokenizer-and-parser), but with small changes to work with our notebooks. We also note that lower performance versions of many of the techniques demonstrated here are available natively within nltk (see the updated [nltk book](http://www.nltk.org/book/)).\n",
    "\n",
    "1. Install [Java 1.8+](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)\n",
    "    + Make sure your `JAVAPATH` is setup if you're on windows\n",
    "2. Download the following zip files from the Stanford NLP group, where DATE is the release date of the files, this will be the value of `stanfordVersion`\n",
    "    + [`stanford-corenlp-full-2016-10-31.zip`](https://stanfordnlp.github.io/CoreNLP/)\n",
    "    + [`stanford-postagger-full-DATE.zip`](http://nlp.stanford.edu/software/tagger.html#Download)\n",
    "    + [`stanford-ner-DATE.zip`](http://nlp.stanford.edu/software/CRF-NER.html#Download)\n",
    "    + [`stanford-parser-full-DATE.zip`](http://nlp.stanford.edu/software/lex-parser.html#Download)\n",
    "3. Unzip the files and place the resulting directories in the same location, this will become `stanfordDir`\n",
    "4. Lookup the version number used by the parser `stanford-parser-VERSION-models.jar` and set to to be `parserVersion`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This is the date at the end of each of the zip files, e.g.\n",
    "#the date in stanford-ner-2016-10-31.zip\n",
    "stanfordVersion = '2016-10-31'\n",
    "\n",
    "#This is the version numbers of the parser models, these\n",
    "#are files in `stanford-parser-full-2016-10-31.zip`, e.g.\n",
    "#stanford-parser-3.7.0-models.jar\n",
    "parserVersion = '3.7.0'\n",
    "\n",
    "#This is where the zip files were unzipped.Make sure to\n",
    "#unzip into directories named after the zip files\n",
    "#Don't just put all the files in `stanford-NLP`\n",
    "stanfordDir = '/mnt/efs/resources/shared/stanford-NLP'\n",
    "\n",
    "#Parser model, there are a few for english and a couple of other languages as well\n",
    "modelName = 'englishPCFG.ser.gz'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now will initialize all the tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up [NER tagger](http://www.nltk.org/api/nltk.tag.html?highlight=stanfordpostagger#nltk.tag.stanford.StanfordNERTagger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nerClassifierPath = os.path.join(stanfordDir,'stanford-ner-{}'.format(stanfordVersion), 'classifiers/english.all.3class.distsim.crf.ser.gz')\n",
    "\n",
    "nerJarPath = os.path.join(stanfordDir,'stanford-ner-{}'.format(stanfordVersion), 'stanford-ner.jar')\n",
    "\n",
    "nerTagger = StanfordNERTagger(nerClassifierPath, nerJarPath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up [POS Tagger](http://www.nltk.org/api/nltk.tag.html?highlight=stanfordpostagger#nltk.tag.stanford.StanfordPOSTagger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "postClassifierPath = os.path.join(stanfordDir, 'stanford-postagger-full-{}'.format(stanfordVersion), 'models/english-bidirectional-distsim.tagger')\n",
    "\n",
    "postJarPath = os.path.join(stanfordDir,'stanford-postagger-full-{}'.format(stanfordVersion), 'stanford-postagger.jar')\n",
    "\n",
    "postTagger = StanfordPOSTagger(postClassifierPath, postJarPath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up [Parser](http://www.nltk.org/api/nltk.parse.html?highlight=stanfordparser#module-nltk.parse.stanford)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "parserJarPath = os.path.join(stanfordDir, 'stanford-parser-full-{}'.format(stanfordVersion), 'stanford-parser.jar')\n",
    "\n",
    "parserModelsPath = os.path.join(stanfordDir, 'stanford-parser-full-{}'.format(stanfordVersion), 'stanford-parser-{}-models.jar'.format(parserVersion))\n",
    "\n",
    "modelPath = os.path.join(stanfordDir, 'stanford-parser-full-{}'.format(stanfordVersion), modelName)\n",
    "\n",
    "#The model files are stored in the jar, we need to extract them for nltk to use\n",
    "if not os.path.isfile(modelPath):\n",
    "    with zipfile.ZipFile(parserModelsPath) as zf:\n",
    "        with open(modelPath, 'wb') as f:\n",
    "            f.write(zf.read('edu/stanford/nlp/models/lexparser/{}'.format(modelName)))\n",
    "\n",
    "parser = stanford.StanfordParser(parserJarPath, parserModelsPath, modelPath)\n",
    "\n",
    "depParser = stanford.StanfordDependencyParser(parserJarPath, parserModelsPath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open Information Extraction is a module packaged within the Stanford Core NLP package, but it is not yet supported by `nltk`. As a result, we will be defining our own function that runs the Stanford Core NLP java code right here. For other projects, it is often useful to use Java or other programs (in C, C++) within a python workflow, and this is an example. `openIE()` takes in a string or list of strings and then produces as output all the subject, verb, object (SVO) triples Stanford Corenlp can find, as a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Watch out, this will very rarely raise an error since it trusts stanford-corenlp \n",
    "def openIE(target):\n",
    "    if isinstance(target, list):\n",
    "        target = '\\n'.join(target)\n",
    "    #setup the java targets\n",
    "    coreDir = '{}/stanford-corenlp-full-{}'.format(stanfordDir, stanfordVersion)\n",
    "    cp = '{0}/stanford-corenlp-{1}.jar:{0}/stanford-corenlp-{1}-models.jar:CoreNLP-to-HTML.xsl:slf4j-api.jar:slf4j-simple.jar'.format(coreDir, parserVersion)\n",
    "    with tempfile.NamedTemporaryFile(mode = 'w', delete = False) as f:\n",
    "        #Core nlp requires a files, so we will make a temp one to pass to it\n",
    "        #This file should be deleted by the OS soon after it has been used\n",
    "        f.write(target)\n",
    "        f.seek(0)\n",
    "        print(\"Starting OpenIE run\")\n",
    "        #If you know what these options do then you should mess with them on your own machine and not the shared server\n",
    "        sp = subprocess.run(['java', '-mx2g', '-cp', cp, 'edu.stanford.nlp.naturalli.OpenIE', '-threads', '1', f.name], stdout = subprocess.PIPE, stderr = subprocess.PIPE)\n",
    "        #Live stderr is non-trivial so this is the best we can do\n",
    "        print(sp.stderr.decode('utf-8'))\n",
    "        retSting = sp.stdout.decode('utf-8')\n",
    "    #Making the DataFrame, again having to pass a fake file, yay POSIX I guess\n",
    "    with io.StringIO(retSting) as f:\n",
    "        df = pandas.read_csv(f, delimiter = '\\t', names =['certainty', 'subject', 'verb', 'object'])\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will illustrate these tools on some *very* short examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text = ['I saw the elephant in my pajamas.', 'The quick brown fox jumped over the lazy dog.', 'While in France, Christine Lagarde discussed short-term stimulus efforts in a recent interview with the Wall Street Journal.', 'Trayvon Benjamin Martin was an African American from Miami Gardens, Florida, who, at 17 years old, was fatally shot by George Zimmerman, a neighborhood watch volunteer, in Sanford, Florida.', 'Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo']\n",
    "tokenized_text = [word_tokenize(t) for t in text]\n",
    "print('\\n'.join(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part-of-Speech (POS) tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In POS tagging, we classify each word by its semantic role in a sentence. The Stanford POS tagger uses the [Penn Treebank tag set]('http://repository.upenn.edu/cgi/viewcontent.cgi?article=1603&context=cis_reports') to POS tag words from input sentences. As discussed in the second assignment, this is a relatively precise tagset, which allows more informative tags, and also more opportunities to err :-).\n",
    "\n",
    "|#. |Tag |Description |\n",
    "|---|----|------------|\n",
    "|1.\t|CC\t|Coordinating conjunction\n",
    "|2.\t|CD\t|Cardinal number\n",
    "|3.\t|DT\t|Determiner\n",
    "|4.\t|EX\t|Existential there\n",
    "|5.\t|FW\t|Foreign word\n",
    "|6.\t|IN\t|Preposition or subordinating conjunction\n",
    "|7.\t|JJ\t|Adjective\n",
    "|8.\t|JJR|\tAdjective, comparative\n",
    "|9.\t|JJS|\tAdjective, superlative\n",
    "|10.|\tLS\t|List item marker\n",
    "|11.|\tMD\t|Modal\n",
    "|12.|\tNN\t|Noun, singular or mass\n",
    "|13.|\tNNS\t|Noun, plural\n",
    "|14.|\tNNP\t|Proper noun, singular\n",
    "|15.|\tNNPS|\tProper noun, plural\n",
    "|16.|\tPDT\t|Predeterminer\n",
    "|17.|\tPOS\t|Possessive ending\n",
    "|18.|\tPRP\t|Personal pronoun\n",
    "|19.|\tPRP\\$|\tPossessive pronoun\n",
    "|20.|\tRB\t|Adverb\n",
    "|21.|\tRBR\t|Adverb, comparative\n",
    "|22.|\tRBS\t|Adverb, superlative\n",
    "|23.|\tRP\t|Particle\n",
    "|24.|\tSYM\t|Symbol\n",
    "|25.|\tTO\t|to\n",
    "|26.|\tUH\t|Interjection\n",
    "|27.|\tVB\t|Verb, base form\n",
    "|28.|\tVBD\t|Verb, past tense\n",
    "|29.|\tVBG\t|Verb, gerund or present participle\n",
    "|30.|\tVBN\t|Verb, past participle\n",
    "|31.|\tVBP\t|Verb, non-3rd person singular present\n",
    "|32.|\tVBZ\t|Verb, 3rd person singular present\n",
    "|33.|\tWDT\t|Wh-determiner\n",
    "|34.|\tWP\t|Wh-pronoun\n",
    "|35.|\tWP$\t|Possessive wh-pronoun\n",
    "|36.|\tWRB\t|Wh-adverb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pos_sents = postTagger.tag_sents(tokenized_text)\n",
    "print(pos_sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks quite good. Now we will try POS tagging with a somewhat larger corpus. We consider a few of the top posts from the reddit data we used last week."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "redditDF = pandas.read_csv('data/reddit.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grabbing the 10 highest scoring posts and tokenizing the sentences. Once again, notice that we aren't going to do any kind of stemming this week (although *semantic* normalization may be performed where we translate synonyms into the same focal word)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "redditTopScores = redditDF.sort_values('score')[-10:]\n",
    "redditTopScores['sentences'] = redditTopScores['text'].apply(lambda x: [nltk.word_tokenize(s) for s in nltk.sent_tokenize(x)])\n",
    "redditTopScores.index = range(len(redditTopScores) - 1, -1,-1) #Reindex to make things nice in the future\n",
    "redditTopScores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "redditTopScores['POS_sents'] = redditTopScores['sentences'].apply(lambda x: postTagger.tag_sents(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "redditTopScores['POS_sents']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And count the number of `NN` (nouns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "countTarget = 'NN'\n",
    "targetCounts = {}\n",
    "for entry in redditTopScores['POS_sents']:\n",
    "    for sentence in entry:\n",
    "        for ent, kind in sentence:\n",
    "            if kind != countTarget:\n",
    "                continue\n",
    "            elif ent in targetCounts:\n",
    "                targetCounts[ent] += 1\n",
    "            else:\n",
    "                targetCounts[ent] = 1\n",
    "sortedTargets = sorted(targetCounts.items(), key = lambda x: x[1], reverse = True)\n",
    "sortedTargets[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the number of top verbs (`VB`)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "countTarget = 'VB'\n",
    "targetCounts = {}\n",
    "for entry in redditTopScores['POS_sents']:\n",
    "    for sentence in entry:\n",
    "        for ent, kind in sentence:\n",
    "            if kind != countTarget:\n",
    "                continue\n",
    "            elif ent in targetCounts:\n",
    "                targetCounts[ent] += 1\n",
    "            else:\n",
    "                targetCounts[ent] = 1\n",
    "sortedTargets = sorted(targetCounts.items(), key = lambda x: x[1], reverse = True)\n",
    "sortedTargets[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the adjectives that modify the word, \"computer\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "NTarget = 'JJ'\n",
    "Word = 'computer'\n",
    "NResults = set()\n",
    "for entry in redditTopScores['POS_sents']:\n",
    "    for sentence in entry:\n",
    "        for (ent1, kind1),(ent2,kind2) in zip(sentence[:-1], sentence[1:]):\n",
    "            if (kind1,ent2.lower())==(NTarget,Word):\n",
    "                NResults.add(ent1)\n",
    "            else:\n",
    "                continue\n",
    "\n",
    "print(NResults)     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating POS tagger\n",
    "\n",
    "We can check the POS tagger by running it on a manually tagged corpus and identifying a reasonable error metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "treeBank = nltk.corpus.treebank\n",
    "treeBank.tagged_sents()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "treeBank.sents()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "stanfordTags = postTagger.tag_sents(treeBank.sents()[:30])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And compare the two"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "NumDiffs = 0\n",
    "for sentIndex in range(len(stanfordTags)):\n",
    "    for wordIndex in range(len(stanfordTags[sentIndex])):\n",
    "        if stanfordTags[sentIndex][wordIndex][1] != treeBank.tagged_sents()[sentIndex][wordIndex][1]:\n",
    "            if treeBank.tagged_sents()[sentIndex][wordIndex][1] != '-NONE-':\n",
    "                print(\"Word: {}  \\tStanford: {}\\tTreebank: {}\".format(stanfordTags[sentIndex][wordIndex][0], stanfordTags[sentIndex][wordIndex][1], treeBank.tagged_sents()[sentIndex][wordIndex][1]))\n",
    "                NumDiffs += 1\n",
    "total = sum([len(s) for s in stanfordTags])\n",
    "print(\"The Precision is {:.3f}%\".format((total-NumDiffs)/total * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can see that the stanford POS tagger is quite good. Nevertheless, for a 20 word sentence, we only have a 66% chance ($1-.96^{20}$) of tagging (and later parsing) it correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your turn*</span>\n",
    "\n",
    "<span style=\"color:red\">In the cells immediately following, perform POS tagging on a meaningful (but modest) subset of a corpus associated with your final project. Examine the list of words associated with at least three different parts of speech. Consider conditional associations (e.g., adjectives associated with nouns or adverbs with verbs of interest). What do these distributions suggest about your corpus?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named-Entity Recognition\n",
    "\n",
    "Named Entity Recognition (NER) is also a classification task, which identifies named objects. Included with Stanford NER are a 4 class model trained on the CoNLL 2003 eng.train, a 7 class model trained on the MUC 6 and MUC 7 training data sets, and a 3 class model trained on both data sets plus some additional data (including ACE 2002 and limited data in-house) on the intersection of those class sets. \n",
    "\n",
    "**3 class**:\tLocation, Person, Organization\n",
    "\n",
    "**4 class**:\tLocation, Person, Organization, Misc\n",
    "\n",
    "**7 class**:\tLocation, Person, Organization, Money, Percent, Date, Time\n",
    "\n",
    "These models each use distributional similarity features, which provide some performance gain at the cost of increasing their size and runtime. Also available are the same models missing those features.\n",
    "\n",
    "(We note that the training data for the 3 class model does not include any material from the CoNLL eng.testa or eng.testb data sets, nor any of the MUC 6 or 7 test or devtest datasets, nor Alan Ritter's Twitter NER data, so all of these would be valid tests of its performance.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we tag our first set of exemplary sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "classified_sents = nerTagger.tag_sents(tokenized_text)\n",
    "print(classified_sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also run NER over our entire corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "redditTopScores['classified_sents'] = redditTopScores['sentences'].apply(lambda x: nerTagger.tag_sents(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "redditTopScores['classified_sents']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the most common entities (which are, of course, boring):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "entityCounts = {}\n",
    "for entry in redditTopScores['classified_sents']:\n",
    "    for sentence in entry:\n",
    "        for ent, kind in sentence:\n",
    "            if ent in entityCounts:\n",
    "                entityCounts[ent] += 1\n",
    "            else:\n",
    "                entityCounts[ent] = 1\n",
    "sortedEntities = sorted(entityCounts.items(), key = lambda x: x[1], reverse = True)\n",
    "sortedEntities[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or those occurring only twice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[x[0] for x in sortedEntities if x[1] == 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also list the most common \"non-objects\". (We note that we're not graphing these because there are so few here.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nonObjCounts = {}\n",
    "for entry in redditTopScores['classified_sents']:\n",
    "    for sentence in entry:\n",
    "        for ent, kind in sentence:\n",
    "            if kind == 'O':\n",
    "                continue\n",
    "            elif ent in nonObjCounts:\n",
    "                nonObjCounts[ent] += 1\n",
    "            else:\n",
    "                nonObjCounts[ent] = 1\n",
    "sortedNonObj = sorted(nonObjCounts.items(), key = lambda x: x[1], reverse = True)\n",
    "sortedNonObj[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the Organizations?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "OrgCounts = {}\n",
    "for entry in redditTopScores['classified_sents']:\n",
    "    for sentence in entry:\n",
    "        for ent, kind in sentence:\n",
    "            if kind != 'ORGANIZATION':\n",
    "                continue\n",
    "            elif ent in OrgCounts:\n",
    "                OrgCounts[ent] += 1\n",
    "            else:\n",
    "                OrgCounts[ent] = 1\n",
    "sortedOrgs = sorted(OrgCounts.items(), key = lambda x: x[1], reverse = True)\n",
    "sortedOrgs[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These, of course, have much smaller counts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your turn*</span>\n",
    "\n",
    "<span style=\"color:red\">In the cells immediately following, perform NER on a (modest) subset of your corpus of interest. List all of the different kinds of entities tagged? What does their distribution suggest about the focus of your corpus? For a subset of your corpus, tally at least one type of named entity and calculate the Precision, Recall and F-score for the NER classification just performed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing\n",
    "\n",
    "Here we will introduce the Stanford Parser by feeding it tokenized text from our initial example sentences. The parser is a dependency parser, but this initial program outputs a simple, self-explanatory phrase-structure representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "parses = list(parser.parse_sents(tokenized_text)) #Converting the iterator to a list so we can call by index. They are still \n",
    "fourthSentParseTree = list(parses[3]) #iterators so be careful about re-running code, without re-running this block\n",
    "print(fourthSentParseTree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trees are a common data structure and there are a large number of things to do with them. What we are intetered in is the relationship between different types of speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def treeRelation(parsetree, relationType, *targets):\n",
    "    if isinstance(parsetree, list):\n",
    "        parsetree = parsetree[0]\n",
    "    if set(targets) & set(parsetree.leaves()) != set(targets):\n",
    "        return []\n",
    "    else:\n",
    "        retList = []\n",
    "        for subT in parsetree.subtrees():\n",
    "            if subT.label() == relationType:\n",
    "                if set(targets) & set(subT.leaves()) == set(targets):\n",
    "                    retList.append([(subT.label(), ' '.join(subT.leaves()))])\n",
    "    return retList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def treeSubRelation(parsetree, relationTypeScope, relationTypeTarget, *targets):\n",
    "    if isinstance(parsetree, list):\n",
    "        parsetree = parsetree[0]\n",
    "    if set(targets) & set(parsetree.leaves()) != set(targets):\n",
    "        return []\n",
    "    else:\n",
    "        retSet = set()\n",
    "        for subT in parsetree.subtrees():\n",
    "            if set(targets) & set(subT.leaves()) == set(targets):\n",
    "                if subT.label() == relationTypeScope:\n",
    "                    for subsub in subT.subtrees():\n",
    "                        if subsub.label()==relationTypeTarget:\n",
    "                            retSet.add(' '.join(subsub.leaves()))\n",
    "    return retSet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "treeRelation(fourthSentParseTree, 'NP', 'Florida', 'who')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that Florida occurs twice in two different nested noun phrases in the sentence. \n",
    "\n",
    "We can also find all of the verbs within the noun phrase defined by one or more target words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "treeSubRelation(fourthSentParseTree, 'NP', 'VBN', 'Florida', 'who')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or if we want to to look at the whole tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fourthSentParseTree[0].pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Or another sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "list(parses[1])[0].pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Dependency parsing and graph representations\n",
    "\n",
    "Dependency parsing was developed to robustly capture linguistic dependencies from text. The complex tags associated with these parses are detailed [here]('http://universaldependencies.org/u/overview/syntax.html'). When parsing with the dependency parser, we will work directly from the untokenized text. Note that no *processing* takes place before parsing sentences--we do not remove so-called stop words or anything that plays a syntactic role in the sentence, although anaphora resolution and related normalization may be performed before or after parsing to enhance the value of information extraction. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "depParses = list(depParser.raw_parse_sents(text)) #Converting the iterator to a list so we can call by index. They are still \n",
    "secondSentDepParseTree = list(depParses[1])[0] #iterators so be careful about re-running code, without re-running this block\n",
    "print(secondSentDepParseTree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a graph and we can convert it to a dot file and use that to visulize it. Try traversing the tree and extracting elements that are nearby one another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "secondSentGraph = graphviz.Source(secondSentDepParseTree.to_dot())\n",
    "secondSentGraph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or another sentence "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "graphviz.Source(list(depParses[3])[0].to_dot())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can also do a dependency parse on the reddit sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "topPostDepParse = list(depParser.parse_sents(redditTopScores['sentences'][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This takes a few seconds, but now lets look at the parse tree from one of the processed sentences.\n",
    "\n",
    "The sentence is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "targetSentence = 7\n",
    "print(' '.join(redditTopScores['sentences'][0][targetSentence]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which leads to a very rich dependancy tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "graphviz.Source(list(topPostDepParse[targetSentence])[0].to_dot())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your turn*</span>\n",
    "\n",
    "<span style=\"color:red\">In the cells immediately following, parse a (modest) subset of your corpus of interest. How deep are the phrase structure and dependency parse trees nested? How does parse depth relate to perceived sentence complexity? What are five things you can extract from these parses for subsequent analysis? (e.g., nouns collocated in a noun phrase; adjectives that modify a noun; etc.) Capture these sets of things for a focal set of words (e.g., \"Bush\", \"Obama\", \"Trump\"). What do they reveal about the roles that these entities are perceive to play in the social world inscribed by your texts?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Information extraction\n",
    "\n",
    "Information extraction approaches typically (as here, with Stanford's Open IE engine) ride atop the dependency parse of a sentence. They are a pre-coded example of the type analyzed in the prior. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF = openIE(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`openIE()` prints everything stanford core produces and we can see from looking at it that initializing the dependency parser takes most of the time, so calling the function will always take at least 12 seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No buffalos (because there were no verbs), but the rest is somewhat promising. Note, however, that it abandoned the key theme of the sentence about the tragic Trayvon Martin death (\"fatally shot\"), likely because it was buried so deeply within the complex phrase structure. This is obviously a challenge. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">*Your thoughts*</span>\n",
    "\n",
    "<span style=\"color:red\">How would you extract relevant information about the Trayvon Martin sentence directly from the dependency parse (above)? Code an example here. (For instance, what compound nouns show up with what verb phrases within the sentence?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can also look for subject, object, target triples in one of the reddit stories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF = openIE(redditTopScores['text'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's almost 200 triples in only:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(redditTopScores['sentences'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sentences and"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum([len(s) for s in redditTopScores['sentences'][0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets find at the most common subject in this story."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF['subject'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I is followed by various male pronouns and compound nouns (e.g., \"old man\"). 'I' occures most often with the following verbs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF[ieDF['subject'] == 'I']['verb'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the following objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ieDF[ieDF['subject'] == 'I']['object'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## <span style=\"color:red\">*Your Turn*</span>\n",
    "\n",
    "<span style=\"color:red\">In the cells immediately following, perform open information extraction on a modest subset of texts relevant to your final project. Analyze the relative attachment of several subjects relative to verbs and objects and visa versa? Describe how you would select among these statements to create a database of high-value statements for your project and do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}