nltk-fraser-talk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<img style=\"float:left\" src=\"http://ipython.org/_static/IPy_header.png\" />\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with a corpus of Malcolm Fraser's speeches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function, division #just some libraries and packages to get us started.\n",
    "import sys\n",
    "from nltk.book import * #some example data that comes with the nltk book\n",
    "from IPython.display import display, clear_output\n",
    "sys.path.append(\"/usr/lib/python2.7/site-packages/\")\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hi everyone!** \n",
    "Welcome to Textual Analysis with the Natural Language Toolkit (NLTK)\n",
    "\n",
    "NLTK is a library of the computer programming language Python for textual analysis.\n",
    "\n",
    "It was written by computational linguistics and is a very powerful tool.\n",
    "\n",
    "Today's focus will be on **showcasing some Natural Language processing skills** with NLTK and using these skills to **investigate the Fraser Speeches Corpus**. \n",
    "\n",
    "But before we get onto that, I'll discuss an example of text mining in academic librarianship research\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Text Mining Analysis of Academic Libraries' Tweets\n",
    "_**By Sultan M. Al-Daihani, Alan Abrahams**_ \n",
    "\n",
    "This study applied a text mining approach to a dataset of tweets by ten academic libraries.\n",
    "\n",
    "They used an interface powered by nltk. \n",
    "\n",
    "The results show variance between academic libraries in distribution of tweets over time. The\n",
    "most frequent word was “open,” which was used in a variety of contexts by the academic libraries. It was\n",
    "noted that the most frequent bi-gram (two-word sequence) in the aggregated tweets was “special collections”.\n",
    "The most frequent tri-gram (three-word sequence) was “save the date”. The most frequent word categories in\n",
    "the semantic analysis for most libraries were related to “knowledge, insight, and information concerning personal\n",
    "and cultural relations”\n",
    "\n",
    "Their findings highlight the importance of using data- and text-mining approaches in understanding the aggregate social data of academic libraries to aid in decision-making and strategic planning for patron outreach and marketing of services.\n",
    "\n",
    "*Any questions or anything before we dive in?*\n",
    "\n",
    "*OK! Just a note. I'll be moving quickly today, but repeating code, which you wouldn't normally do, for didatic purposes...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Malcolm Fraser and his speeches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we are going to be working with a corpus of speeches made by Malcolm Fraser. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this code allows us to display images and webpages in our notebook\n",
    "from IPython.display import display\n",
    "from IPython.display import display_pretty, display_html, display_jpeg, display_png, display_svg\n",
    "from IPython.display import Image\n",
    "from IPython.display import HTML\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because our project here is *corpus driven*, we don't necessarily need to know about Malcolm Fraser and his speeches in order to analyse the data: we may be happy to let things emerge from the data themselves. Even so, it's good to keep remind ourselves of some of his political history.\n",
    "\n",
    "Malcolm Fraser was a member of Australian parliament between 1955 and 1983, holding the seat of Wannon in western Victoria. He held a number of ministries, including Education and Science, and Defence. \n",
    "\n",
    "He became leader of the Liberal Party in March 1975 and Prime Minister of Australia in December 1975, following the dismissal of the Whitlam government in November 1975.\n",
    "\n",
    "He retired from parliament following the defeat of the Liberal Party at the 1983 election and in 2009 resigned from the Liberal Party after becoming increasingly critical of some of its policies. He died on 20 March, 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "HTML('<iframe src=http://en.wikipedia.org/wiki/Malcolm_Fraser width=950 height=350></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In 2004, Malcolm Fraser made the University of Melbourne the official custodian of his personal papers. The collection consists of a large number of photographs, speeches and personal papers, including Neville Fraser's WWI diaries and materials relating to CARE Australia, which Mr Fraser helped to found in 1987. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "HTML('<iframe src=http://www.unimelb.edu.au/malcolmfraser/ width=950 height=350></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Between 1954 until 1983, Malcolm Fraser regularly made a talk to his electorate that was broadcast on Sunday evening on local radio.  \n",
    "\n",
    "The speeches were transcribed years ago. Optical Character Recognition (OCR) was used to digitise the transcripts. This means that the texts are not of perfect quality. \n",
    "\n",
    "Some have been manually corrected, which has removed extraneous characters and mangled words, but even so there are still some quirks in the formatting. \n",
    "\n",
    "For much of this session, we are going to manipulate the corpus data, and use the data to restructure the corpus. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common part of corpus building is corpus cleaning. Reasons for cleaning include:\n",
    "\n",
    "1. Not break the code with unexpected input\n",
    "2. Ensure that searches match as many examples as possible\n",
    "3. Increasing readability, the accuracy of taggers, stemmers, parsers, etc.\n",
    "\n",
    "The level and kind of cleaning depends on your data and the aims of your project. In the case of very clean data (lucky you!), there may be little that needs to be done. With messy data, you may need to go as far as to correct variant spellings (online conversation, very old books)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*What are the characteristics of clean and messy data? Any personal experiences? Discuss with your neighbours.* \n",
    "\n",
    "It will be important to bear these characteristics in mind once you start work with or building your own datasets and corpora. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, let's load in our text.\n",
    "\n",
    "Download the txt files available in this url: http://archives.unimelb.edu.au/malcolmfraser/explore/radiotalks\n",
    "\n",
    "Via file management, upload the zip folder we downloaded from the University of Melbourne website into the same folder that you are running the notebook in.\n",
    "\n",
    "We can also look at file contents within the Jupyter Notebook itself.\n",
    "\n",
    "Importing os allows us to open, read, write and do other this with folders and files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we're going to import some functions that will help us tokenize text. Tokenisation is an import concept in natural language processing. It tells the computer to count and do things with words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# import tokenizers\n",
    "from nltk import word_tokenize\n",
    "from nltk.text import Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to manually unzip the files in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import zipfile\n",
    "zip_ref = zipfile.ZipFile('UMA_Fraser_Radio_Talks.zip', 'r')\n",
    "zip_ref.extractall('/Users/resplat/VALA-FraserCorpus')\n",
    "zip_ref.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we make a 'list' of files in the directory 'UMA_Fraser_Radio_Talks'. A list is a data structure in Python. It essentially turns our files into a list that we can easily access. Below, we've asked for the first three files in that list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# make a list of files in the directory 'UMA_Fraser_Radio_Talks'\n",
    "files = os.listdir('UMA_Fraser_Radio_Talks')\n",
    "print(files[0:3]) #note that Python starts counting at 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, since we'll be referring to this path quite a bit, let's make it into a variable. This makes our code easier to use on other projects (and saves typing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "corpus_path = 'UMA_Fraser_Radio_Talks'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now tell Python to get the contents of a file in the file list and print it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print(file contents)\n",
    "#change zero to something else to print(a different file)\n",
    "# \"r\" tells python that it has the ability to read the files\n",
    "files = open(os.path.join(corpus_path, files[1]), \"r\", encoding=\"latin\")\n",
    "text = files.read()\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring further: splitting up text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've had a look at one file, but the real strength of NLTK is to be able to explore large bodies of text. \n",
    "\n",
    "When we manually inspected the first file, we saw that it contained a metadata section, before the body of the text. \n",
    "\n",
    "We can ask Python to show us just the start of the file. For analysing the text, it is useful to split the metadata section off, so that we can interrogate it separately but also so that it won't distort our results when we analyse the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# split the file we read in above into two parts\n",
    "# split on the characters <!--end metadata-->\n",
    "data = text.split(\"<!--end metadata-->\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# view the first part. Note that Python starts counting from 0\n",
    "print(data[0])\n",
    "# if you want to view the second half of the split i.e. the data, not metadata, enter data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# split into lines, add '*' to the start of each line\n",
    "# \\n is a newline character\n",
    "for line in data[0].split('\\n'):\n",
    "    print('*', line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# skip empty lines and any line that starts with '<'\n",
    "for line in data[0].split('\\n'):\n",
    "    if not line:\n",
    "        continue\n",
    "    if line[0] == '<':\n",
    "        continue\n",
    "    print('*', line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# split the metadata items on ':' so that we can interrogate each one\n",
    "for line in data[0].split('\\n'):\n",
    "    if not line:\n",
    "        continue\n",
    "    if line[0] == '<':\n",
    "        continue\n",
    "    element = line.split(':')\n",
    "    print('*', element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the problem here?? ^^^^ Look at the 'Collection URI'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# actually, only split on the first colon\n",
    "for line in data[0].split('\\n'):\n",
    "    if not line:\n",
    "        continue\n",
    "    if line[0] == '<':\n",
    "        continue\n",
    "    element = line.split(':', 1)\n",
    "    print('*', element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Challenge**: Building a Dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've already worked with strings (a 'string' of characters) and lists (lists of words, numbers etc...). Another kind of data structure in Python is a *dictionary*.\n",
    "\n",
    "Here is how a simple dictionary works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a dictionary\n",
    "commonwords = {'the': 4023, 'of': 3809, 'a': 3098}\n",
    "# search the dictionary for 'of'\n",
    "commonwords['of']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(commonwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(text) #the files we read into python and printed to the screen as a big block of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(data) #the variable we used to split the text into metadata and data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The point of dictionaries is to store a *key* (the word) and a *value* (the count). When you ask for the key, you get its value. So it's has different properties to a list where you just get an index of the item's place in a list.\n",
    "\n",
    "Notice that you use curly braces for dictionaries, but square brackets for lists.\n",
    "\n",
    "Dictionaries are a great way to work with the metadata in our corpus. Let's build a dictionary called *metadata*:\n",
    "\n",
    "Your first line will look like this:\n",
    "\n",
    "      metadata = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metadata = {} # create an empty dictionary to populate\n",
    "for line in data[0].split('\\n'):\n",
    "    if not line:\n",
    "        continue\n",
    "    if line[0] == '<':\n",
    "        continue\n",
    "    element = line.split(':', 1) # This code is the same as earlier\n",
    "    metadata[element[0]] = element[-1] #This is new! It associates the metadata category with the value next to it\n",
    "print(metadata) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# now are metadata categories are searchable\n",
    "# look up the Date\n",
    "print(metadata['Date']) # you can change this to 'Title', etc..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenge**: define a function that creates a dictionary of the metadata for each file and gets rid of the whitespace at the start of each element\n",
    "\n",
    "**Hint**: to get rid of the whitespace use the *.strip()* command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# open the first file, read it and then split it into two parts, metadata and body\n",
    "data = open(os.path.join(corpus_path, 'UDS2013680-1-full.txt'), 'r', encoding='latin')\n",
    "data = data.read().split(\"<!--end metadata-->\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def parse_metadata(text): # syntax to create our own reuseable code, called functions\n",
    "    metadata = {}\n",
    "    for line in text.split('\\n'):\n",
    "        if not line:\n",
    "            continue\n",
    "        if line[0] == '<':\n",
    "            continue\n",
    "        element = line.split(':', 1)\n",
    "        metadata[element[0]] = element[-1].strip(' ') #note this is also exactly the same as our first dictionary\n",
    "    return metadata "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test it out! Essentially, we've put save all our code into the parse_metadata function, so we can run all that code after the first colon without having to type it all out everytime!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "parse_metadata(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we're confident that the function works, let's find out a bit about the corpus.\n",
    "As a start, it would be useful to know which years the texts are from. Are they evenly distributed over time? A graph will tell us!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#import conditional frequency distribution\n",
    "from nltk.probability import ConditionalFreqDist\n",
    "import matplotlib\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cfdist = ConditionalFreqDist()\n",
    "for filename in os.listdir(corpus_path):\n",
    "    text = open(os.path.join(corpus_path, filename), 'r', encoding='latin').read()\n",
    "    #split text of file on 'end metadata'\n",
    "    text = text.split(\"<!--end metadata-->\")\n",
    "    #parse metadata using previously defined function \"parse_metadata\"\n",
    "    metadata = parse_metadata(text[0])\n",
    "    #skip all speeches for which there is no exact date\n",
    "    if metadata['Date'][0] == 'c': # if you change [0] to [1] you get double years\n",
    "        continue\n",
    "    #build a frequency distribution graph by year, that is, take the final bit of the 'Date' string after '/'\n",
    "    cfdist['count'][metadata['Date'].split('/')[-1]] += 1\n",
    "cfdist.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's build another graph, but this time by the 'Description' field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cfdist2 = ConditionalFreqDist()\n",
    "for filename in os.listdir(corpus_path):\n",
    "    text = open(os.path.join(corpus_path, filename), 'r', encoding='latin').read()\n",
    "    text = text.split(\"<!--end metadata-->\")\n",
    "    metadata = parse_metadata(text[0])\n",
    "    if metadata['Date'][0] == 'c':\n",
    "        continue\n",
    "    cfdist2['count'][metadata['Description']] += 1\n",
    "cfdist2.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've got messy data! What's the lesson here?\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus challenge**: Build a frequency distribution graph that includes speeches without an exact date.\n",
    "Hint: you'll need to tell Python to ignore the 'c' and just take the digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cfdist3 = ConditionalFreqDist()\n",
    "for filename in os.listdir(corpus_path):\n",
    "    text = open(os.path.join(corpus_path, filename), 'r', encoding='latin').read()\n",
    "    text = text.split(\"<!--end metadata-->\")\n",
    "    metadata = parse_metadata(text[0])\n",
    "    date = metadata['Date']\n",
    "    if date[0] == 'c':\n",
    "        year = date[1:]\n",
    "    elif date[0] != 'c':\n",
    "        year = date.split('/')[-1]\n",
    "    cfdist3['count'][year] += 1\n",
    "cfdist3.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ordering our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, we looked at features of language in whole books. The way in which you organise your data will affect the ways in which you can interrogate it. Because our data samples span a long stretch of time, it might be interesting to investigate the ways in which Malcolm Fraser's language changes over time. \n",
    "\n",
    "In order to study this, it is helpful to structure our data according to the year of the sample. This simply means creating folders for each sample year, and moving the text of each speech into the correct one.\n",
    "\n",
    "We can use our metadata parser to help with this task. Then, after structuring our corpus by date, we want the metadata gone, so that when we count language features in the files, we are not also counting the metadata.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Side note - Regular expressions\n",
    "Regular expressions are a powerful means of searching for patterns in data. In this case, we're going to construct a regular expression to find the year of each speech. There are various ways in which we could write this expression, depending on how confident we are that the data is clean and the range of dates we're interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import os\n",
    "# a path to our soonwordso-be organised corpus\n",
    "newpath = '../corpora/fraser-year'\n",
    "os.makedirs(newpath)\n",
    "files = os.listdir(corpus_path)\n",
    "# define a regex to match year portion of date\n",
    "yearfinder = re.compile('19[0-9]{2}')\n",
    "for filename in files:\n",
    "    # split file contents at end of metadata\n",
    "    text = open(os.path.join(corpus_path, filename), 'r', encoding='latin')\n",
    "    data = text.read().split(\"<!--end metadata-->\")\n",
    "    # get date from data[0]\n",
    "    # use our metadata parser to get metadata\n",
    "    metadata = parse_metadata(data[0])\n",
    "    #look up date field of dict entry\n",
    "    date = metadata.get('Date')\n",
    "    # search date for year\n",
    "    yearmatch = re.search(yearfinder, str(date))\n",
    "    #get the year as a string\n",
    "    year = str(yearmatch.group())\n",
    "    # make a directory with the year name\n",
    "    if not os.path.exists(os.path.join(newpath, year)):\n",
    "        os.makedirs(os.path.join(newpath, year))\n",
    "    # make a new file with the same name as the old one in the new dir\n",
    "    fo = open(os.path.join(newpath, year, filename),\"w\")\n",
    "    # write the content portion, without metadata\n",
    "    fo.write(data[1])\n",
    "    fo.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did it work? How can we check?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(os.listdir(newpath))\n",
    "print(os.listdir(newpath + '/1981'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using NLTK to analyse the Fraser Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The books were were working with yesterday had already had some processing done on them so that we could use NLTK to find features of the language. Remember that Python regards a text file as a single long string of characters. The first thing to do is to start breaking the text up into sentences and words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk import word_tokenize\n",
    "speech = open('../corpora/fraser-year/1975/UDS2013680-678-full.txt', \"r\").read() \n",
    "tokens = word_tokenize(speech)\n",
    "print(tokens[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Breaking a speech into tokens lets us do the sort of word counting that we were doing yesterday on the speeches. We can do some more interesting linguistic analysis if we use Part of Speech tagging. NLTK has a number of different Part of Speech tags that we could use, but the simplest one is called 'Universal', and we'll use that here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sentence = \"They refuse to permit us the refuse permit\"\n",
    "words = word_tokenize(sentence)\n",
    "tagged = nltk.pos_tag(words, tagset='universal')\n",
    "print(tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part of Speech tagging creates bigrams, that is, it associates the word with its tag in a pair of items that we can see above in brackets.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tag_fd = nltk.FreqDist(tag for (word, tag) in tagged)\n",
    "tag_fd.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge!\n",
    "Use Part of Speech tagging to tag the speech that we have just tokenised the do the following:\n",
    "* Find the most common parts of speech\n",
    "* Find the most common verbs and create a frequency Distribution graph of your result\n",
    "* Find the 10 most common nouns in the speech\n",
    "\n",
    "*Hint: to find the most common verbs and nouns, you will need to create a list that contains only the verbs or only the nouns from the speech. Use a for loop to create your list. Then create a frequency distribution*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tagged_speech = nltk.pos_tag(tokens, tagset = 'universal')\n",
    "speech_fd = nltk.FreqDist(tag for (word, tag) in tagged_speech)\n",
    "speech_fd.most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "verblist = []\n",
    "for (word, tag) in tagged_speech:\n",
    "    if tag == 'VERB':\n",
    "        verblist.append(word)\n",
    "# Check the length of the list of verbs. \n",
    "#If it matches the number of verbs above, you can be fairly sure your loop has worked as expected\n",
    "print(len(verblist))\n",
    "verb_fd = nltk.FreqDist(verblist)\n",
    "print(verb_fd.most_common()[:10])\n",
    "verb_fd.plot(cumulative = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nounlist = []\n",
    "for (word, tag) in tagged_speech:\n",
    "    if tag == 'NOUN':\n",
    "        nounlist.append(word)\n",
    "print(nounlist[:10])\n",
    "print(len(nounlist))\n",
    "noun_fd = nltk.FreqDist(nounlist)\n",
    "print(noun_fd.most_common()[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extension**\n",
    "There are a few things to note about this result - Prime and Minister have been returned as two different, equally frequent nouns. Because we're humans, not computers, we know it's likely that what we're actually seeing is 'Prime Minister'. It's also unlikely that 'North' 'South' occur alone - perhaps Mr Fraser was talking baout North and South Vietnam? We could test for bigrams (words that typically occur side by side) to see if this is the case. In order to perform this test, we must first convert our list of tokens into and NLTK text. We can then use specific NLTK functions on the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(type(tokens))\n",
    "speech_text = nltk.Text(tokens)\n",
    "print(type(speech_text))\n",
    "speech_text.collocations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "speech_text.concordance(\"wool\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some linguistics..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Functional linguistics* is a research area concerned with how *realised language* (lexis and grammar) work to achieve meaningful social functions.\n",
    "\n",
    "One functional linguistic theory is *Systemic Functional Linguistics*, developed by Michael Halliday (Prof. Emeritus at University of Sydney).\n",
    "\n",
    "Central to the theory is a division between **experiential meanings** and **interpersonal meanings**.\n",
    "\n",
    "* Experiential meanings communicate what happened to whom, under what circumstances.\n",
    "* Interpersonal meanings negotiate identities and role relationships between speakers \n",
    "\n",
    "Halliday argues that these two kinds of meaning are realised **simultaneously** through different parts of English grammar.\n",
    "\n",
    "* Experiential meanings are made through **transitivity choices**.\n",
    "* Interpersonal meanings are made through **mood choices**\n",
    "\n",
    "\n",
    "Transitivity choices include fitting together configurations of:\n",
    "\n",
    "* Participants (*a man, green bikes*)\n",
    "* Processes (*sleep, has always been, is considering*)\n",
    "* Circumstances (*on the weekend*, *in Australia*)\n",
    "\n",
    "Mood features of a language include:\n",
    "\n",
    "* Mood types (*declarative, interrogative, imperative*)\n",
    "* Modality (*would, can, might*)\n",
    "* Lexical density--wordshe number of words per clause, the number of content to non-content words, etc.\n",
    "\n",
    "Lexical density is usually a good indicator of the general tone of texts. The language of academia, for example, often has a huge number of nouns to verbs. We can approximate an academic tone simply by making nominally dense clauses: \n",
    "\n",
    "      The consideration of interest is the potential for a participant of a certain demographic to be in Group A or Group B*.\n",
    "\n",
    "Notice how not only are there many nouns (*consideration*, *interest*, *potential*, etc.), but that the verbs are very simple (*is*, *to be*).\n",
    "\n",
    "In comparison, informal speech is characterised by smaller clauses, and thus more verbs.\n",
    "\n",
    "      A: Did you feel like dropping by?\n",
    "      B: I thought I did, but now I don't think I want to\n",
    "\n",
    "Here, we have only a few, simple nouns (*you*, *I*), with more expressive verbs (*feel*, *dropping by*, *think*, *want*)\n",
    "\n",
    "> **Note**: SFL argues that through *grammatical metaphor*, one linguistic feature can stand in for another. *Would you please shut the door?* is an interrogative, but it functions as a command. *invitation* is a nominalisation of a process, *invite*. We don't have time to deal with these kinds of realisations, unfortunately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the context of Fraser's speech, there are nearly twice as many nouns as verbs, and the verbs are generally quite simple ones (parts of To Be and To Have make up about a quarter). This suggests that Fraser's speech, even when giving a radio talk to his electorate, is more towards the formal end of the spectrum. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap\n",
    "So far today we have:\n",
    "* Imported text into NLTK\n",
    "* Used functions and loops to investigate metadata and organise our corpus\n",
    "* Tokenised raw text into words\n",
    "* Tagged words as parts of speech\n",
    "* Converted a list into NLTK Text for further analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stopwords\n",
    "Yesterday, when we did our frequency counts of the books in the NLTK Library, we noticed that a lot of speace was taken up by little words like 'and' and 'of' and 'the' which don't add a lot to our understanding of text. These are called 'stop words'. It will help our analysis if we exclude them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fdist1 = nltk.FreqDist(tokens) #this is a frequency distribution of all the words in the corpus. It is not conditional\n",
    "fdist1.most_common()[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(len(speech_text))\n",
    "print(len(set(speech_text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#First let's get rid of the puncutation\n",
    "speech = [word for word in speech_text if word.isalpha()]\n",
    "print(len(speech))#Then get rid of capitals\n",
    "vocab = [word.lower() for word in speech]\n",
    "print(len(set(vocab)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "#Create a variable that contains all the stopwords in the NLTK corpus\n",
    "ignored_words = nltk.corpus.stopwords.words('english')\n",
    "unstopped = [word for word in vocab if word not in stopwords.words('english')]\n",
    "fdist2 = nltk.FreqDist(unstopped)\n",
    "fdist2.most_common()[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list we have now is probably more intersting if we wanted to get a sense of the key issues that Fraser was talking about in this speech. Note, we're working with a very small sample here, only 800 words long. This sort of analysis is much more useful over really big corpora.\n",
    "\n",
    "*Note: We could have condensed the first two steps into a single line of code that looked like this:*\n",
    "\n",
    "        unstopped = [word for word in speech if word.lower() not in stopwords.words('english') and word.isalpha()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collocation\n",
    "We've just used collocation to test a hypothesis about the most common nouns in the speech we were investigating. Collocation can be quite a powerful tool for finding features of language.\n",
    "\n",
    "First, let's look for bigrams in the whole list of tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.collocations import *\n",
    "bigram_measures = nltk.collocations.BigramAssocMeasures()\n",
    "finder = BigramCollocationFinder.from_words(tokens)\n",
    "sorted(finder.nbest(bigram_measures.raw_freq, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't tell us much. Let's try again with 'unstopped' our list of tokens with the punctuation and stopwords removed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.collocations import *\n",
    "bigram_measures = nltk.collocations.BigramAssocMeasures()\n",
    "finder = BigramCollocationFinder.from_words(unstopped)\n",
    "sorted(finder.nbest(bigram_measures.raw_freq, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as identifying collocations (words that appear near each other), we can also look for n-grams or clusters, which appear immediately adjacent to each other. Repeated N-grams are a good way to get a sense of what a text is about. First, let's see how n-grams are created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(sent2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from nltk.util import ngrams\n",
    "trigrams = ngrams(sent2, 3)\n",
    "for gram in trigrams:\n",
    "    print(gram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of trigrams in the sentence, and they don't tell us much. It's when n-grams are repeated that they start to get interesting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "#this will let us find duplicates in our list of n-grams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Define a function that will find duplicate lists within a list (i.e. duplicate n-grams within a text)\n",
    "def list_duplicates(seq):\n",
    "    tally = defaultdict(list) \n",
    "    for i, item in enumerate(seq): #returns bigrams containing a count and the values obtained from iterating over seq\n",
    "        tally[item].append(i) #append new items to the existing items \n",
    "    return ((len(locs), key) for key, locs in tally.items() if len(locs) > threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `defaultdict()` means that if a key is not found in the dictionary, then instead of a KeyError being thrown, a new entry is created. The type of this new entry is given by the argument of defaultdict, in this case a `list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Define a function that will find n-grams that occur at least 4 times\n",
    "def ngrammer(text, gramsize, threshold = 4): #give mutiple arguments to the function\n",
    "    def list_duplicates(seq): #you can define functions without functions\n",
    "        tally = defaultdict(list)\n",
    "        for i, item in enumerate(seq):\n",
    "            tally[item].append(i)\n",
    "        return ((len(locs), key) for key, locs in tally.items() if len(locs) > threshold) #return n-grams if > threshold\n",
    "    raw_grams = ngrams(text, gramsize) \n",
    "    dupes = list_duplicates(raw_grams)\n",
    "    return sorted(dupes, reverse = True) #show dupes from highest to lowest, hence reverse=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sense = [word for word in text2 if word.isalpha()] #remove non-alphabetical characters from Sense and Sensibility\n",
    "ngrammer(sense, 3, threshold = 20) #test our ngrammer function on Sense and Sensibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ngrammer(tokens, 3, threshold = 2) #Test function on Fraser data\n",
    "#try changing the size and threhold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}