hist3368-week8-lemmatizing-in-pandas.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hist 3368 \n",
    "## Cleaning, Lemmatizing, and Visualization with Congress in Pandas\n",
    "\n",
    "#### By Jo Guldi"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Some Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/group/history/hist_3368-jguldi\n"
     ]
    }
   ],
   "source": [
    "cd /scratch/group/history/hist_3368-jguldi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Give this several minutes; we're reading in big data:***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "congress = pd.read_csv(\"congress1967-2010.csv\")\n",
    "#congress = pd.read_csv(\"eighties_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do a couple of basic cleaning steps. Let's look at the actual text output of the Content column to get an idea of what we're dealing with.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Those who do not enjoy the privilege of the floor will please retire from the Chamber.\n",
      "cleared of all attaches. unless they have absolutely important business to attend to in the Chamber.\n",
      "lly needed for the next few minutes of the deliberations of the Senate will tetire from the Chamber.\n"
     ]
    }
   ],
   "source": [
    "for contenttext in congress['speech'].head(3): # for the first three entries in the 'Content' column\n",
    "    print(contenttext[-100:]) # print the last 100 characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that there are uppercase words, punctuation marks, and stopwords that will interfere with our analysis unless we do away with them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's package all of these commands into a function, defined with \"def,\" and use .apply() to apply the function to each item in the column 'speech.'''**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that we can split the text of the 'Content' column into words, lowercase them, stopword them, and lemmatize them using some familiar commands. \n",
    "\n",
    "    .lower()\n",
    "    .split()\n",
    "    wn.morphy()\n",
    "    if word in stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also add some steps to screen out digits and initials:\n",
    "\n",
    "    if not word.isdigit()\n",
    "    if len(word) > 1\n",
    "    \n",
    "Note the use of \"len()\", which asks the \"length\" of a string in characters.  If the length of a word -- len(word) -- is greater than 1, we keep it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load stopwords and software\n",
    "from nltk.corpus import stopwords # this calls all multilingual stopword lists from NLTK\n",
    "from nltk.corpus import wordnet as wn\n",
    "stop = stopwords.words('english') # this command calls only the English stopwords, labeling them \"stop\"\n",
    "stop_set = set(stop) # use the Python native command \"set\" to streamline how the stopwords are stored, improve performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a function that does all the cleanup\n",
    "\n",
    "def cleaning_step(row):\n",
    "    \n",
    "    clean_row = row.replace('[^\\w\\s]','') # remove punctuation\n",
    "    clean_row = clean_row.split() # split into words\n",
    "    clean_row = [wn.morphy(word.lower()) for word in clean_row  # lemmatize\n",
    "                  if word not in stop_set\n",
    "                 if not word.isdigit() # if it isn't a number)\n",
    "                 if len(word) > 1] # if it's longer than one character\n",
    "    clean_row = filter(None, clean_row) # remove any 'None's that result from cases such as wn.morphy(\"the\")\n",
    "    clean_row = ' '.join(clean_row) # glue the words back together into one string per row\n",
    "    \n",
    "    return(clean_row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***This may take some time.  Lemmatizing is computation intensive. Allot 30 minutes.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "congress['speech'] = congress['speech'].apply(cleaning_step) \n",
    "congress[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the data to see what we've done. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for contenttext in congress['speech'].head(3): # for the first three entries in the 'Content' column\n",
    "    print(contenttext[-1000:]) # print the last 100 characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the data for later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ~/digital-history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "congress.to_csv(\"lemmatized-congress1968.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for use if you need to re-load\n",
    "# congress = pd.read_csv(\"lemmatized-congress1967-2010.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overall Visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's paste together all the words in the 'speech' column to get a list that we'll call 'allwords.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allwords = \" \".join(congress['speech'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a rough sense of what's in the 'Content' Column by creating a wordcloud."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The wordcloud package has its own built-in function to split a block of text.  It just needs one big block of text assembled from all the rows in the 'Content' column.  We'll use the join() command to paste together all the entries in df['Content'], calling the result 'allwords.'  Then we'l use the WordCloud().generate() command to make a wordcloud from the variable 'allwords'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import software \n",
    "!pip install wordcloud --user\n",
    "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "stop_words = set(STOPWORDS)\n",
    "\n",
    "# make a wordcloud\n",
    "wordcloud = WordCloud(stopwords=stop, background_color=\"white\").generate(allwords)\n",
    "plt.figure(figsize=(12, 12))\n",
    "plt.imshow(wordcloud, interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's visualize the most frequent words, breaking the variable 'allwords' down into individual words using split().  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordlist = allwords.split()\n",
    "wordlist[:10] # look at the first ten elements of the list only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, count the individual words using the pandas commands \"Series()\" and \"value_counts()\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordcounts = pd.Series(wordlist).value_counts()[:20]\n",
    "wordcounts[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, plot those values as a well-labeled barchart.  Notice that the axes are well-labeled and that the chart has a title that describes the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordcounts.plot(kind='bar', \n",
    "                title='Most frequent words in the CONTENT column of EDGAR for 8 key companies',\n",
    "                 figsize=(6, 6)\n",
    "               )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}